Tutorials

Symfony: Monitor Client Sites for Tech Stack Changes with Website Detector API

Symfony: Monitor Client Sites for Tech Stack Changes with Website Detector API

A client site can change underneath you without a deployment: a CMS upgrade replaces plugins, a CDN appears, analytics disappears, or a redesign quietly moves the site to another platform. Those changes can affect integrations, performance assumptions, security reviews, and maintenance estimates.

This tutorial builds a production-oriented Symfony monitor that checks a public website with the Website Technology Detector API, stores a normalized baseline, and emails a developer only when the detected technology stack changes. It runs as a scheduled console command, uses bounded retries, and treats the remote response as untrusted data.

Get access and copy a service token

  1. Register at https://ai.mihajlo.mk/register, or use https://ai.mihajlo.mk/login if you already have an account.
  2. Open the Website Technology Detector service page.
  3. Choose an available Free, Plus, or Pro plan and complete its activation.
  4. Open the official service documentation.
  5. Find the Service token panel and copy the service-scoped token.

The service requires authentication. It accepts a Bearer token, an X-API-Token header, or a token query parameter. We will use the Bearer form because credentials in query strings can leak into access logs, browser history, and monitoring systems.

Regenerating the service token revokes the previously active token. Coordinate rotation with deployment so the application receives the new value before or immediately after regeneration.

Confirm the endpoint before writing application code

The exact request is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. Its JSON body contains url.

curl --request POST \
  --url 'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer YOUR_SERVICE_TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{"url":"https://www.example.com"}'

A successful response contains confidence-scored technology detections, evidence, version information when available, and redirect information. The integration below deliberately avoids depending on one undocumented outer envelope. It recognizes the documented concepts at the application boundary and rejects malformed JSON.

For local development, put the credential in .env.local, which must remain uncommitted. In production, inject the same variable through the hosting platform or Symfony secrets rather than baking it into an image.

WEBSITE_DETECTOR_TOKEN=YOUR_SERVICE_TOKEN
[email protected]
[email protected]
MAILER_DSN=smtp://USERNAME:[email protected]:587

Architecture and project setup

A scheduled Symfony command is a better fit than an HTTP controller here. Checks should continue without a browser request, and transient detector or mail failures need to produce a failed process that cron or the deployment platform can observe.

The command calls a dedicated API client, maps the response into a stable domain snapshot, and passes it to a file-backed state store. The first successful run creates a baseline without sending an alert. Later runs send an email before committing a changed snapshot, so a mail failure does not silently consume the notification.

This file store is intentionally small and practical for one application instance. A multi-replica deployment should replace it with a database-backed repository or shared durable storage and retain equivalent locking semantics.

composer require symfony/http-client symfony/mailer
composer require --dev phpunit/phpunit:^11.0

mkdir -p src/Technology tests/Technology var/technology-monitor
src/
  Command/MonitorTechnologiesCommand.php
  Technology/DetectionReport.php
  Technology/DetectorException.php
  Technology/SnapshotStore.php
  Technology/WebsiteDetector.php
tests/
  Technology/WebsiteDetectorTest.php
var/
  technology-monitor/

Messenger would add little value for a single scheduled network operation. If hundreds of unrelated sites eventually need independent schedules and concurrency, dispatching one message per site becomes worthwhile. Until then, the command is easier to deploy and diagnose.

Map the remote response defensively

The mapper searches recursively because transport envelopes can evolve. A node is treated as a technology only when it has a non-empty name and at least one detection signal such as confidence, evidence, or version data. Unknown fields remain harmless, missing optional fields become null or empty arrays, and list-like evidence is sorted to prevent order-only alerts.

<?php
// src/Technology/DetectionReport.php
namespace App\Technology;

final class DetectionReport
{
    private function __construct(
        public readonly array $technologies,
        public readonly array $redirects,
    ) {}

    public static function fromPayload(array $payload): self
    {
        $technologies = [];
        $redirects = [];
        self::walk($payload, '$', $technologies, $redirects);

        $unique = [];
        foreach ($technologies as $technology) {
            $unique[hash('sha256', self::encode($technology))] = $technology;
        }

        $technologies = array_values($unique);
        usort(
            $technologies,
            fn (array $a, array $b): int => self::encode($a) <=> self::encode($b)
        );
        ksort($redirects);

        return new self($technologies, $redirects);
    }

    public function snapshot(): array
    {
        return [
            'technologies' => $this->technologies,
            'redirects' => $this->redirects,
        ];
    }

    public function fingerprint(): string
    {
        return hash('sha256', self::encode($this->snapshot()));
    }

    private static function walk(
        mixed $value,
        string $path,
        array &$technologies,
        array &$redirects,
    ): void {
        if (!is_array($value)) {
            return;
        }

        $fields = [];
        foreach ($value as $key => $child) {
            if (is_string($key)) {
                $fields[strtolower(str_replace('-', '_', $key))] = $child;
            }
        }

        $name = $fields['name'] ?? $fields['technology'] ?? null;
        $hasSignal = array_key_exists('confidence', $fields)
            || array_key_exists('evidence', $fields)
            || array_key_exists('version', $fields)
            || array_key_exists('versions', $fields);

        if (is_string($name) && trim($name) !== '' && $hasSignal) {
            $confidence = $fields['confidence'] ?? null;

            $technologies[] = [
                'name' => trim($name),
                'confidence' => is_numeric($confidence) ? (float) $confidence : null,
                'versions' => self::stableList(
                    $fields['versions'] ?? $fields['version'] ?? []
                ),
                'evidence' => self::stableList($fields['evidence'] ?? []),
            ];
        }

        foreach ($value as $key => $child) {
            $segment = is_string($key) ? $key : (string) $key;
            $normalized = strtolower(str_replace('-', '_', $segment));

            if (str_contains($normalized, 'redirect') || $normalized === 'final_url') {
                $redirects[$path.'.'.$segment] = self::canonicalize($child);
            }

            self::walk($child, $path.'.'.$segment, $technologies, $redirects);
        }
    }

    private static function stableList(mixed $value): array
    {
        $items = is_array($value) ? array_values($value) : [$value];
        $items = array_map(self::canonicalize(...), $items);
        usort($items, fn (mixed $a, mixed $b): int => self::encode($a) <=> self::encode($b));

        return $items;
    }

    private static function canonicalize(mixed $value): mixed
    {
        if (!is_array($value)) {
            return $value;
        }

        if (!array_is_list($value)) {
            ksort($value);
        }

        return array_map(self::canonicalize(...), $value);
    }

    private static function encode(mixed $value): string
    {
        return json_encode(
            $value,
            JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE
        );
    }
}

Including redirects in the snapshot is useful because a move to another host can explain an apparent stack replacement. Request IDs, timestamps, and unrelated metadata are excluded, avoiding noisy alerts.

Build the resilient HTTP client

Failures need categories rather than an undifferentiated exception. Authentication and validation errors require intervention and must not be retried. Transport failures, rate limits, and selected upstream errors may be transient, so the client makes at most three attempts with bounded backoff.

<?php
// src/Technology/DetectorException.php
namespace App\Technology;

final class DetectorException extends \RuntimeException
{
    public function __construct(
        public readonly string $kind,
        string $message,
        ?\Throwable $previous = null,
    ) {
        parent::__construct($message, 0, $previous);
    }
}
<?php
// src/Technology/WebsiteDetector.php
namespace App\Technology;

use Psr\Log\LoggerInterface;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
use Symfony\Contracts\HttpClient\HttpClientInterface;
use Symfony\Contracts\HttpClient\ResponseInterface;

final class WebsiteDetector
{
    private const ENDPOINT =
        'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies';

    public function __construct(
        private HttpClientInterface $http,
        private LoggerInterface $logger,
        private string $token,
    ) {}

    public function detect(string $url): DetectionReport
    {
        for ($attempt = 1; $attempt <= 3; $attempt++) {
            try {
                $response = $this->http->request('POST', self::ENDPOINT, [
                    'headers' => [
                        'Accept' => 'application/json',
                        'Authorization' => 'Bearer '.$this->token,
                    ],
                    'json' => ['url' => $url],
                    'max_duration' => 20.0,
                    'timeout' => 10.0,
                ]);

                $status = $response->getStatusCode();

                if ($this->isRetryable($status) && $attempt < 3) {
                    $this->logger->warning('Technology detector retry scheduled', [
                        'status' => $status,
                        'attempt' => $attempt,
                    ]);
                    usleep($this->delayMicroseconds($response, $attempt));
                    continue;
                }

                if ($status < 200 || $status >= 300) {
                    throw $this->statusException($status);
                }

                try {
                    $payload = json_decode(
                        $response->getContent(false),
                        true,
                        512,
                        JSON_THROW_ON_ERROR
                    );
                } catch (\JsonException $exception) {
                    throw new DetectorException(
                        'invalid_response',
                        'Detector returned invalid JSON.',
                        $exception
                    );
                }

                if (!is_array($payload)) {
                    throw new DetectorException(
                        'invalid_response',
                        'Detector response must be a JSON object or array.'
                    );
                }

                return DetectionReport::fromPayload($payload);
            } catch (TransportExceptionInterface $exception) {
                if ($attempt === 3) {
                    throw new DetectorException(
                        'transport',
                        'Detector could not be reached after three attempts.',
                        $exception
                    );
                }

                $this->logger->warning('Technology detector transport retry', [
                    'attempt' => $attempt,
                ]);
                usleep(250_000 * (2 ** ($attempt - 1)));
            }
        }

        throw new DetectorException('transport', 'Detector request did not complete.');
    }

    private function isRetryable(int $status): bool
    {
        return in_array($status, [429, 502, 503, 504], true);
    }

    private function statusException(int $status): DetectorException
    {
        $kind = match (true) {
            $status === 401 || $status === 403 => 'authentication',
            $status === 400 || $status === 422 => 'validation',
            $status === 429 => 'rate_limit',
            $status >= 500 => 'upstream',
            default => 'http',
        };

        return new DetectorException($kind, "Detector returned HTTP {$status}.");
    }

    private function delayMicroseconds(
        ResponseInterface $response,
        int $attempt,
    ): int {
        $retryAfter = $response->getHeaders(false)['retry-after'][0] ?? null;

        if (is_string($retryAfter) && ctype_digit($retryAfter)) {
            return min((int) $retryAfter, 10) * 1_000_000;
        }

        return 250_000 * (2 ** ($attempt - 1));
    }
}

The client never logs the token or response body. The response body may contain evidence that is useful to the application but inappropriate for a general-purpose production log.

Store a baseline without losing alerts

The store uses an exclusive file lock. If notification delivery throws, it leaves the old snapshot intact, making the next scheduled run try again.

<?php
// src/Technology/SnapshotStore.php
namespace App\Technology;

final class SnapshotStore
{
    public function __construct(private string $directory) {}

    public function record(
        string $site,
        DetectionReport $report,
        callable $onChange,
    ): string {
        if (!is_dir($this->directory)
            && !mkdir($this->directory, 0770, true)
            && !is_dir($this->directory)) {
            throw new \RuntimeException('Cannot create snapshot directory.');
        }

        $path = $this->directory.'/'.hash('sha256', $site).'.json';
        $handle = fopen($path, 'c+');

        if ($handle === false || !flock($handle, LOCK_EX)) {
            throw new \RuntimeException('Cannot lock technology snapshot.');
        }

        try {
            rewind($handle);
            $stored = stream_get_contents($handle);
            $previous = trim((string) $stored) === ''
                ? null
                : json_decode($stored, true, 512, JSON_THROW_ON_ERROR);
            $current = $report->snapshot();

            if ($previous === null) {
                $status = 'baseline';
            } else {
                $oldFingerprint = hash(
                    'sha256',
                    json_encode($previous, JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES)
                );

                if (hash_equals($oldFingerprint, $report->fingerprint())) {
                    return 'unchanged';
                }

                $onChange($previous, $current);
                $status = 'changed';
            }

            rewind($handle);
            ftruncate($handle, 0);
            fwrite(
                $handle,
                json_encode(
                    $current,
                    JSON_THROW_ON_ERROR
                    | JSON_PRETTY_PRINT
                    | JSON_UNESCAPED_SLASHES
                )
            );
            fflush($handle);

            return $status;
        } finally {
            flock($handle, LOCK_UN);
            fclose($handle);
        }
    }
}

Create the monitoring command

<?php
// src/Command/MonitorTechnologiesCommand.php
namespace App\Command;

use App\Technology\DetectorException;
use App\Technology\SnapshotStore;
use App\Technology\WebsiteDetector;
use Psr\Log\LoggerInterface;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Mailer\MailerInterface;
use Symfony\Component\Mime\Email;

#[AsCommand(name: 'app:monitor-technologies')]
final class MonitorTechnologiesCommand extends Command
{
    public function __construct(
        private WebsiteDetector $detector,
        private SnapshotStore $store,
        private MailerInterface $mailer,
        private LoggerInterface $logger,
        private string $recipient,
        private string $sender,
    ) {
        parent::__construct();
    }

    protected function configure(): void
    {
        $this->addArgument('site', InputArgument::REQUIRED, 'Public HTTP(S) URL');
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $site = (string) $input->getArgument('site');
        $scheme = strtolower((string) parse_url($site, PHP_URL_SCHEME));

        if (!filter_var($site, FILTER_VALIDATE_URL)
            || !in_array($scheme, ['http', 'https'], true)) {
            $output->writeln('<error>Provide a valid HTTP(S) URL.</error>');
            return Command::INVALID;
        }

        try {
            $report = $this->detector->detect($site);
            $status = $this->store->record(
                $site,
                $report,
                function (array $before, array $after) use ($site): void {
                    $body = "Technology changes detected for {$site}\n\n"
                        ."Previous:\n".json_encode($before, JSON_PRETTY_PRINT)
                        ."\n\nCurrent:\n".json_encode($after, JSON_PRETTY_PRINT);

                    $this->mailer->send(
                        (new Email())
                            ->from($this->sender)
                            ->to($this->recipient)
                            ->subject('Website technology stack changed')
                            ->text($body)
                    );
                }
            );

            $this->logger->info('Technology monitor completed', [
                'site' => $site,
                'result' => $status,
                'detections' => count($report->technologies),
            ]);
            $output->writeln("Monitor result: {$status}");

            return Command::SUCCESS;
        } catch (DetectorException $exception) {
            $this->logger->error('Technology detector failed', [
                'site' => $site,
                'failure_kind' => $exception->kind,
                'message' => $exception->getMessage(),
            ]);
            $output->writeln('<error>Detector failed: '
                .$exception->kind.'</error>');

            return Command::FAILURE;
        } catch (\Throwable $exception) {
            $this->logger->error('Technology monitor failed', [
                'site' => $site,
                'exception' => $exception::class,
            ]);
            $output->writeln('<error>Monitor failed.</error>');

            return Command::FAILURE;
        }
    }
}

Bind the environment-backed values in config/services.yaml:

services:
    App\Technology\WebsiteDetector:
        arguments:
            $token: '%env(WEBSITE_DETECTOR_TOKEN)%'

    App\Technology\SnapshotStore:
        arguments:
            $directory: '%kernel.project_dir%/var/technology-monitor'

    App\Command\MonitorTechnologiesCommand:
        arguments:
            $recipient: '%env(TECH_MONITOR_RECIPIENT)%'
            $sender: '%env(TECH_MONITOR_FROM)%'

Test the boundary with MockHttpClient

Tests must never call the live service or contain a real token. The first test verifies request construction and mapping. The second confirms that an authentication failure is classified and attempted only once.

<?php
// tests/Technology/WebsiteDetectorTest.php
namespace App\Tests\Technology;

use App\Technology\DetectorException;
use App\Technology\WebsiteDetector;
use PHPUnit\Framework\TestCase;
use Psr\Log\NullLogger;
use Symfony\Component\HttpClient\MockHttpClient;
use Symfony\Component\HttpClient\Response\MockResponse;

final class WebsiteDetectorTest extends TestCase
{
    public function testItSendsUrlAndMapsAResponse(): void
    {
        $http = new MockHttpClient(function (
            string $method,
            string $url,
            array $options,
        ): MockResponse {
            self::assertSame('POST', $method);
            self::assertSame(
                'https://ai.mihajlo.mk/api/website-technology-detector/'
                .'v1/detect-technologies',
                $url
            );
            self::assertSame(['url' => 'https://www.example.com'], $options['json']);

            return new MockResponse(json_encode([
                'technologies' => [[
                    'name' => 'Example CMS',
                    'confidence' => 0.96,
                    'version' => '1.2',
                    'evidence' => ['public marker'],
                ]],
                'redirects' => [],
            ], JSON_THROW_ON_ERROR));
        });

        $report = (new WebsiteDetector(
            $http,
            new NullLogger(),
            'test-token'
        ))->detect('https://www.example.com');

        self::assertSame('Example CMS', $report->technologies[0]['name']);
        self::assertSame(0.96, $report->technologies[0]['confidence']);
    }

    public function testAuthenticationFailureIsNotRetried(): void
    {
        $http = new MockHttpClient(new MockResponse('', ['http_code' => 401]));
        $detector = new WebsiteDetector($http, new NullLogger(), 'test-token');

        try {
            $detector->detect('https://www.example.com');
            self::fail('Expected DetectorException.');
        } catch (DetectorException $exception) {
            self::assertSame('authentication', $exception->kind);
        }

        self::assertSame(1, $http->getRequestsCount());
    }
}
php bin/phpunit
php bin/console app:monitor-technologies https://www.example.com
php bin/console app:monitor-technologies https://www.example.com

The first command run should report baseline; the second should report unchanged. To verify email delivery without falsifying an API response, copy the generated snapshot, alter one saved technology value, run the command again, and restore or remove the test snapshot afterward.

Deploy, schedule, and observe it

Make var/technology-monitor writable by the application user and persistent across releases. Run the scheduler as a single instance. This cron entry checks every six hours and prevents overlapping executions:

17 */6 * * * cd /srv/app && flock -n var/client-tech.cron.lock php bin/console app:monitor-technologies https://www.example.com --env=prod >> var/log/technology-monitor.log 2>&1

Alert on non-zero exit codes or repeated log events with failure_kind. A rate_limit failure suggests reducing frequency or reviewing the active plan. An authentication failure usually means the token is wrong, expired, or was revoked by regeneration. A validation failure points to the submitted URL. Repeated transport or upstream failures deserve investigation but should not overwrite the last good baseline.

Do not expose this command through an unauthenticated web route. Keep tokens out of source control, exception messages, fixtures, and deployment logs. Restrict environment access, rotate the token deliberately, and treat evidence returned about a client site as operational data.

Final verification checklist

  • The account and Free, Plus, or Pro plan are active, and the service-scoped token comes from the documentation page.
  • WEBSITE_DETECTOR_TOKEN exists only in environment-backed configuration.
  • The client uses the exact POST endpoint, sends only the documented url input, and applies bounded timeouts.
  • Authentication and validation failures are not retried; transient failures receive at most three attempts.
  • The first successful run creates a baseline without sending a misleading alert.
  • A changed snapshot sends email before the new state is committed.
  • Tests use MockHttpClient and never contact the live service.
  • The snapshot directory is writable, persistent, backed up appropriately, and shared if more than one instance can run the command.
  • Production monitoring captures failed exit codes and structured failure categories without recording credentials.

A useful site monitor is not merely a scheduled API call. It is a carefully drawn boundary between an external observation and a decision worth interrupting someone for. Normalize the evidence, preserve the last trusted state, retry only failures that might recover, and make every alert explainable. Then a quiet change to a client website becomes a manageable engineering event instead of a surprise discovered during the next incident.

Blog author portrait

Mihajlo

I’m Mihajlo — a developer driven by curiosity, discipline, and the constant urge to create something meaningful. I share insights, tutorials, and free services to help others simplify their work and grow in the ever-evolving world of software and AI.

Search

Categories

Recent Posts

Architecting for AI: Build Software That Owns Its Evolution
Beyond the Prompt: Architecting Software Agents That Learn from Your System's Core
System Design: Architecting for Resilience in an AI-Driven World
Lead Your Devs to Own the Product Lifecycle
Build Software That Teaches Your AI Agents Better