Symfony CRM: Obogatite potencijalne klijente automatskim uvidima u tehnologije web-stranica

Symfony CRM: Enrich Leads with Automatic Website Technology Insights

Symfony CRM: Obogatite potencijalne klijente automatskim uvidima u tehnologije web-stranica

Mihajlo
5. ruj 2026.
Vodiči

A lead’s website often reveals more than a discovery form does. A concise note such as “WordPress, WooCommerce, Cloudflare; analytics detected” gives an agency immediate context for qualification, technical discovery, and proposal writing.

This tutorial adds that note to a small Symfony CRM. Website inspection runs asynchronously, the remote response is normalized at the application boundary, and failures remain visible without blocking lead creation. The result is useful enrichment rather than an opaque dump of API data.

Get access to the detector

Open the Website Technology Detector service page.
Choose an available Free, Plus, or Pro plan and complete its activation.
Open the official service documentation.
Find the Service token panel and copy the service-scoped token.

The service requires authentication. It accepts a Bearer token, an X-API-Token header, or a token query parameter. This implementation uses the Bearer form so the credential stays out of URLs, access logs, browser history, and monitoring labels.

Regenerating the service token revokes the previously active token. Treat rotation as a deployment operation: update the secret in every environment before retiring workers that still hold the old value.

Confirm the endpoint

The exact request is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. Its JSON body contains a single url value. Before writing application code, make one minimal request from a trusted shell:

read -s WTD_TOKEN
curl --request POST \
  --url https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies \
  --header "Authorization: Bearer ${WTD_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com"}'
unset WTD_TOKEN

Do not paste the response into a public issue or fixture: evidence and redirect information can disclose details about a prospect’s site. Store the development credential in the uncommitted .env.local file:

WEBSITE_TECHNOLOGY_TOKEN=YOUR_SERVICE_TOKEN

In production, inject the same variable through the hosting platform’s secret manager or Symfony’s secrets system. Never commit the real value to .env.

Architecture: enrich outside the web request

The CRM already has a Doctrine Lead entity containing an ID and website URL. We will add normalized insight data, a readable summary, and explicit scan state. A console command dispatches lead IDs through Symfony Messenger; a handler reloads each lead, calls the detector, maps its response, and persists the result.

This boundary matters. Lead creation stays fast even when a public website is slow, the API is rate-limited, or a worker is temporarily unavailable. Passing only the database ID through Messenger also avoids serializing stale Doctrine entities.

The relevant structure is deliberately small:

src/
  Command/EnrichLeadsCommand.php
  Entity/Lead.php
  Message/DetectLeadTechnologies.php
  MessageHandler/DetectLeadTechnologiesHandler.php
  WebsiteTechnology/DetectorClient.php
  WebsiteTechnology/DetectorException.php
  WebsiteTechnology/TechnologyInsight.php
tests/
  WebsiteTechnology/DetectorClientTest.php
config/
  packages/messenger.yaml
  services.yaml

Persist normalized results and visible state

Add these fields and methods to Lead. The JSON column retains normalized evidence and redirects for later inspection, while the text column supports fast display in lists and exports.

<?php

namespace App\Entity;

use Doctrine\DBAL\Types\Types;
use Doctrine\ORM\Mapping as ORM;

class Lead
{
    #[ORM\Column(length: 20)]
    private string $technologyScanStatus = 'pending';

    #[ORM\Column(type: Types::TEXT, nullable: true)]
    private ?string $technologySummary = null;

    #[ORM\Column(type: Types::JSON, nullable: true)]
    private ?array $technologyInsights = null;

    #[ORM\Column(type: Types::TEXT, nullable: true)]
    private ?string $technologyScanError = null;

    #[ORM\Column(nullable: true)]
    private ?\DateTimeImmutable $technologyScannedAt = null;

    public function markTechnologyScanRunning(): void
    {
        $this->technologyScanStatus = 'running';
        $this->technologyScanError = null;
    }

    public function applyTechnologyInsight(
        string $summary,
        array $insights,
        \DateTimeImmutable $scannedAt,
    ): void {
        $this->technologySummary = $summary;
        $this->technologyInsights = $insights;
        $this->technologyScannedAt = $scannedAt;
        $this->technologyScanStatus = 'complete';
        $this->technologyScanError = null;
    }

    public function markTechnologyScanFailed(string $safeMessage): void
    {
        $this->technologyScanStatus = 'failed';
        $this->technologyScanError = mb_substr($safeMessage, 0, 1000);
    }
}

Generate and review the migration rather than hand-writing SQL that may not match your database platform:

php bin/console make:migration
php bin/console doctrine:migrations:migrate --no-interaction

Normalize the API contract at one boundary

The detector returns deterministic findings with confidence, evidence, version information, and redirect information. Remote JSON should still be considered untrusted. Missing keys, unexpected scalar values, and malformed entries must not leak throughout the domain.

<?php

namespace App\WebsiteTechnology;

final readonly class TechnologyInsight
{
    public function __construct(
        public array $technologies,
        public array $redirects,
    ) {}

    public static function fromPayload(array $payload): self
    {
        $items = is_array($payload['technologies'] ?? null)
            ? $payload['technologies']
            : [];

        $technologies = [];

        foreach ($items as $item) {
            if (!is_array($item) || !is_string($item['name'] ?? null)) {
                continue;
            }

            $name = trim($item['name']);
            if ($name === '') {
                continue;
            }

            $confidence = is_numeric($item['confidence'] ?? null)
                ? (float) $item['confidence']
                : null;

            $evidence = is_array($item['evidence'] ?? null)
                ? array_values(array_filter(
                    $item['evidence'],
                    static fn (mixed $value): bool => is_string($value)
                ))
                : [];

            $versions = is_array($item['versions'] ?? null)
                ? array_values(array_filter(
                    $item['versions'],
                    static fn (mixed $value): bool => is_string($value)
                ))
                : [];

            $technologies[] = [
                'name' => $name,
                'confidence' => $confidence,
                'versions' => $versions,
                'evidence' => $evidence,
            ];
        }

        $redirects = is_array($payload['redirects'] ?? null)
            ? array_values($payload['redirects'])
            : [];

        return new self($technologies, $redirects);
    }

    public function summary(): string
    {
        if ($this->technologies === []) {
            return 'No technologies were confidently identified.';
        }

        return implode('; ', array_map(
            static function (array $technology): string {
                $version = $technology['versions'][0] ?? null;

                return $version === null
                    ? $technology['name']
                    : $technology['name'].' '.$version;
            },
            $this->technologies,
        ));
    }

    public function toArray(): array
    {
        return [
            'technologies' => $this->technologies,
            'redirects' => $this->redirects,
        ];
    }
}

The summary intentionally avoids converting confidence into a percentage because a client should not guess the service’s scale. The original numeric score remains available in normalized JSON. Evidence is retained for explanation and troubleshooting, not concatenated into a noisy CRM list.

Build a bounded, failure-aware HTTP client

Symfony’s HttpClientInterface provides dependency injection, testable transports, and bounded request options. The client below retries only transport failures, HTTP 429 responses, and server errors. Validation and authentication failures are permanent until the input or configuration changes.

<?php

namespace App\WebsiteTechnology;

use Psr\Log\LoggerInterface;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
use Symfony\Contracts\HttpClient\HttpClientInterface;

final class DetectorException extends \RuntimeException
{
    public function __construct(
        string $message,
        public readonly bool $retryable,
    ) {
        parent::__construct($message);
    }
}

final readonly class DetectorClient
{
    private const ENDPOINT =
        'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies';

    public function __construct(
        private HttpClientInterface $httpClient,
        private LoggerInterface $logger,
        private string $websiteTechnologyToken,
    ) {
        if (trim($websiteTechnologyToken) === '') {
            throw new \LogicException('Website technology token is not configured.');
        }
    }

    public function detect(string $url): TechnologyInsight
    {
        for ($attempt = 1; $attempt <= 3; $attempt++) {
            try {
                $response = $this->httpClient->request('POST', self::ENDPOINT, [
                    'headers' => [
                        'Authorization' => 'Bearer '.$this->websiteTechnologyToken,
                        'Accept' => 'application/json',
                    ],
                    'json' => ['url' => $url],
                    'timeout' => 5.0,
                    'max_duration' => 15.0,
                ]);

                $status = $response->getStatusCode();

                if ($status >= 200 && $status < 300) {
                    try {
                        $payload = json_decode(
                            $response->getContent(false),
                            true,
                            512,
                            JSON_THROW_ON_ERROR,
                        );
                    } catch (\JsonException $exception) {
                        throw new DetectorException(
                            'Detector returned invalid JSON.',
                            false,
                        );
                    }

                    if (!is_array($payload)) {
                        throw new DetectorException(
                            'Detector returned an unexpected payload.',
                            false,
                        );
                    }

                    return TechnologyInsight::fromPayload($payload);
                }

                $retryable = $status === 429 || $status >= 500;

                $this->logger->warning('Technology detection request failed.', [
                    'status' => $status,
                    'attempt' => $attempt,
                    'retryable' => $retryable,
                ]);

                if (!$retryable) {
                    throw new DetectorException(
                        'Detector rejected the request with HTTP '.$status.'.',
                        false,
                    );
                }

                if ($attempt === 3) {
                    throw new DetectorException(
                        'Detector remained unavailable after bounded retries.',
                        true,
                    );
                }

                $retryAfter = $response->getHeaders(false)['retry-after'][0] ?? null;
                $delay = ctype_digit((string) $retryAfter)
                    ? min(5, (int) $retryAfter)
                    : min(4, 2 ** ($attempt - 1));

                usleep($delay * 1_000_000);
            } catch (TransportExceptionInterface $exception) {
                $this->logger->warning('Technology detector transport failure.', [
                    'attempt' => $attempt,
                    'exception' => $exception::class,
                ]);

                if ($attempt === 3) {
                    throw new DetectorException(
                        'Detector could not be reached after bounded retries.',
                        true,
                    );
                }

                usleep((2 ** ($attempt - 1)) * 1_000_000);
            }
        }

        throw new DetectorException('Technology detection failed.', true);
    }
}

The logs contain status and attempt information, but never the token, response body, evidence, or lead URL. That is enough for operational diagnosis without turning logs into a second CRM database.

Wire environment configuration

Bind the constructor argument in config/services.yaml:

services:
  _defaults:
    autowire: true
    autoconfigure: true
    bind:
      string $websiteTechnologyToken: '%env(WEBSITE_TECHNOLOGY_TOKEN)%'

  App\:
    resource: '../src/'

Process leads through Messenger

The message contains only the lead ID. The handler validates that the lead still exists and still has an HTTP or HTTPS website before making a billable or quota-consuming request.

<?php

namespace App\Message;

final readonly class DetectLeadTechnologies
{
    public function __construct(public int $leadId) {}
}

// src/MessageHandler/DetectLeadTechnologiesHandler.php
namespace App\MessageHandler;

use App\Entity\Lead;
use App\Message\DetectLeadTechnologies;
use App\WebsiteTechnology\DetectorClient;
use App\WebsiteTechnology\DetectorException;
use Doctrine\ORM\EntityManagerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;

#[AsMessageHandler]
final readonly class DetectLeadTechnologiesHandler
{
    public function __construct(
        private EntityManagerInterface $entityManager,
        private DetectorClient $detector,
    ) {}

    public function __invoke(DetectLeadTechnologies $message): void
    {
        $lead = $this->entityManager->find(Lead::class, $message->leadId);

        if (!$lead instanceof Lead) {
            return;
        }

        $url = $lead->getWebsiteUrl();
        $scheme = is_string($url) ? parse_url($url, PHP_URL_SCHEME) : null;

        if (!in_array($scheme, ['http', 'https'], true)) {
            $lead->markTechnologyScanFailed('Website must use HTTP or HTTPS.');
            $this->entityManager->flush();
            return;
        }

        $lead->markTechnologyScanRunning();
        $this->entityManager->flush();

        try {
            $insight = $this->detector->detect($url);

            $lead->applyTechnologyInsight(
                $insight->summary(),
                $insight->toArray(),
                new \DateTimeImmutable(),
            );
        } catch (DetectorException $exception) {
            $category = $exception->retryable
                ? 'Temporary detector failure; retry later.'
                : 'Technology detection request was rejected.';

            $lead->markTechnologyScanFailed($category);
        }

        $this->entityManager->flush();
    }
}

Configure an asynchronous transport in config/packages/messenger.yaml. The transport DSN remains environment-backed; Doctrine transport is a practical default for a small CRM, provided symfony/doctrine-messenger is installed.

framework:
  messenger:
    failure_transport: failed
    transports:
      async:
        dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
      failed: 'doctrine://default?queue_name=failed'
    routing:
      App\Message\DetectLeadTechnologies: async

An app:enrich-leads command can query leads whose status is pending, or failed leads whose retry window has elapsed, and dispatch one DetectLeadTechnologies message per ID. Keep that repository query paginated so a large import does not load every entity at once.

php bin/console app:enrich-leads
php bin/console messenger:consume async \
  --time-limit=3600 \
  --memory-limit=128M \
  --limit=500

Run the worker under systemd, Supervisor, or the process manager supplied by the hosting platform. Restart workers during deployments because long-running PHP processes do not automatically load new code or rotated environment values.

Test without calling the live service

MockHttpClient keeps tests deterministic and prevents quota consumption. This test also proves that malformed technology entries are discarded while valid evidence and redirects survive normalization.

<?php

namespace App\Tests\WebsiteTechnology;

use App\WebsiteTechnology\DetectorClient;
use PHPUnit\Framework\TestCase;
use Psr\Log\NullLogger;
use Symfony\Component\HttpClient\MockHttpClient;
use Symfony\Component\HttpClient\Response\MockResponse;

final class DetectorClientTest extends TestCase
{
    public function testItMapsAValidResponseDefensively(): void
    {
        $responses = [
            new MockResponse(json_encode([
                'technologies' => [
                    [
                        'name' => 'Example CMS',
                        'confidence' => 0.97,
                        'versions' => ['4.2'],
                        'evidence' => ['generator metadata'],
                    ],
                    ['confidence' => 0.4],
                ],
                'redirects' => [
                    ['from' => 'http://example.com', 'to' => 'https://example.com'],
                ],
            ], JSON_THROW_ON_ERROR), [
                'http_code' => 200,
                'response_headers' => ['content-type: application/json'],
            ]),
        ];

        $client = new MockHttpClient($responses);
        $detector = new DetectorClient($client, new NullLogger(), 'test-token');

        $insight = $detector->detect('https://example.com');

        self::assertSame('Example CMS 4.2', $insight->summary());
        self::assertCount(1, $insight->technologies);
        self::assertSame(0.97, $insight->technologies[0]['confidence']);
        self::assertCount(1, $insight->redirects);
    }
}

Add companion cases for HTTP 401, HTTP 429 exhaustion, malformed JSON, transport exceptions, an absent lead, and an invalid URL scheme. Never use a production token in the test environment.

Security, operations, and common failures

Only submit websites that the service is permitted to inspect. Validate URLs before dispatch, restrict schemes to HTTP and HTTPS, and do not allow internal schemes such as file. If users can supply arbitrary URLs, assess server-side request risks at the product boundary even though the remote service performs the fetch.

401 or 403: confirm the service-scoped token, plan activation, and deployment secret. Do not retry automatically.
400-series validation error: inspect the submitted public URL and contract. Retrying identical input wastes quota.
429: respect bounded backoff, reduce worker concurrency, and reschedule failed leads later.
500-series or transport error: retain the lead, record a safe temporary state, and retry through a controlled batch.
Empty detections: treat a valid empty result as a successful scan, not an exception.
Stuck running state: monitor scan age and requeue records whose worker was terminated mid-job.

Track counts and latency by outcome, not by website. Alert on sustained authentication failures, unusual rate limiting, growing queue depth, and old running records. A correlation ID attached to the message and log context is useful, provided it contains no credential or prospect data.

Final verification checklist

The real token exists only in environment-backed secret storage.
The application calls the exact POST endpoint with a JSON url.
Lead creation succeeds even when the detector is unavailable.
Confidence, evidence, versions, and redirects are validated before persistence.
Authentication and validation failures are not blindly retried.
Timeouts, retry counts, and backoff are bounded.
Logs exclude tokens, raw responses, evidence, and lead URLs.
Workers restart after code deployment or token rotation.
Mock-based tests run without external network access.
A processed lead displays a readable technology summary and a clear scan state.

The most valuable part of this integration is not the HTTP request. It is the disciplined path from an unreliable external observation to a trustworthy CRM fact: queued, validated, explainable, observable, and safe to revisit. Build that boundary well, and a simple website URL becomes useful context for every conversation that follows.