Symfony CRM: Automatically Analyze Lead Websites with AI Technology Detection
A lead record that says “uses WordPress” is mildly useful. A lead record that says “WordPress, WooCommerce, Cloudflare, and a specific analytics stack, detected with supporting evidence” can shape discovery questions, estimates, and outreach before anyone opens the website manually.
This tutorial builds that capability into a small Symfony CRM. When a user requests an analysis, Symfony queues background work, calls the Website Technology Detector API, validates the response at the application boundary, and stores a concise summary alongside normalized evidence and redirect data. The design remains deliberately small, but it accounts for duplicate messages, slow networks, rate limits, malformed responses, credential safety, and deployment.
Get access before writing integration code
First, register an account, or sign in if you already have one.
Open the Website Technology Detector service page. Choose an available Free, Plus, or Pro plan and complete its activation. The appropriate plan depends on your expected analysis volume and operational requirements; the application architecture below works with any of the available plans.
Next, open the official service documentation. Find the Service token panel and copy the service-scoped token from there. Regenerating this token revokes the previously active token, so deploy the replacement anywhere the old value is configured before depending on it.
This service is not anonymous: every request needs a token. It accepts a Bearer token, an X-API-Token header, or a token query parameter. We will use a Bearer token because it keeps authentication out of the URL, where query parameters are more likely to appear in access logs and monitoring systems.
Verify the endpoint directly
The exact call is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. It accepts JSON containing url. Before involving Symfony, make one minimal request from a secure terminal:
curl --request POST \
'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies' \
--header 'Authorization: Bearer YOUR_SERVICE_TOKEN' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"url":"https://example.com"}'Do not paste a real token into shell history on a shared machine. For routine testing, read it from an environment variable managed by your shell or secret manager.
Store the credential in Symfony’s environment-backed configuration. Put placeholders, never production secrets, in the committed .env file:
TECHNOLOGY_DETECTOR_ENDPOINT=https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies
TECHNOLOGY_DETECTOR_TOKEN=YOUR_SERVICE_TOKEN
MESSENGER_TRANSPORT_DSN=doctrine://default?queue_name=technologySupply the real token through the deployment platform’s secret facility, an uncommitted .env.local during local development, or Symfony’s secrets system.
Architecture: keep the web request short
Website inspection is a poor fit for an interactive controller request. Remote sites can redirect, respond slowly, or trigger transient upstream failures. Symfony Messenger lets the controller acknowledge the action promptly while a worker performs the analysis.
The feature has four boundaries:
- The controller validates authorization, CSRF protection, and the lead’s public URL.
- A Messenger message carries the lead ID and a URL snapshot.
- An HTTP client owns authentication, timeouts, status classification, and JSON decoding.
- A domain mapper converts detections, confidence, evidence, versions, and redirect information into CRM-safe data.
The URL snapshot matters. If someone edits the website while an old message is waiting, the handler can discard that stale message instead of attaching yesterday’s result to today’s address.
Install the Symfony components
Assume an existing PHP 8.3+ Symfony application using Doctrine and a Lead entity with id and websiteUrl properties. Add the first-party HTTP and queue components:
composer require symfony/http-client symfony/messenger \
symfony/doctrine-messenger doctrine/doctrine-bundle
composer require --dev symfony/test-pack
php bin/console messenger:setup-transportsUse a structure that makes the external boundary obvious:
src/
Controller/AnalyzeLeadTechnologyController.php
Entity/Lead.php
Message/AnalyzeLeadTechnology.php
MessageHandler/AnalyzeLeadTechnologyHandler.php
Technology/DetectorException.php
Technology/TechnologyDetectorClient.php
Technology/TechnologyReport.php
tests/
Technology/TechnologyDetectorClientTest.phpConfigure bounded retries and dependency injection
Symfony HttpClient can retry only the failures that have a realistic chance of succeeding later. Authentication and validation responses must not be retried blindly.
# config/packages/framework.yaml
framework:
http_client:
default_options:
timeout: 10
max_duration: 25
retry_failed:
max_retries: 3
delay: 500
multiplier: 2
max_delay: 4000
jitter: 0.2
http_codes: [429, 502, 503, 504]
messenger:
failure_transport: failed
transports:
async: '%env(MESSENGER_TRANSPORT_DSN)%'
failed: 'doctrine://default?queue_name=failed'
routing:
App\Message\AnalyzeLeadTechnology: asyncThis produces a bounded backoff for rate limits and transient upstream failures. A final 429 is preserved as a structured rate-limit failure rather than causing an uncontrolled retry loop.
# config/services.yaml
services:
App\Technology\TechnologyDetectorClient:
arguments:
$endpoint: '%env(TECHNOLOGY_DETECTOR_ENDPOINT)%'
$token: '%env(TECHNOLOGY_DETECTOR_TOKEN)%'Model the result without trusting remote JSON
The API returns confidence-scored detections with evidence, versions, and redirect information. Treat all of it as untrusted input. The mapper below searches for the documented result collections without depending on a particular outer envelope, validates every detection, and caps the displayed list.
<?php
// src/Technology/TechnologyReport.php
namespace App\Technology;
final readonly class TechnologyReport
{
public function __construct(
public string $summary,
public array $detections,
public array $redirects,
) {}
public static function fromPayload(array $payload): self
{
$items = self::findArray($payload, 'detections');
$redirects = self::findArray($payload, 'redirects') ?? [];
$normalized = [];
foreach ($items ?? [] as $item) {
if (!is_array($item)) {
continue;
}
$name = $item['technology'] ?? $item['name'] ?? null;
$confidence = $item['confidence'] ?? null;
if (!is_string($name) || trim($name) === '' ||
(!is_int($confidence) && !is_float($confidence))) {
continue;
}
$versions = $item['versions'] ?? [];
if (is_string($versions)) {
$versions = [$versions];
}
$normalized[] = [
'technology' => trim($name),
'confidence' => $confidence,
'versions' => is_array($versions)
? array_values(array_filter($versions, 'is_string'))
: [],
'evidence' => is_array($item['evidence'] ?? null)
? $item['evidence']
: [],
];
}
if (($items ?? null) !== null && $normalized === [] && $items !== []) {
throw new \UnexpectedValueException('No valid detections in API response.');
}
usort(
$normalized,
fn (array $a, array $b): int => $b['confidence'] <=> $a['confidence']
);
$labels = array_map(static function (array $item): string {
$version = $item['versions'][0] ?? null;
$label = $item['technology'].($version ? ' '.$version : '');
return sprintf('%s (confidence %s)', $label, $item['confidence']);
}, array_slice($normalized, 0, 8));
return new self(
$labels ? implode(', ', $labels) : 'No supported technologies detected.',
$normalized,
$redirects,
);
}
private static function findArray(array $node, string $key): ?array
{
if (isset($node[$key]) && is_array($node[$key])) {
return $node[$key];
}
foreach ($node as $value) {
if (is_array($value) && ($found = self::findArray($value, $key)) !== null) {
return $found;
}
}
return null;
}
}The confidence value is displayed as returned rather than being converted into a percentage. That avoids silently assuming a scale. Raw responses should not be logged; evidence can be verbose and may contain page-derived material that has no place in routine logs.
Build the HTTP boundary
<?php
// src/Technology/TechnologyDetectorClient.php
namespace App\Technology;
use Psr\Log\LoggerInterface;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
use Symfony\Contracts\HttpClient\HttpClientInterface;
final readonly class TechnologyDetectorClient
{
public function __construct(
private HttpClientInterface $http,
private LoggerInterface $logger,
private string $endpoint,
private string $token,
) {}
public function detect(string $url): TechnologyReport
{
try {
$response = $this->http->request('POST', $this->endpoint, [
'auth_bearer' => $this->token,
'headers' => ['Accept' => 'application/json'],
'json' => ['url' => $url],
'timeout' => 10,
'max_duration' => 25,
]);
$status = $response->getStatusCode();
$body = $response->getContent(false);
} catch (TransportExceptionInterface $e) {
throw new DetectorException('transport', true, $e);
}
if (strlen($body) > 1_000_000) {
throw new DetectorException('response_too_large', false);
}
if ($status === 401 || $status === 403) {
throw new DetectorException('authentication', false);
}
if ($status === 429) {
throw new DetectorException('rate_limited', true);
}
if ($status === 400 || $status === 422) {
throw new DetectorException('invalid_request', false);
}
if ($status < 200 || $status >= 300) {
throw new DetectorException('upstream_'.$status, $status >= 500);
}
try {
$payload = json_decode($body, true, 512, JSON_THROW_ON_ERROR);
if (!is_array($payload)) {
throw new \JsonException('JSON root is not an object.');
}
return TechnologyReport::fromPayload($payload);
} catch (\JsonException|\UnexpectedValueException $e) {
$this->logger->warning('Technology detector returned an invalid response', [
'endpoint' => $this->endpoint,
'status' => $status,
]);
throw new DetectorException('malformed_response', false, $e);
}
}
}<?php
// src/Technology/DetectorException.php
namespace App\Technology;
final class DetectorException extends \RuntimeException
{
public function __construct(
public readonly string $kind,
public readonly bool $retryable,
?\Throwable $previous = null,
) {
parent::__construct($kind, 0, $previous);
}
}Logs contain the failure category and HTTP status, but neither the token nor response body. Symfony’s HTTP retry layer has already exhausted its bounded attempts before the final status reaches this classification.
Persist explicit CRM states
Add nullable technologySummary and JSON technologyReport fields to Lead, plus a status, error category, and timestamp. The important domain methods are small:
public function queueTechnologyAnalysis(): void
{
$this->technologyAnalysisStatus = 'queued';
$this->technologyAnalysisError = null;
}
public function completeTechnologyAnalysis(TechnologyReport $report): void
{
$this->technologySummary = $report->summary;
$this->technologyReport = [
'detections' => $report->detections,
'redirects' => $report->redirects,
];
$this->technologyAnalysisStatus = 'complete';
$this->technologyAnalysisError = null;
$this->technologyAnalyzedAt = new \DateTimeImmutable();
}
public function failTechnologyAnalysis(string $kind): void
{
$this->technologyAnalysisStatus = 'failed';
$this->technologyAnalysisError = $kind;
}Generate and inspect the migration before applying it:
php bin/console make:migration
php bin/console doctrine:migrations:migrate --no-interactionDispatch and process the analysis
<?php
// src/Message/AnalyzeLeadTechnology.php
namespace App\Message;
final readonly class AnalyzeLeadTechnology
{
public function __construct(
public int $leadId,
public string $url,
) {}
}<?php
// src/MessageHandler/AnalyzeLeadTechnologyHandler.php
namespace App\MessageHandler;
use App\Entity\Lead;
use App\Message\AnalyzeLeadTechnology;
use App\Technology\DetectorException;
use App\Technology\TechnologyDetectorClient;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
#[AsMessageHandler]
final readonly class AnalyzeLeadTechnologyHandler
{
public function __construct(
private EntityManagerInterface $em,
private TechnologyDetectorClient $detector,
private LoggerInterface $logger,
) {}
public function __invoke(AnalyzeLeadTechnology $message): void
{
$lead = $this->em->find(Lead::class, $message->leadId);
if (!$lead || $lead->getWebsiteUrl() !== $message->url) {
return;
}
try {
$lead->completeTechnologyAnalysis(
$this->detector->detect($message->url)
);
} catch (DetectorException $e) {
$lead->failTechnologyAnalysis($e->kind);
$this->logger->warning('Lead technology analysis failed', [
'lead_id' => $message->leadId,
'kind' => $e->kind,
'retryable' => $e->retryable,
]);
}
$this->em->flush();
}
}The controller should enforce your existing lead-edit permission, validate a CSRF token, and accept only public HTTP or HTTPS URLs. Reject embedded credentials, localhost, and literal private or reserved IP addresses. After validation, call queueTechnologyAnalysis(), flush the lead, and dispatch new AnalyzeLeadTechnology($lead->getId(), $lead->getWebsiteUrl()).
That ordering makes the visible state truthful before the worker starts. If dispatch itself fails, catch that exception in the controller, mark the lead failed with a queue-specific category, and show a normal CRM error message rather than pretending the work was accepted.
Test the boundary with MockHttpClient
Tests should never contact the live service or contain a real credential. Symfony’s MockHttpClient makes the transport deterministic and lets the test inspect the outgoing request.
<?php
namespace App\Tests\Technology;
use App\Technology\DetectorException;
use App\Technology\TechnologyDetectorClient;
use PHPUnit\Framework\TestCase;
use Psr\Log\NullLogger;
use Symfony\Component\HttpClient\MockHttpClient;
use Symfony\Component\HttpClient\Response\MockResponse;
final class TechnologyDetectorClientTest extends TestCase
{
public function testItMapsAValidDetection(): void
{
$response = new MockResponse(json_encode([
'detections' => [[
'technology' => 'Example CMS',
'confidence' => 95,
'versions' => ['4.2'],
'evidence' => ['header' => 'example'],
]],
'redirects' => [],
], JSON_THROW_ON_ERROR));
$mock = new MockHttpClient(function (string $method, string $url, array $options) use ($response) {
self::assertSame('POST', $method);
self::assertSame(['url' => 'https://example.com'], $options['json']);
self::assertSame('test-token', $options['auth_bearer']);
return $response;
});
$client = new TechnologyDetectorClient(
$mock,
new NullLogger(),
'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies',
'test-token',
);
$report = $client->detect('https://example.com');
self::assertStringContainsString('Example CMS 4.2', $report->summary);
self::assertSame(95, $report->detections[0]['confidence']);
}
public function testItClassifiesAuthenticationFailureWithoutRetryAdvice(): void
{
$client = new TechnologyDetectorClient(
new MockHttpClient(new MockResponse('', ['http_code' => 401])),
new NullLogger(),
'https://example.invalid/detect',
'test-token',
);
try {
$client->detect('https://example.com');
self::fail('Expected DetectorException.');
} catch (DetectorException $e) {
self::assertSame('authentication', $e->kind);
self::assertFalse($e->retryable);
}
}
}Add handler tests for a deleted lead, a changed URL, successful persistence, and a classified failure. Those cases protect the asynchronous behavior that a client-only test cannot exercise.
Deploy and operate the worker
Deploy the migration, provide the real environment variables through the platform, and run Messenger under a process supervisor:
php bin/console doctrine:migrations:migrate --no-interaction
php bin/console messenger:setup-transports
php bin/console messenger:consume async \
--time-limit=3600 \
--memory-limit=256M \
--failure-limit=5Restart workers after every deployment so they load the new code and configuration. Alert on repeated authentication failures, sustained rate_limited results, malformed responses, queue depth, and message age. A sudden authentication failure often means the service token was regenerated but not replaced in every environment.
Common failures are usually straightforward: 401 or 403 indicates token configuration or plan access; 400 or 422 points to an invalid URL; 429 means the application should slow down or revisit its plan; repeated transport failures warrant checking DNS and outbound HTTPS access. A successful response with no detections is not an application error and should remain a readable “no supported technologies detected” result.
Final verification checklist
- The service plan is active and the service-scoped token comes from the documentation’s Service token panel.
- No real token appears in source control, fixtures, application logs, URLs, or screenshots.
- The application sends
POSTJSON with exactly the lead’surlto the documented endpoint. - Timeouts and retries are bounded, and authentication or validation failures are not retried.
- Stale queue messages cannot overwrite results after a lead URL changes.
- Detections, confidence, evidence, versions, and redirects are validated before persistence.
- The CRM shows queued, complete, and failed states instead of hiding asynchronous work.
- Tests use
MockHttpClient, and production workers are supervised and restarted on deployment.
The valuable result is not merely an API response stored in a JSON column. It is a dependable piece of CRM context: readable at a glance, traceable to evidence, refreshed intentionally, and honest about failure. That is the difference between an impressive demo integration and a feature a small agency can safely use every day.
