Symfony: Uncover Lead Website Tech Stacks for Smarter Agency Outreach
A lead’s website can reveal whether an agency is a natural fit before anyone writes the first outreach email. A Symfony specialist needs a different conversation with a prospect running Symfony than with one using WordPress, Shopify, or an unidentified custom stack.
This tutorial adds that context to a small Symfony-based agency CRM. A scheduled command submits each lead’s public website to the Website Technology Detector API, maps the confidence-scored results at a defensive application boundary, and stores a concise summary such as Symfony 7 (confidence 98); PHP 8.3 (confidence 92).
The implementation favors a boring, reliable batch command over a synchronous request from the CRM interface. Detection can involve redirects and remote network work; none of that should delay the page where someone edits a lead.
Get access before writing integration code
- Create an account at https://ai.mihajlo.mk/register, or use https://ai.mihajlo.mk/login if you already have one.
- Open the Website Technology Detector service page.
- Choose the available Free, Plus, or Pro plan and complete its activation.
- Open the official service documentation.
- Find the Service token panel and copy the service-scoped token shown there.
This service requires authentication. It accepts a Bearer token, an X-API-Token header, or a token query parameter. The application below uses the Bearer form because it keeps credentials out of URLs, browser history, proxy logs, and routine access logs.
Regenerating the service token revokes the previously active token. Treat rotation as a deployment change: install the new value everywhere the command runs, verify it, and expect processes still using the old value to receive an authentication failure.
Confirm the exact endpoint
The required call is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. Its JSON request contains one field, url. Before changing the CRM, make a minimal request from a secure terminal:
curl --request POST \
--url 'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies' \
--header 'Authorization: Bearer YOUR_SERVICE_TOKEN' \
--header 'Content-Type: application/json' \
--data '{"url":"https://example.com"}'Never paste a real token into shell history shared with other users. For repeated testing, load it from a protected environment variable and reference that variable from the command.
Store the token in Symfony configuration
Keep the placeholder in the committed .env file and put the real secret in an uncommitted .env.local during local development. In production, inject it through the hosting platform’s secret or environment configuration.
# .env
WEBSITE_TECHNOLOGY_DETECTOR_TOKEN=YOUR_SERVICE_TOKEN# config/services.yaml
services:
_defaults:
autowire: true
autoconfigure: true
App\Integration\TechnologyDetector:
arguments:
$serviceToken: '%env(WEBSITE_TECHNOLOGY_DETECTOR_TOKEN)%'The project needs Symfony’s first-party HTTP client. If it is not already installed, add it with composer require symfony/http-client. Tests use the package’s MockHttpClient, so they make no external requests.
Shape the feature around a narrow API boundary
The integration has three responsibilities:
TechnologyDetectorowns authentication, timeouts, retries, status handling, and JSON decoding.TechnologyReportconverts uncertain external JSON into stable domain data.RefreshLeadTechnologiesCommandscans leads and persists summaries without involving a web request.
The detector may return detections, versions, evidence, confidence scores, and redirect information. Those concepts belong in the domain model, but external JSON should not leak throughout the CRM. Field placement may vary, so the mapper searches nested objects and accepts only appropriately typed values. Preserve the raw successful payload for diagnosis, subject to a sensible database size limit and retention policy.
Map detections defensively
Create src/Integration/TechnologyReport.php. This mapper does not assume that detections live under a particular response envelope. A candidate must have a non-empty name and a numeric confidence value. Versions and evidence remain optional.
<?php
namespace App\Integration;
final readonly class TechnologyReport
{
public function __construct(
public array $detections,
public array $redirectInformation,
public array $raw,
) {}
public static function fromPayload(array $payload): self
{
$detections = [];
$redirects = [];
$walk = function (mixed $node, ?string $key = null) use (&$walk, &$detections, &$redirects): void {
if (!is_array($node)) {
return;
}
if ($key !== null && str_contains(strtolower($key), 'redirect')) {
$redirects[] = $node;
}
$name = $node['name'] ?? $node['technology'] ?? null;
$confidence = $node['confidence'] ?? null;
if (is_string($name) && trim($name) !== '' && is_numeric($confidence)) {
$versions = $node['versions'] ?? $node['version'] ?? [];
$evidence = $node['evidence'] ?? [];
$detections[] = [
'name' => trim($name),
'confidence' => (float) $confidence,
'versions' => self::scalarList($versions),
'evidence' => is_array($evidence) ? $evidence : [$evidence],
];
}
foreach ($node as $childKey => $child) {
$walk($child, is_string($childKey) ? $childKey : null);
}
};
$walk($payload);
$unique = [];
foreach ($detections as $detection) {
$identity = strtolower($detection['name'].'|'.implode(',', $detection['versions']));
$unique[$identity] = $detection;
}
usort(
$unique,
static fn (array $a, array $b): int => $b['confidence'] <=> $a['confidence'],
);
return new self(array_values($unique), $redirects, $payload);
}
public function summary(): string
{
if ($this->detections === []) {
return 'No technology detections returned';
}
return implode('; ', array_map(static function (array $detection): string {
$version = $detection['versions'][0] ?? null;
return sprintf(
'%s%s (confidence %s)',
$detection['name'],
$version !== null ? ' '.$version : '',
rtrim(rtrim(number_format($detection['confidence'], 2, '.', ''), '0'), '.'),
);
}, $this->detections));
}
private static function scalarList(mixed $value): array
{
$values = is_array($value) ? $value : [$value];
return array_values(array_map(
static fn (string|int|float $item): string => (string) $item,
array_filter($values, static fn (mixed $item): bool => is_scalar($item)),
));
}
}Notice that the summary calls the value a confidence score rather than adding a percent sign. That avoids silently asserting a scale that is not part of the supplied contract. Evidence remains available for audit or a future detail panel without cluttering the lead list.
Build the resilient HTTP client
Create a small exception carrying a machine-readable failure kind:
<?php
namespace App\Integration;
final class DetectionException extends \RuntimeException
{
public function __construct(
public readonly string $kind,
string $message,
public readonly ?int $statusCode = null,
) {
parent::__construct($message);
}
}Now create src/Integration/TechnologyDetector.php. Connection and total-duration limits prevent a slow dependency from occupying the command indefinitely. Only transport failures, HTTP 429, and server failures are retried. Authentication and validation errors are terminal because another identical call will not repair them.
<?php
namespace App\Integration;
use Psr\Log\LoggerInterface;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
use Symfony\Contracts\HttpClient\HttpClientInterface;
final readonly class TechnologyDetector
{
private const ENDPOINT =
'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies';
public function __construct(
private HttpClientInterface $httpClient,
private LoggerInterface $logger,
private string $serviceToken,
) {}
public function detect(string $url): TechnologyReport
{
$this->assertPublicHttpUrl($url);
$delays = [250_000, 750_000, 1_500_000];
for ($attempt = 0; $attempt < 3; ++$attempt) {
try {
$response = $this->httpClient->request('POST', self::ENDPOINT, [
'headers' => [
'Authorization' => 'Bearer '.$this->serviceToken,
'Accept' => 'application/json',
],
'json' => ['url' => $url],
'timeout' => 5.0,
'max_duration' => 20.0,
]);
$status = $response->getStatusCode();
if (($status === 429 || $status >= 500) && $attempt < 2) {
$response->cancel();
usleep($delays[$attempt]);
continue;
}
if ($status === 401 || $status === 403) {
throw new DetectionException('authentication', 'Service authentication failed.', $status);
}
if ($status === 400 || $status === 422) {
throw new DetectionException('validation', 'The service rejected the website URL.', $status);
}
if ($status === 429) {
throw new DetectionException('rate_limited', 'Service quota or rate limit reached.', $status);
}
if ($status < 200 || $status >= 300) {
throw new DetectionException('upstream', 'Technology detection failed upstream.', $status);
}
try {
$payload = json_decode(
$response->getContent(false),
true,
512,
JSON_THROW_ON_ERROR,
);
} catch (\JsonException $exception) {
throw new DetectionException('invalid_response', 'Service returned invalid JSON.');
}
if (!is_array($payload)) {
throw new DetectionException('invalid_response', 'Service returned an unexpected JSON value.');
}
return TechnologyReport::fromPayload($payload);
} catch (TransportExceptionInterface $exception) {
if ($attempt < 2) {
usleep($delays[$attempt]);
continue;
}
throw new DetectionException('transport', 'Technology service was unreachable.');
}
}
throw new DetectionException('transport', 'Technology service was unreachable.');
}
private function assertPublicHttpUrl(string $url): void
{
$parts = parse_url($url);
$scheme = strtolower((string) ($parts['scheme'] ?? ''));
$host = $parts['host'] ?? null;
if (!in_array($scheme, ['http', 'https'], true) || !is_string($host) || $host === '') {
throw new DetectionException('validation', 'A complete HTTP or HTTPS website URL is required.');
}
if (isset($parts['user']) || isset($parts['pass'])) {
throw new DetectionException('validation', 'Website URLs containing credentials are not accepted.');
}
}
}The log should record the lead identifier, failure kind, status code, attempt outcome, and duration. It should not record the Authorization header, token, complete response body, or URL query string. The example keeps logging at the command layer so it has the useful lead context.
Persist the CRM-facing state
Add these fields to the existing Doctrine Lead entity, then generate and review a migration with php bin/console make:migration and apply it with php bin/console doctrine:migrations:migrate:
#[ORM\Column(length: 2000, nullable: true)]
private ?string $technologySummary = null;
#[ORM\Column(length: 24, options: ['default' => 'pending'])]
private string $technologyScanState = 'pending';
#[ORM\Column(nullable: true)]
private ?\DateTimeImmutable $technologyScannedAt = null;
#[ORM\Column(length: 500, nullable: true)]
private ?string $technologyScanError = null;
#[ORM\Column(type: Types::JSON, nullable: true)]
private ?array $technologyScanPayload = null;
public function recordTechnologyReport(TechnologyReport $report): void
{
$this->technologySummary = mb_substr($report->summary(), 0, 2000);
$this->technologyScanState = 'complete';
$this->technologyScannedAt = new \DateTimeImmutable();
$this->technologyScanError = null;
$this->technologyScanPayload = $report->raw;
}
public function recordTechnologyFailure(DetectionException $exception): void
{
$this->technologyScanState = $exception->kind;
$this->technologyScannedAt = new \DateTimeImmutable();
$this->technologyScanError = mb_substr($exception->getMessage(), 0, 500);
}Retaining the previous summary on failure is deliberate. A temporary outage should not erase useful sales context. The state and timestamp tell the interface that the displayed result may be stale.
Refresh leads with a scheduled command
Create src/Command/RefreshLeadTechnologiesCommand.php. For a small CRM, sequential processing keeps quotas and failure behavior easy to understand. Flush periodically so the ORM does not retain every lead for the entire run.
<?php
namespace App\Command;
use App\Entity\Lead;
use App\Integration\DetectionException;
use App\Integration\TechnologyDetector;
use App\Repository\LeadRepository;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
#[AsCommand('app:refresh-lead-technologies')]
final class RefreshLeadTechnologiesCommand extends Command
{
public function __construct(
private readonly LeadRepository $leads,
private readonly TechnologyDetector $detector,
private readonly EntityManagerInterface $entityManager,
private readonly LoggerInterface $logger,
) {
parent::__construct();
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$query = $this->leads->createQueryBuilder('lead')
->andWhere('lead.websiteUrl IS NOT NULL')
->orderBy('lead.id', 'ASC')
->getQuery();
$processed = 0;
$failed = 0;
foreach ($query->toIterable() as $lead) {
assert($lead instanceof Lead);
try {
$lead->recordTechnologyReport(
$this->detector->detect($lead->getWebsiteUrl()),
);
} catch (DetectionException $exception) {
++$failed;
$lead->recordTechnologyFailure($exception);
$this->logger->warning('Lead technology detection failed.', [
'lead_id' => $lead->getId(),
'kind' => $exception->kind,
'status_code' => $exception->statusCode,
]);
}
if (++$processed % 20 === 0) {
$this->entityManager->flush();
$this->entityManager->clear();
}
}
$this->entityManager->flush();
$output->writeln(sprintf('Processed %d leads; %d failed.', $processed, $failed));
return $failed === 0 ? Command::SUCCESS : Command::FAILURE;
}
}Schedule php bin/console app:refresh-lead-technologies --env=prod at an interval appropriate to the plan and how frequently lead websites change. Configure the scheduler to prevent overlapping runs. For a larger database, query only stale or pending records and introduce a cursor-based batch limit before considering Messenger workers.
Test without spending quota
MockHttpClient verifies the outgoing request and supplies a deterministic response. The test below also confirms that the domain summary includes version and confidence data.
<?php
namespace App\Tests\Integration;
use App\Integration\TechnologyDetector;
use PHPUnit\Framework\TestCase;
use Psr\Log\NullLogger;
use Symfony\Component\HttpClient\MockHttpClient;
use Symfony\Component\HttpClient\Response\MockResponse;
final class TechnologyDetectorTest extends TestCase
{
public function testItMapsAValidDetectionResponse(): void
{
$client = new MockHttpClient(function (string $method, string $url, array $options): MockResponse {
self::assertSame('POST', $method);
self::assertSame(
'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies',
$url,
);
self::assertStringContainsString(
'Bearer test-token',
implode("\n", $options['headers']),
);
return new MockResponse(json_encode([
'detections' => [[
'name' => 'Symfony',
'confidence' => 98,
'versions' => ['7'],
'evidence' => ['deterministic test evidence'],
]],
'redirects' => [],
], JSON_THROW_ON_ERROR), ['http_code' => 200]);
});
$report = (new TechnologyDetector($client, new NullLogger(), 'test-token'))
->detect('https://example.com');
self::assertSame('Symfony 7 (confidence 98)', $report->summary());
self::assertCount(1, $report->detections);
}
public function testItDoesNotRetryAuthenticationFailure(): void
{
$requests = 0;
$client = new MockHttpClient(function () use (&$requests): MockResponse {
++$requests;
return new MockResponse('{"error":"unauthorized"}', ['http_code' => 401]);
});
try {
(new TechnologyDetector($client, new NullLogger(), 'invalid'))
->detect('https://example.com');
self::fail('Expected an authentication failure.');
} catch (\App\Integration\DetectionException $exception) {
self::assertSame('authentication', $exception->kind);
self::assertSame(1, $requests);
}
}
}The response fixture exercises the mapper’s documented semantic inputs; it is not a substitute for checking the current official response examples. Add captured, sanitized contract fixtures only after removing customer URLs, tokens, and unnecessary evidence.
Deploy and verify the complete path
Inject the production token, run the reviewed Doctrine migration, warm the Symfony cache, and execute one manual refresh before enabling the schedule. Alert on repeated authentication failures, sustained rate limiting, growing transport failures, and unusually long command duration. A 429 normally calls for a lower schedule frequency or plan review, not aggressive retries.
Common failures are usually straightforward: 401 or 403 means the token is missing, revoked, or attached incorrectly; 400 or 422 points to a malformed or unsupported URL; repeated 429 responses indicate exhausted quota or excessive concurrency; transport errors suggest DNS, TLS, firewall, or upstream availability trouble; and a successful response with no mapped detections warrants comparing the sanitized payload with the official documentation and updating only the boundary mapper.
Final verification checklist
- The token exists only in environment-backed configuration and never appears in logs or fixtures.
- The application sends a JSON
urlto the exact HTTPS endpoint with Bearer authentication. - Connection and total response time are bounded.
- Only transient transport, rate-limit, and server failures are retried.
- Detections, confidence, versions, evidence, and redirect information are mapped or retained defensively.
- A failed refresh preserves the last useful summary and records a structured state.
- Automated tests make no live service calls.
- The production scheduler prevents overlapping runs and respects plan limits.
- Each scanned lead displays a readable summary with a state and scan timestamp.
The valuable result is not a pile of fingerprinting JSON. It is one calm, readable line beside each lead: enough technical context to shape a relevant conversation, backed by evidence when someone needs to inspect it, and maintained by an integration that fails predictably instead of surprising the CRM’s users.
