Native PHP 8.3: Add AI-Detected Tech Stacks to CRM Leads Automatically
A lead arrives with a polished website and a vague request: “We need help modernizing our platform.” Before anyone schedules a discovery call, the agency wants useful context. Is the site running WordPress? Does it expose React, an analytics platform, a CDN, or a particular server stack?
This tutorial builds that enrichment step for a small CRM using Native PHP 8.3, cURL, PDO, and the Website Technology Detector API. A background command finds leads awaiting analysis, submits each public website, converts the confidence-scored detections into a readable summary, and records structured success or failure states.
The result is intentionally modest: no framework, message broker, or distributed workflow engine. The production discipline comes from a narrow API boundary, defensive response mapping, bounded retries, deterministic tests, safe credential handling, and observable failure modes.
Get access and copy the service-scoped token
This service requires a token. Complete the following onboarding flow before writing integration code:
- Register at https://ai.mihajlo.mk/register, or use https://ai.mihajlo.mk/login if you already have an account.
- Open the Website Technology Detector service page.
- Choose an available Free, Plus, or Pro plan and complete its activation.
- Open the official service documentation.
- Find the Service token panel and copy the service-scoped token shown there.
Regenerating the service token revokes the previously active token. Treat rotation as a deployment change: update every environment using the old value, deploy or restart the relevant processes, verify the new credential, and only then consider the rotation complete.
Confirm the endpoint with a minimal request
The exact operation is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. It accepts JSON containing url. Authentication may use a Bearer token, an X-API-Token header, or a token query parameter. The implementation below uses Bearer authentication because query parameters commonly appear in access logs and monitoring systems.
export DETECTOR_TEST_TOKEN='YOUR_SERVICE_TOKEN'
curl --request POST \
--url 'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies' \
--header "Authorization: Bearer ${DETECTOR_TEST_TOKEN}" \
--header 'Content-Type: application/json' \
--data '{"url":"https://example.com"}'
unset DETECTOR_TEST_TOKENA successful call should return JSON containing technology detections and their supporting information. Do not yet build business logic around a copied sample response. The application boundary should validate the live contract defensively because missing, null, or malformed fields must not corrupt a lead record.
Store configuration outside source control
For local development, create .env.local and exclude it from Git. In production, inject the same names through the process manager, container platform, or secret store. Environment values take precedence over the local file in the configuration loader used later.
WEBSITE_DETECTOR_TOKEN=YOUR_SERVICE_TOKEN
CRM_DSN="mysql:host=127.0.0.1;dbname=agency_crm;charset=utf8mb4"
CRM_DB_USER=crm_worker
CRM_DB_PASSWORD=YOUR_DATABASE_PASSWORD/.env.local
/vendor/
/.phpunit.cache/Choose a small, reliable architecture
Technology detection performs an external network request, so it should not extend the lead-creation response. Instead, lead creation or website editing sets technology_status to pending. A scheduled CLI command claims a small batch, calls the detector, maps the result, and updates each lead.
For a small agency CRM, a single scheduled worker is easier to operate than introducing a queue. It also has an honest limitation: only one worker should run at a time unless claiming is upgraded to use database-specific locking. A scheduler lock such as flock is sufficient for the initial design.
The feature can live in this compact structure:
crm/
├── bin/enrich-leads.php
├── src/
│ ├── Config.php
│ ├── Detector/
│ │ ├── CurlTransport.php
│ │ ├── DetectorClient.php
│ │ └── DetectionReport.php
│ └── LeadRepository.php
├── tests/DetectorClientTest.php
├── .env.local
├── .gitignore
└── composer.jsonStart with PHP 8.3, the cURL, JSON, and PDO extensions, plus Composer. PHPUnit is the only package required for this tutorial:
composer init --name=agency/crm-enrichment --require=php:^8.3 --no-interaction
composer require --dev phpunit/phpunit:^11.0
composer dump-autoloadConfigure Composer’s PSR-4 autoloading so Agency\Crm\ maps to src/ and Agency\Crm\Tests\ maps to tests/.
Add explicit lead enrichment state
The summary is useful to a person; status and error fields are useful to operations. Add columns appropriate to your database migration system rather than hiding failures in an empty summary:
ALTER TABLE leads
ADD COLUMN technology_summary TEXT NULL,
ADD COLUMN technology_status VARCHAR(20) NOT NULL DEFAULT 'pending',
ADD COLUMN technology_error VARCHAR(255) NULL,
ADD COLUMN technology_scanned_at TIMESTAMP NULL;Use pending, processing, complete, retry, and failed as application states. When a lead’s website changes, clear its old summary and return it to pending.
Build a strict HTTP boundary
The transport should know cURL, while the client should know authentication, retry policy, JSON, and the service contract. Separating them makes tests deterministic and prevents domain code from depending on cURL handles.
<?php
namespace Agency\Crm\Detector;
final readonly class HttpResponse
{
public function __construct(
public int $status,
public string $body,
public array $headers = [],
) {}
}
interface Transport
{
public function post(string $url, array $headers, string $body): HttpResponse;
}
final class TransportException extends \RuntimeException {}
final class CurlTransport implements Transport
{
public function post(string $url, array $headers, string $body): HttpResponse
{
$responseHeaders = [];
$handle = curl_init($url);
curl_setopt_array($handle, [
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $body,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_CONNECTTIMEOUT_MS => 2000,
CURLOPT_TIMEOUT_MS => 10000,
CURLOPT_HEADERFUNCTION => static function (
\CurlHandle $handle,
string $line
) use (&$responseHeaders): int {
$length = strlen($line);
$parts = explode(':', $line, 2);
if (count($parts) === 2) {
$responseHeaders[strtolower(trim($parts[0]))] = trim($parts[1]);
}
return $length;
},
]);
$bodyText = curl_exec($handle);
if ($bodyText === false) {
$message = curl_error($handle);
curl_close($handle);
throw new TransportException($message);
}
$status = (int) curl_getinfo($handle, CURLINFO_RESPONSE_CODE);
curl_close($handle);
return new HttpResponse($status, $bodyText, $responseHeaders);
}
}
final class DetectorException extends \RuntimeException
{
public function __construct(
string $message,
public readonly bool $retryable,
public readonly ?int $httpStatus = null,
) {
parent::__construct($message);
}
}
final readonly class DetectionReport
{
public function __construct(
public string $summary,
public array $technologies,
public array $redirects,
public ?string $finalUrl,
) {}
public static function fromPayload(array $payload): self
{
$root = isset($payload['data']) && is_array($payload['data'])
? $payload['data']
: $payload;
$items = $root['technologies'] ?? null;
if (!is_array($items)) {
throw new DetectorException(
'Response does not contain a valid technologies collection.',
false
);
}
$technologies = [];
foreach ($items as $item) {
if (!is_array($item) || !is_string($item['name'] ?? null)) {
continue;
}
$name = trim($item['name']);
if ($name === '') {
continue;
}
$version = is_string($item['version'] ?? null)
? trim($item['version'])
: null;
$confidence = is_int($item['confidence'] ?? null)
|| is_float($item['confidence'] ?? null)
? $item['confidence']
: null;
$evidence = [];
if (is_array($item['evidence'] ?? null)) {
foreach ($item['evidence'] as $value) {
if (is_string($value) && trim($value) !== '') {
$evidence[] = trim($value);
}
}
}
$technologies[] = [
'name' => $name,
'version' => $version !== '' ? $version : null,
'confidence' => $confidence,
'evidence' => $evidence,
];
}
usort(
$technologies,
static fn(array $a, array $b): int => strcmp($a['name'], $b['name'])
);
$parts = array_map(
static function (array $technology): string {
$text = $technology['name'];
if ($technology['version'] !== null) {
$text .= ' ' . $technology['version'];
}
if ($technology['confidence'] !== null) {
$text .= ' (confidence '
. (string) $technology['confidence'] . ')';
}
return $text;
},
$technologies
);
$redirects = is_array($root['redirects'] ?? null)
? $root['redirects']
: [];
$finalUrl = is_string($root['final_url'] ?? null)
? $root['final_url']
: null;
return new self(
$parts === [] ? 'No technologies detected.' : implode(', ', $parts),
$technologies,
$redirects,
$finalUrl,
);
}
}The mapper accepts an optional data envelope, but it never fabricates missing detections, versions, confidence, evidence, or redirects. Malformed technology entries are ignored; a missing technology collection is a schema failure. Evidence remains available in the domain result even though the concise CRM summary omits it.
Retry only failures that may improve
Network failures, HTTP 429 responses, and server-side 5xx responses may be transient. Authentication and validation failures are not. The client caps attempts at three, honors a numeric Retry-After value within a safe bound, and otherwise applies short exponential backoff with jitter.
<?php
namespace Agency\Crm\Detector;
final class DetectorClient
{
private const ENDPOINT =
'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies';
public function __construct(
private readonly Transport $transport,
private readonly string $token,
private readonly \Closure $sleep,
) {}
public function detect(string $url): DetectionReport
{
$json = json_encode(['url' => $url], JSON_THROW_ON_ERROR);
for ($attempt = 1; $attempt <= 3; $attempt++) {
try {
$response = $this->transport->post(
self::ENDPOINT,
[
'Authorization: Bearer ' . $this->token,
'Content-Type: application/json',
'Accept: application/json',
],
$json
);
} catch (TransportException $exception) {
if ($attempt === 3) {
throw new DetectorException(
'Detector network request failed.',
true
);
}
($this->sleep)($this->backoffMilliseconds($attempt, null));
continue;
}
if ($response->status >= 200 && $response->status < 300) {
try {
$payload = json_decode(
$response->body,
true,
512,
JSON_THROW_ON_ERROR
);
} catch (\JsonException) {
throw new DetectorException(
'Detector returned invalid JSON.',
false,
$response->status
);
}
if (!is_array($payload)) {
throw new DetectorException(
'Detector returned an unexpected JSON value.',
false,
$response->status
);
}
return DetectionReport::fromPayload($payload);
}
$transient = $response->status === 429
|| $response->status >= 500;
if (!$transient || $attempt === 3) {
throw new DetectorException(
'Detector request failed with HTTP ' . $response->status,
$transient,
$response->status
);
}
($this->sleep)(
$this->backoffMilliseconds(
$attempt,
$response->headers['retry-after'] ?? null
)
);
}
throw new \LogicException('Retry loop exited unexpectedly.');
}
private function backoffMilliseconds(
int $attempt,
?string $retryAfter
): int {
if ($retryAfter !== null && ctype_digit($retryAfter)) {
return min((int) $retryAfter, 30) * 1000;
}
return min(4000, 250 * (2 ** ($attempt - 1)))
+ random_int(0, 100);
}
}Run enrichment as a scheduled command
The command below shows the essential orchestration. Its repository is responsible for atomically changing a pending row to processing; if that update affects no rows, another process already claimed it. Keep transaction duration short and never hold a database transaction open during the API call.
<?php
declare(strict_types=1);
require dirname(__DIR__) . '/vendor/autoload.php';
use Agency\Crm\Detector\CurlTransport;
use Agency\Crm\Detector\DetectorClient;
use Agency\Crm\Detector\DetectorException;
$local = is_file(dirname(__DIR__) . '/.env.local')
? parse_ini_file(dirname(__DIR__) . '/.env.local', false, INI_SCANNER_RAW)
: [];
$env = static fn(string $key): ?string =>
getenv($key) !== false ? getenv($key) : ($local[$key] ?? null);
$token = $env('WEBSITE_DETECTOR_TOKEN');
if (!is_string($token) || trim($token) === '') {
throw new RuntimeException('WEBSITE_DETECTOR_TOKEN is not configured.');
}
$pdo = new PDO(
(string) $env('CRM_DSN'),
(string) $env('CRM_DB_USER'),
(string) $env('CRM_DB_PASSWORD'),
[PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]
);
$client = new DetectorClient(
new CurlTransport(),
trim($token),
static fn(int $milliseconds) => usleep($milliseconds * 1000)
);
$rows = $pdo->query(
"SELECT id, website_url
FROM leads
WHERE technology_status IN ('pending', 'retry')
AND website_url IS NOT NULL
ORDER BY id
LIMIT 20"
)->fetchAll(PDO::FETCH_ASSOC);
foreach ($rows as $row) {
$claim = $pdo->prepare(
"UPDATE leads
SET technology_status = 'processing', technology_error = NULL
WHERE id = ? AND technology_status IN ('pending', 'retry')"
);
$claim->execute([$row['id']]);
if ($claim->rowCount() !== 1) {
continue;
}
try {
$report = $client->detect($row['website_url']);
$update = $pdo->prepare(
"UPDATE leads
SET technology_summary = ?,
technology_status = 'complete',
technology_error = NULL,
technology_scanned_at = CURRENT_TIMESTAMP
WHERE id = ?"
);
$update->execute([$report->summary, $row['id']]);
error_log(json_encode([
'event' => 'lead_technology_detection_complete',
'lead_id' => $row['id'],
'technology_count' => count($report->technologies),
], JSON_THROW_ON_ERROR));
} catch (DetectorException $exception) {
$status = $exception->retryable ? 'retry' : 'failed';
$update = $pdo->prepare(
"UPDATE leads
SET technology_status = ?, technology_error = ?
WHERE id = ?"
);
$update->execute([
$status,
substr($exception->getMessage(), 0, 255),
$row['id'],
]);
error_log(json_encode([
'event' => 'lead_technology_detection_failed',
'lead_id' => $row['id'],
'retryable' => $exception->retryable,
'http_status' => $exception->httpStatus,
], JSON_THROW_ON_ERROR));
}
}Validate website input before it reaches this command. Accept only absolute http or https URLs with a non-empty host, impose a sensible length limit, and reject credentials embedded in URLs. The detector is intended for public websites; do not use it as a mechanism for probing private or unauthorized systems.
Test without making external requests
A fake transport makes status codes, headers, retries, and payloads reproducible. It also proves that authentication failures are not retried and that the readable summary is constructed at the application boundary.
<?php
namespace Agency\Crm\Tests;
use Agency\Crm\Detector\DetectorClient;
use Agency\Crm\Detector\DetectorException;
use Agency\Crm\Detector\HttpResponse;
use Agency\Crm\Detector\Transport;
use PHPUnit\Framework\TestCase;
final class FakeTransport implements Transport
{
public int $calls = 0;
public function __construct(private array $responses) {}
public function post(string $url, array $headers, string $body): HttpResponse
{
return $this->responses[$this->calls++];
}
}
final class DetectorClientTest extends TestCase
{
public function testItRetriesRateLimitAndMapsTechnologies(): void
{
$transport = new FakeTransport([
new HttpResponse(429, '{}', ['retry-after' => '1']),
new HttpResponse(200, json_encode([
'technologies' => [
[
'name' => 'Example CMS',
'version' => '4.2',
'confidence' => 0.96,
'evidence' => ['generator metadata'],
],
],
'redirects' => [],
'final_url' => 'https://example.com/',
], JSON_THROW_ON_ERROR)),
]);
$delays = [];
$client = new DetectorClient(
$transport,
'test-token',
static function (int $milliseconds) use (&$delays): void {
$delays[] = $milliseconds;
}
);
$report = $client->detect('https://example.com');
self::assertSame(2, $transport->calls);
self::assertSame([1000], $delays);
self::assertSame(
'Example CMS 4.2 (confidence 0.96)',
$report->summary
);
}
public function testItDoesNotRetryAuthenticationFailure(): void
{
$transport = new FakeTransport([
new HttpResponse(401, '{"message":"unauthorized"}'),
]);
$client = new DetectorClient(
$transport,
'invalid-token',
static function (int $milliseconds): void {}
);
try {
$client->detect('https://example.com');
self::fail('Expected DetectorException.');
} catch (DetectorException $exception) {
self::assertFalse($exception->retryable);
self::assertSame(401, $exception->httpStatus);
self::assertSame(1, $transport->calls);
}
}
}vendor/bin/phpunit testsDeploy, observe, and diagnose
Run the worker under the same PHP version and extensions used during testing. A cron entry can execute it every few minutes, but wrap it with a non-blocking lock so two invocations do not overlap:
*/5 * * * * flock -n /var/run/crm-enrichment.lock /usr/bin/php /var/www/crm/bin/enrich-leads.phpUse a lock path writable by the worker account, apply least-privilege permissions to the database user, and ensure the runtime can read the injected secret without making it visible to the web server’s document root. Never log authorization headers, tokens, complete API bodies, or query-string credentials.
Track counts of completed, retrying, and permanently failed leads, plus request duration and HTTP status classes. Alert on sustained authentication failures, schema failures, or a growing pending backlog. The structured logs deliberately include a lead identifier and outcome, but not the token or full website content.
Common failures
- HTTP 401 or 403: verify the service-scoped token and confirm that a regenerated token was deployed everywhere. Do not retry automatically.
- HTTP 400 or 422: inspect URL validation and the JSON body. These are request problems, not transient outages.
- HTTP 429: respect
Retry-After, keep batches small, and review plan capacity. Do not create unbounded retries. - HTTP 5xx or network timeout: retry briefly with backoff, then leave the lead in
retryfor a later scheduled run. - Schema failure after HTTP 2xx: retain the lead as failed, record a sanitized error, and compare the boundary mapper with the official documentation before changing it.
- Rows stuck in processing: add a maintenance rule that returns sufficiently old processing rows to retry after confirming no worker is still active.
Final verification checklist
- The activated plan and service token belong to the Website Technology Detector service.
- The client sends
POSTto the exact/v1/detect-technologiesendpoint with a JSONurl. - The token comes from environment-backed configuration and never enters source control or logs.
- Connection and total timeouts are bounded.
- Only network failures, 429 responses, and 5xx responses are retried.
- Detections, confidence, versions, evidence, and redirects are validated before entering CRM state.
- Lead updates distinguish complete, retryable, and permanent failures.
- PHPUnit tests run without accessing the external service.
- The scheduler prevents overlapping workers and exposes actionable logs.
- Changing a lead’s website schedules a fresh detection.
A technology summary is a small CRM field, but producing it reliably exercises the same judgment as a much larger integration. Keep the remote service behind a strict boundary, preserve uncertainty instead of inventing data, and make every failure visible. Then the enrichment becomes what a good automation should be: quiet when healthy, clear when broken, and genuinely useful when a person opens the lead.
