Tutorials

Laravel CRM: Enrich Leads with Automated Website Technology Summaries

Laravel CRM: Enrich Leads with Automated Website Technology Summaries

A lead’s website can tell an agency where a useful conversation might begin: a CMS upgrade, an ecommerce integration, or a frontend maintenance problem. That context becomes valuable when it appears beside the lead, before someone opens a dozen browser tabs.

We will build a small Laravel CRM backend that stores leads, enriches their websites through the Website Technology Detector API, and saves a readable technology report. Artisan provides creation and inspection; a scheduled command handles enrichment independently of an interactive request.

Get access and copy the service token

  1. Register an account or sign in.

  2. Open the Website Technology Detector service page, choose an available Free, Plus, or Pro plan, and complete activation.

  3. Open the official documentation. Find the Service token panel and copy its service-scoped token.

This service requires a token. Regenerating it revokes the previous active token, so coordinate rotation with deployment. Authentication supports Bearer tokens, an X-API-Token header, or a token query parameter. We use Bearer authentication to keep credentials out of URLs.

The exact operation is POST https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies. This minimal request sends the required JSON property:

curl --request POST \
  'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies' \
  --header 'Authorization: Bearer YOUR_SERVICE_TOKEN' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{"url":"https://example.com"}'

Replace the placeholder locally without saving a real token in shell history or shared output. Inspect the documented response before treating any particular field as a business rule.

Create the Laravel project

Use PHP 8.3+, Composer, and SQLite support. This implementation targets Laravel 12 and uses its HTTP client, database cache locks, scheduler, and testing facilities.

composer create-project laravel/laravel agency-crm "^12.0"
cd agency-crm
php artisan make:model Lead -m
php artisan make:command EnrichLeads

Keep the skeleton’s SQLite database and cache-table migrations. Add these values to the project’s .env, which must remain outside version control:

DB_CONNECTION=sqlite
CACHE_STORE=database
WEBSITE_TECHNOLOGY_TOKEN=YOUR_SERVICE_TOKEN

Add this entry inside the array returned by config/services.php:

'website_technology' => [
    'token' => env('WEBSITE_TECHNOLOGY_TOKEN'),
],

The files below form the complete integration: a lead model and migration, an API boundary, a failure type, an Artisan command, and feature tests. Processing ten due leads per invocation keeps the architecture modest. Background scheduling avoids making a salesperson wait for a remote website inspection; a separate queue is unnecessary for this workload.

Persist reports and explicit processing states

Replace the generated migration’s up() method with this implementation. Keep its generated down(), which drops leads.

public function up(): void
{
    Schema::create('leads', function (Blueprint $table) {
        $table->id();
        $table->string('website_url', 2048);
        $table->string('technology_state')->default('pending');
        $table->unsignedInteger('technology_attempts')->default(0);
        $table->timestamp('technology_next_at')->nullable();
        $table->text('technology_summary')->nullable();
        $table->json('technology_report')->nullable();
        $table->string('technology_error')->nullable();
        $table->timestamps();
        $table->index(['technology_state', 'technology_next_at']);
    });
}

Use this model in app/Models/Lead.php:

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class Lead extends Model
{
    protected $fillable = ['website_url'];

    protected function casts(): array
    {
        return [
            'technology_report' => 'array',
            'technology_next_at' => 'datetime',
        ];
    }
}

A received report means the API returned structured JSON. It does not mean every technology is certain, that a missing version is known, or that an empty report proves a website uses no detectable technologies.

Build a defensive API boundary

The supplied contract identifies confidence, evidence, versions, and redirect information without specifying their JSON paths. Guessing paths would produce a fragile integration. Our domain mapper instead renders the actual response tree into labeled lines, preserving those details wherever they occur. It keeps the original structured report for inspection and never converts confidence numbers into unsupported percentages.

Create app/Services/DetectionFailure.php:

<?php

namespace App\Services;

final class DetectionFailure extends \RuntimeException
{
    public function __construct(
        string $reason,
        public readonly ?int $retryAfter = null,
    ) {
        parent::__construct($reason);
    }
}

Create app/Services/TechnologyDetector.php:

<?php

namespace App\Services;

use Illuminate\Http\Client\ConnectionException;
use Illuminate\Support\Facades\Http;

final class TechnologyDetector
{
    public function detect(string $url): array
    {
        try {
            $response = Http::withToken(
                config('services.website_technology.token')
            )->acceptJson()->connectTimeout(5)->timeout(25)
                ->withOptions(['allow_redirects' => false])
                ->post(
                    'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies',
                    ['url' => $url]
                );
        } catch (ConnectionException) {
            throw new DetectionFailure('connection_failure', 60);
        }

        $status = $response->status();

        if ($status === 429) {
            $header = $response->header('Retry-After');
            $seconds = ctype_digit($header)
                ? (float) $header
                : max(0, (strtotime($header) ?: time()) - time());

            if ($seconds > 86400) {
                throw new DetectionFailure('quota_wait_exceeds_one_day');
            }

            throw new DetectionFailure(
                'rate_limited',
                max(60, (int) $seconds)
            );
        }

        if ($status === 408 || $response->serverError()) {
            throw new DetectionFailure("http_{$status}", 60);
        }

        if (! $response->successful()) {
            throw new DetectionFailure("http_{$status}");
        }

        if (strlen($response->body()) > 262144) {
            throw new DetectionFailure('report_too_large');
        }

        try {
            $report = json_decode(
                $response->body(), true, 64, JSON_THROW_ON_ERROR
            );
        } catch (\JsonException) {
            throw new DetectionFailure('invalid_json');
        }

        if (! is_array($report)) {
            throw new DetectionFailure('unexpected_report_shape');
        }

        $lines = [];
        $walk = function (array $node, string $path = 'Report')
            use (&$walk, &$lines): void {
                foreach ($node as $key => $value) {
                    if (count($lines) >= 200) {
                        return;
                    }

                    $label = is_int($key)
                        ? 'Item '.($key + 1)
                        : ucwords(str_replace(['_', '-'], ' ', $key));
                    $label = preg_replace('/[\x00-\x1F\x7F]/', ' ', $label);
                    $name = "{$path} / {$label}";

                    if (is_array($value) && $value !== []) {
                        $walk($value, $name);
                    } else {
                        $lines[] = $name.': '.json_encode(
                            $value,
                            JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE
                        );
                    }
                }
            };

        $walk($report);

        if (count($lines) === 200) {
            $lines[] = 'Display limit reached; inspect the stored report.';
        }

        return [
            'report' => $report,
            'summary' => implode("\n", $lines)
                ?: 'No report details returned; technology presence is unknown.',
        ];
    }
}

The returned report and summary keys belong to our application, not the external API. This conservative mapping remains useful when response fields evolve. A more selective sales-oriented summary should follow verified documentation and captured, sanitized contract fixtures.

HTTP redirects are disabled for the API request itself. Redirect information about the inspected website remains part of the report. The response-size check limits stored data after download; it is not a streaming memory limit.

Process leads with bounded retries

Replace app/Console/Commands/EnrichLeads.php with the following. Missing credentials fail before consuming attempts. Each lead gets at most five attempts; authentication and validation failures stop immediately.

<?php

namespace App\Console\Commands;

use App\Models\Lead;
use App\Services\DetectionFailure;
use App\Services\TechnologyDetector;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Validator;

class EnrichLeads extends Command
{
    protected $signature = 'crm:tech {--add=} {--show=}';
    protected $description = 'Create, inspect, or enrich CRM leads';

    public function handle(TechnologyDetector $detector): int
    {
        if ($url = $this->option('add')) {
            Validator::make(['url' => $url], [
                'url' => ['required', 'url:http,https', 'max:2048'],
            ])->validate();

            $parts = parse_url($url);
            if (isset($parts['user']) || isset($parts['pass'])) {
                $this->error('URLs must not contain credentials.');
                return self::FAILURE;
            }

            $this->info((string) Lead::create(['website_url' => $url])->id);
            return self::SUCCESS;
        }

        if ($id = $this->option('show')) {
            $lead = Lead::findOrFail($id);
            $this->line($lead->technology_state);
            $this->line($lead->technology_summary
                ?? $lead->technology_error ?? 'Awaiting enrichment');
            return self::SUCCESS;
        }

        if (! config('services.website_technology.token')) {
            $this->error('Missing service token.');
            return self::FAILURE;
        }

        $ids = Lead::where('technology_state', 'pending')
            ->where(fn ($q) => $q->whereNull('technology_next_at')
                ->orWhere('technology_next_at', '<=', now()))
            ->orderBy('id')->limit(10)->pluck('id');

        foreach ($ids as $id) {
            Cache::lock("lead-technology:{$id}", 120)->get(
                function () use ($id, $detector): void {
                    $lead = Lead::findOrFail($id);
                    if ($lead->technology_state !== 'pending'
                        || $lead->technology_next_at?->isFuture()) {
                        return;
                    }

                    $attempt = $lead->technology_attempts + 1;
                    $lead->forceFill([
                        'technology_attempts' => $attempt,
                    ])->save();

                    try {
                        $result = $detector->detect($lead->website_url);
                        $changes = [
                            'technology_state' => 'received',
                            'technology_summary' => $result['summary'],
                            'technology_report' => $result['report'],
                            'technology_error' => null,
                            'technology_next_at' => null,
                        ];
                    } catch (DetectionFailure $e) {
                        $retry = $e->retryAfter !== null && $attempt < 5;
                        $delay = max(
                            $e->retryAfter ?? 0,
                            min(3600, 60 * 2 ** min($attempt - 1, 6))
                                + random_int(0, 30)
                        );
                        $changes = [
                            'technology_state' => $retry ? 'pending' : 'failed',
                            'technology_error' => $e->getMessage(),
                            'technology_next_at' => $retry
                                ? now()->addSeconds($delay) : null,
                        ];
                    }

                    $lead->forceFill($changes)->save();
                    Log::info('lead_technology.result', [
                        'lead_id' => $id,
                        'attempt' => $attempt,
                        'state' => $lead->technology_state,
                        'reason' => $lead->technology_error,
                    ]);
                }
            );
        }

        return self::SUCCESS;
    }
}

Connection failures and eligible HTTP errors become persisted retry decisions, rather than sleeping processes. Numeric and HTTP-date Retry-After values are respected within a one-day retry horizon. Longer waits require operator review.

A crash after the remote call but before saving can repeat a detection. This design provides recoverable processing, not exactly-once requests. Check remaining quota before replaying failed leads.

Test behavior without calling the service

Create tests/Feature/LeadTechnologyTest.php. The deliberately arbitrary response tree tests structural preservation without pretending to document external field names.

<?php

namespace Tests\Feature;

use App\Models\Lead;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Support\Facades\Http;
use Tests\TestCase;

class LeadTechnologyTest extends TestCase
{
    use RefreshDatabase;

    public function test_report_is_saved_and_request_is_authenticated(): void
    {
        config(['services.website_technology.token' => 'test-placeholder']);
        Http::preventStrayRequests();
        Http::fake(['*' => Http::response([
            ['ExampleCMS', 0.8, ['HTML marker'], '1.2', 'https://example.com'],
        ])]);

        $lead = Lead::create(['website_url' => 'https://example.com']);
        $this->artisan('crm:tech')->assertSuccessful();

        $this->assertSame('received', $lead->fresh()->technology_state);
        $this->assertStringContainsString(
            'HTML marker', $lead->fresh()->technology_summary
        );

        Http::assertSent(fn ($request) =>
            $request->method() === 'POST'
            && $request->url() ===
                'https://ai.mihajlo.mk/api/website-technology-detector/v1/detect-technologies'
            && $request->hasHeader('Authorization', 'Bearer test-placeholder')
            && $request['url'] === 'https://example.com'
        );
    }

    public function test_rate_limit_defers_and_authentication_failure_stops(): void
    {
        config(['services.website_technology.token' => 'test-placeholder']);
        Http::fakeSequence()
            ->push([], 429, ['Retry-After' => '120'])
            ->push([], 401);

        $lead = Lead::create(['website_url' => 'https://example.com']);
        $this->artisan('crm:tech')->assertSuccessful();
        $this->assertTrue(
            $lead->fresh()->technology_next_at->gte(now()->addSeconds(120))
        );

        $this->artisan('crm:tech')->assertSuccessful();
        Http::assertSentCount(1);

        $this->travel(3)->minutes();
        $this->artisan('crm:tech')->assertSuccessful();
        $this->assertSame('failed', $lead->fresh()->technology_state);
        $this->assertSame('http_401', $lead->fresh()->technology_error);
    }
}

Schedule, secure, and verify

Add this to routes/console.php:

\Illuminate\Support\Facades\Schedule::command('crm:tech')
    ->everyMinute()
    ->withoutOverlapping(10);

Deploy with one scheduler, persistent SQLite storage, and the shared database cache. Run these checks before enabling the scheduler:

php artisan migrate --force
php artisan test
php artisan config:cache
php artisan crm:tech --add=https://example.com
php artisan crm:tech
php artisan crm:tech --show=1

Install the usual once-per-minute scheduler entry, replacing the project path:

* * * * * cd /absolute/path/agency-crm && php artisan schedule:run

Only submit public websites you are authorized to inspect. URL validation checks syntax; it does not establish that DNS targets or redirects are public. The detector must enforce its own outbound network restrictions. Laravel contacts only the fixed API endpoint.

Keep reports access-controlled alongside lead data. If you later display summaries in Blade, use escaped interpolation. Exclude authorization headers and response bodies from HTTP tracing. Monitor pending-lead age, failed states, and rate_limited events; a successful scheduler exit does not imply every lead succeeded.

For persistent 401 or 403 responses, check activation and token rotation, then rebuild cached configuration. For quota failures, review the plan before retrying. Invalid JSON or unexpected shapes require inspecting a sanitized response against the official documentation.

  • Confirm a real lead reaches received and its report includes available evidence, confidence, versions, and redirects.
  • Verify failed authentication stops retries and rate limits postpone work.
  • Confirm the scheduler runs and pending age decreases.
  • Check that credentials and raw reports stay out of logs.

The useful result is a better first conversation. Keep the evidence attached, preserve uncertainty, and let the technology summary give the agency a reason to ask a sharper question.

Blog author portrait

Mihajlo

I’m Mihajlo — a developer driven by curiosity, discipline, and the constant urge to create something meaningful. I share insights, tutorials, and free services to help others simplify their work and grow in the ever-evolving world of software and AI.