Tutorials

Prefill CRM Leads: PHP Scrapes Websites for Company Data Instantly

Prefill CRM Leads: PHP Scrapes Websites for Company Data Instantly

A CRM lead often begins with a frustratingly small amount of information: a company website and nothing else. The salesperson can research the business manually, but that turns a quick lead capture into repetitive data entry. A better workflow accepts the website, extracts structured company and contact information, and returns a draft that the salesperson can review rather than reconstruct.

This tutorial builds that workflow as a small Native PHP 8.3 application. It exposes a JSON endpoint, calls the Website to Company data service through a dedicated API boundary, maps the response into a domain object, and stores a CRM lead draft in SQLite. The implementation includes bounded timeouts, selective retries, structured failures, redacted logs, and deterministic PHPUnit tests.

Get access and copy the service token

Register at https://ai.mihajlo.mk/register, or sign in through https://ai.mihajlo.mk/login if you already have an account.

Open the Website to Company data service page, choose an available Free, Plus, or Pro plan, and complete its activation. Then visit the official service documentation. Find the Service token panel and copy the service-scoped token shown there.

This service requires that token. Regenerating it revokes the previously active token, so coordinate rotation with deployment rather than regenerating it casually. Keep the value out of source control, logs, test fixtures, screenshots, and browser-delivered configuration.

Confirm the exact API contract

The integration sends a GET request to https://ai.mihajlo.mk/api/website-to-company-data/v1/extract. Authentication uses the token={serviceToken} query parameter, while the public company URL goes in the website query parameter.

Make one minimal request before building the application:

curl --get 'https://ai.mihajlo.mk/api/website-to-company-data/v1/extract' \
  --data-urlencode 'token=YOUR_SERVICE_TOKEN' \
  --data-urlencode 'website=https://example.com' \
  --header 'Accept: application/json'

A query-string token may appear in proxy access logs or diagnostics. Redact query parameters at every intermediary you control, and never paste a real request URL into an issue tracker.

For local development, place the credential in the project’s .env file:

WEBSITE_COMPANY_TOKEN=YOUR_SERVICE_TOKEN
CRM_DATABASE_DSN=sqlite:var/crm.sqlite

Add .env and the database file to .gitignore. In production, inject the same variables through the process manager or secrets platform instead of shipping an environment file with the release.

Shape the application around one narrow workflow

The browser or CRM screen submits only website to POST /leads/prefill. The controller validates the input, the API client performs enrichment, and a mapper accepts only the documented company, contact, email, phone, and people fields. The resulting draft is persisted and returned to the UI for review.

The API call remains synchronous because the salesperson is waiting for the form to be populated. That improves interaction simplicity, but it makes strict time limits important. A background queue would be preferable for bulk imports, not for this single-record workflow.

Use PHP 8.3 with the cURL, PDO SQLite, and JSON extensions, plus Composer. The runtime HTTP integration itself uses native cURL. The only runtime package is vlucas/phpdotenv:^5.6 for safe local configuration loading; phpunit/phpunit:^11.5 supplies the test runner.

{
  "name": "acme/crm-prefill",
  "require": {
    "php": "^8.3",
    "ext-curl": "*",
    "ext-json": "*",
    "ext-pdo": "*",
    "ext-pdo_sqlite": "*",
    "vlucas/phpdotenv": "^5.6"
  },
  "require-dev": {
    "phpunit/phpunit": "^11.5"
  },
  "autoload": {
    "psr-4": {
      "App\\": "src/"
    }
  },
  "autoload-dev": {
    "psr-4": {
      "Tests\\": "tests/"
    }
  }
}
crm-prefill/
├── .env
├── composer.json
├── public/index.php
├── bin/migrate.php
├── src/Http/Transport.php
├── src/Http/HttpResponse.php
├── src/Http/TransportException.php
├── src/Http/CurlTransport.php
├── src/CompanyData.php
├── src/CompanyDataException.php
├── src/Sleeper.php
├── src/NativeSleeper.php
├── src/WebsiteCompanyClient.php
├── src/LeadRepository.php
├── src/LeadPrefillController.php
└── tests/WebsiteCompanyClientTest.php

Build a replaceable native cURL boundary

The transport abstraction is deliberately small. Production gets cURL, while tests get an in-memory fake. This keeps network behavior out of domain mapping tests without introducing a general-purpose HTTP abstraction the project does not need.

<?php
// src/Http/Transport.php
namespace App\Http;

interface Transport
{
    public function get(string $url, array $query): HttpResponse;
}

// src/Http/HttpResponse.php
namespace App\Http;

final readonly class HttpResponse
{
    public function __construct(
        public int $status,
        public array $headers,
        public string $body,
    ) {}
}

// src/Http/TransportException.php
namespace App\Http;

final class TransportException extends \RuntimeException {}

// src/Http/CurlTransport.php
namespace App\Http;

final class CurlTransport implements Transport
{
    public function get(string $url, array $query): HttpResponse
    {
        $headers = [];
        $requestUrl = $url . '?' . http_build_query(
            $query,
            '',
            '&',
            PHP_QUERY_RFC3986
        );

        $handle = curl_init($requestUrl);
        curl_setopt_array($handle, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_CONNECTTIMEOUT_MS => 1500,
            CURLOPT_TIMEOUT_MS => 5000,
            CURLOPT_FOLLOWLOCATION => false,
            CURLOPT_SSL_VERIFYPEER => true,
            CURLOPT_SSL_VERIFYHOST => 2,
            CURLOPT_HTTPHEADER => ['Accept: application/json'],
            CURLOPT_USERAGENT => 'crm-prefill/1.0',
            CURLOPT_HEADERFUNCTION => static function ($curl, string $line) use (&$headers): int {
                $parts = explode(':', $line, 2);
                if (count($parts) === 2) {
                    $headers[strtolower(trim($parts[0]))] = trim($parts[1]);
                }
                return strlen($line);
            },
        ]);

        $body = curl_exec($handle);
        if ($body === false) {
            $message = curl_error($handle);
            curl_close($handle);
            throw new TransportException($message);
        }

        $status = (int) curl_getinfo($handle, CURLINFO_RESPONSE_CODE);
        curl_close($handle);

        return new HttpResponse($status, $headers, $body);
    }
}

Map uncertain JSON at the application boundary

External JSON must not leak unexamined into the CRM model. The service contract names five returned fields, but code should still reject incompatible shapes. Here, company and contact data may be an object, text value, or null; email and phone must be strings or null; people must be a list. No undocumented fallback field names are guessed.

<?php
// src/CompanyData.php
namespace App;

final readonly class CompanyData
{
    public function __construct(
        public array|string|null $company,
        public array|string|null $contact,
        public ?string $email,
        public ?string $phone,
        public array $people,
    ) {}

    public static function fromPayload(array $payload): self
    {
        $record = static function (string $key) use ($payload): array|string|null {
            $value = $payload[$key] ?? null;
            if ($value !== null && !is_array($value) && !is_string($value)) {
                throw new CompanyDataException('schema', "Invalid {$key} field");
            }
            return $value;
        };

        $text = static function (string $key) use ($payload): ?string {
            $value = $payload[$key] ?? null;
            if ($value !== null && !is_string($value)) {
                throw new CompanyDataException('schema', "Invalid {$key} field");
            }
            return $value;
        };

        $people = $payload['people'] ?? [];
        if (!is_array($people) || !array_is_list($people)) {
            throw new CompanyDataException('schema', 'Invalid people field');
        }

        return new self(
            $record('company'),
            $record('contact'),
            $text('email'),
            $text('phone'),
            $people,
        );
    }

    public function toArray(): array
    {
        return get_object_vars($this);
    }
}

// src/CompanyDataException.php
namespace App;

final class CompanyDataException extends \RuntimeException
{
    public function __construct(public readonly string $kind, string $message)
    {
        parent::__construct($message);
    }
}

// src/Sleeper.php
namespace App;

interface Sleeper
{
    public function pause(int $milliseconds): void;
}

// src/NativeSleeper.php
namespace App;

final class NativeSleeper implements Sleeper
{
    public function pause(int $milliseconds): void
    {
        usleep($milliseconds * 1000);
    }
}

Retry only failures that may recover

The client attempts the request at most twice. Network failures and server-side 5xx responses receive one short retry. A 429 response is retried only when it includes a numeric Retry-After value no greater than two seconds; otherwise it becomes a structured rate-limit failure. Authentication and validation errors are never retried.

<?php
// src/WebsiteCompanyClient.php
namespace App;

use App\Http\Transport;
use App\Http\TransportException;

final readonly class WebsiteCompanyClient
{
    private const ENDPOINT =
        'https://ai.mihajlo.mk/api/website-to-company-data/v1/extract';

    public function __construct(
        private Transport $transport,
        private Sleeper $sleeper,
        private string $token,
    ) {}

    public function extract(string $website, string $requestId): CompanyData
    {
        $website = $this->validateWebsite($website);

        for ($attempt = 1; $attempt <= 2; $attempt++) {
            try {
                $response = $this->transport->get(self::ENDPOINT, [
                    'token' => $this->token,
                    'website' => $website,
                ]);
            } catch (TransportException $exception) {
                $this->log('transport_failure', $requestId, $attempt);
                if ($attempt === 2) {
                    throw new CompanyDataException(
                        'unavailable',
                        'Enrichment service unavailable'
                    );
                }
                $this->sleeper->pause(250);
                continue;
            }

            $this->log('upstream_response', $requestId, $attempt, $response->status);

            if ($response->status === 200) {
                try {
                    $payload = json_decode(
                        $response->body,
                        true,
                        64,
                        JSON_THROW_ON_ERROR
                    );
                } catch (\JsonException) {
                    throw new CompanyDataException('schema', 'Invalid JSON response');
                }

                if (!is_array($payload) || array_is_list($payload)) {
                    throw new CompanyDataException('schema', 'Invalid response object');
                }

                return CompanyData::fromPayload($payload);
            }

            if (in_array($response->status, [401, 403], true)) {
                throw new CompanyDataException('authentication', 'Service token rejected');
            }

            if (in_array($response->status, [400, 422], true)) {
                throw new CompanyDataException('validation', 'Website rejected by service');
            }

            if ($response->status === 429) {
                $delay = $response->headers['retry-after'] ?? null;
                if (
                    $attempt === 1
                    && is_string($delay)
                    && ctype_digit($delay)
                    && (int) $delay <= 2
                ) {
                    $this->sleeper->pause((int) $delay * 1000);
                    continue;
                }
                throw new CompanyDataException('rate_limit', 'Service limit reached');
            }

            if ($response->status >= 500 && $attempt === 1) {
                $this->sleeper->pause(250);
                continue;
            }

            throw new CompanyDataException('upstream', 'Unexpected service response');
        }

        throw new CompanyDataException('unavailable', 'Enrichment service unavailable');
    }

    private function validateWebsite(string $website): string
    {
        $website = trim($website);
        $parts = parse_url($website);

        if (
            strlen($website) > 2048
            || filter_var($website, FILTER_VALIDATE_URL) === false
            || !is_array($parts)
            || !in_array($parts['scheme'] ?? '', ['http', 'https'], true)
            || empty($parts['host'])
            || isset($parts['user'])
            || isset($parts['pass'])
            || strtolower($parts['host']) === 'localhost'
        ) {
            throw new CompanyDataException('validation', 'Enter a public company website');
        }

        $host = $parts['host'];
        if (
            filter_var($host, FILTER_VALIDATE_IP) !== false
            && filter_var(
                $host,
                FILTER_VALIDATE_IP,
                FILTER_FLAG_NO_PRIV_RANGE | FILTER_FLAG_NO_RES_RANGE
            ) === false
        ) {
            throw new CompanyDataException('validation', 'Private addresses are not allowed');
        }

        return $website;
    }

    private function log(
        string $event,
        string $requestId,
        int $attempt,
        ?int $status = null
    ): void {
        error_log((string) json_encode([
            'event' => $event,
            'request_id' => $requestId,
            'attempt' => $attempt,
            'upstream_status' => $status,
        ], JSON_UNESCAPED_SLASHES));
    }
}

The log deliberately excludes the token, query string, full website, response body, email, phone, and people data. A request identifier still lets operators connect retries and failures. Production monitoring should count outcomes by failure kind and status, track latency, and alert on sustained authentication or schema failures.

Persist and return the lead draft

Create the SQLite schema once during deployment. Keeping enrichment as a draft is important: extracted data may be incomplete or stale, so the salesperson remains the final authority before conversion into a qualified lead.

<?php
// bin/migrate.php
require dirname(__DIR__) . '/vendor/autoload.php';

Dotenv\Dotenv::createImmutable(dirname(__DIR__))->safeLoad();
$pdo = new PDO($_ENV['CRM_DATABASE_DSN'] ?? 'sqlite:var/crm.sqlite');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec(
    'CREATE TABLE IF NOT EXISTS lead_drafts (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        website TEXT NOT NULL,
        company_json TEXT,
        contact_json TEXT,
        email TEXT,
        phone TEXT,
        people_json TEXT NOT NULL,
        created_at TEXT NOT NULL
    )'
);

// src/LeadRepository.php
namespace App;

final readonly class LeadRepository
{
    public function __construct(private \PDO $pdo) {}

    public function create(string $website, CompanyData $data): int
    {
        $statement = $this->pdo->prepare(
            'INSERT INTO lead_drafts
             (website, company_json, contact_json, email, phone, people_json, created_at)
             VALUES (:website, :company, :contact, :email, :phone, :people, :created)'
        );
        $statement->execute([
            'website' => $website,
            'company' => json_encode($data->company, JSON_THROW_ON_ERROR),
            'contact' => json_encode($data->contact, JSON_THROW_ON_ERROR),
            'email' => $data->email,
            'phone' => $data->phone,
            'people' => json_encode($data->people, JSON_THROW_ON_ERROR),
            'created' => gmdate('c'),
        ]);

        return (int) $this->pdo->lastInsertId();
    }
}

// src/LeadPrefillController.php
namespace App;

final readonly class LeadPrefillController
{
    public function __construct(
        private WebsiteCompanyClient $client,
        private LeadRepository $leads,
    ) {}

    public function handle(array $input, string $requestId): array
    {
        if (!isset($input['website']) || !is_string($input['website'])) {
            throw new CompanyDataException('validation', 'website is required');
        }

        $website = trim($input['website']);
        $data = $this->client->extract($website, $requestId);
        $id = $this->leads->create($website, $data);

        return ['lead_id' => $id, 'status' => 'draft', 'prefill' => $data->toArray()];
    }
}

The front controller wires the application together and translates domain failures into stable HTTP responses. Detailed upstream bodies remain server-side and are not reflected to the salesperson.

<?php
// public/index.php
declare(strict_types=1);

use App\CompanyDataException;
use App\Http\CurlTransport;
use App\LeadPrefillController;
use App\LeadRepository;
use App\NativeSleeper;
use App\WebsiteCompanyClient;
use Dotenv\Dotenv;

require dirname(__DIR__) . '/vendor/autoload.php';
Dotenv::createImmutable(dirname(__DIR__))->safeLoad();

header('Content-Type: application/json');
$requestId = bin2hex(random_bytes(12));

try {
    if ($_SERVER['REQUEST_METHOD'] !== 'POST'
        || parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH) !== '/leads/prefill') {
        http_response_code(404);
        echo json_encode(['error' => 'not_found', 'request_id' => $requestId]);
        exit;
    }

    $token = $_ENV['WEBSITE_COMPANY_TOKEN'] ?? '';
    if ($token === '' || $token === 'YOUR_SERVICE_TOKEN') {
        throw new RuntimeException('WEBSITE_COMPANY_TOKEN is not configured');
    }

    $input = json_decode(
        file_get_contents('php://input'),
        true,
        32,
        JSON_THROW_ON_ERROR
    );
    if (!is_array($input)) {
        throw new CompanyDataException('validation', 'JSON object required');
    }

    $pdo = new PDO($_ENV['CRM_DATABASE_DSN'] ?? 'sqlite:var/crm.sqlite');
    $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

    $controller = new LeadPrefillController(
        new WebsiteCompanyClient(new CurlTransport(), new NativeSleeper(), $token),
        new LeadRepository($pdo),
    );

    http_response_code(201);
    echo json_encode(
        $controller->handle($input, $requestId) + ['request_id' => $requestId],
        JSON_THROW_ON_ERROR
    );
} catch (CompanyDataException $exception) {
    $status = match ($exception->kind) {
        'validation' => 422,
        'rate_limit' => 429,
        'authentication' => 502,
        default => 502,
    };
    http_response_code($status);
    echo json_encode([
        'error' => $exception->kind,
        'message' => $exception->getMessage(),
        'request_id' => $requestId,
    ]);
} catch (JsonException) {
    http_response_code(400);
    echo json_encode(['error' => 'invalid_json', 'request_id' => $requestId]);
} catch (Throwable $exception) {
    error_log(json_encode(['event' => 'internal_error', 'request_id' => $requestId]));
    http_response_code(500);
    echo json_encode(['error' => 'internal', 'request_id' => $requestId]);
}

Test success, retries, and hard failures

A deterministic fake transport makes failure paths fast and reliable. The test below verifies boundary mapping, the single permitted server-error retry, and the rule that authentication failures must not retry.

<?php
// tests/WebsiteCompanyClientTest.php
namespace Tests;

use App\CompanyDataException;
use App\Http\HttpResponse;
use App\Http\Transport;
use App\Sleeper;
use App\WebsiteCompanyClient;
use PHPUnit\Framework\TestCase;

final class WebsiteCompanyClientTest extends TestCase
{
    public function testMapsDocumentedFields(): void
    {
        $transport = new FakeTransport([
            new HttpResponse(200, [], json_encode([
                'company' => ['name' => 'Example Ltd'],
                'contact' => ['name' => 'Sales'],
                'email' => '[email protected]',
                'phone' => '+1 555 0100',
                'people' => [['name' => 'Alex']],
            ], JSON_THROW_ON_ERROR)),
        ]);

        $data = $this->client($transport, new FakeSleeper())
            ->extract('https://example.com', 'request-1');

        self::assertSame('[email protected]', $data->email);
        self::assertSame('Example Ltd', $data->company['name']);
        self::assertCount(1, $transport->requests);
    }

    public function testRetriesOneServerFailure(): void
    {
        $transport = new FakeTransport([
            new HttpResponse(503, [], ''),
            new HttpResponse(200, [], '{"company":null,"contact":null,"email":null,"phone":null,"people":[]}'),
        ]);
        $sleeper = new FakeSleeper();

        $this->client($transport, $sleeper)
            ->extract('https://example.com', 'request-2');

        self::assertCount(2, $transport->requests);
        self::assertSame([250], $sleeper->delays);
    }

    public function testDoesNotRetryAuthenticationFailure(): void
    {
        $transport = new FakeTransport([new HttpResponse(401, [], '')]);

        try {
            $this->client($transport, new FakeSleeper())
                ->extract('https://example.com', 'request-3');
            self::fail('Expected authentication failure');
        } catch (CompanyDataException $exception) {
            self::assertSame('authentication', $exception->kind);
            self::assertCount(1, $transport->requests);
        }
    }

    private function client(Transport $transport, Sleeper $sleeper): WebsiteCompanyClient
    {
        return new WebsiteCompanyClient($transport, $sleeper, 'test-token');
    }
}

final class FakeTransport implements Transport
{
    public array $requests = [];

    public function __construct(private array $responses) {}

    public function get(string $url, array $query): HttpResponse
    {
        $this->requests[] = compact('url', 'query');
        return array_shift($this->responses);
    }
}

final class FakeSleeper implements Sleeper
{
    public array $delays = [];

    public function pause(int $milliseconds): void
    {
        $this->delays[] = $milliseconds;
    }
}

Deploy and verify the complete path

Install dependencies with a locked production build, run the migration once, point the web server’s document root at public, and ensure the PHP worker can write to var. The production process must receive WEBSITE_COMPANY_TOKEN and CRM_DATABASE_DSN. Terminate TLS at the web server, restrict the prefill route to authenticated CRM users, and apply application-side request limits so one user cannot consume the service plan’s allowance.

composer install --no-dev --classmap-authoritative
php bin/migrate.php

# Local verification only
php -S 127.0.0.1:8080 -t public

curl --request POST 'http://127.0.0.1:8080/leads/prefill' \
  --header 'Content-Type: application/json' \
  --data '{"website":"https://example.com"}'

vendor/bin/phpunit --testdox

Common failures have distinct signatures. A 422 means the submitted URL or service input was rejected. A local 429 means the upstream limit persisted and should be presented as “try later,” not silently looped. An authentication failure usually indicates an expired, regenerated, or incorrectly deployed token. Repeated schema failures indicate that the returned JSON no longer matches the boundary contract and deserve an alert rather than permissive guessing.

Final verification checklist

  • The real token exists only in environment-backed configuration.
  • The service request uses GET, the exact /v1/extract endpoint, and the token and website query parameters.
  • The boundary maps only company, contact, email, phone, and people.
  • Connection and total response timeouts are bounded.
  • Authentication and validation failures are never retried.
  • Logs contain request IDs and statuses, but no credentials or enriched personal data.
  • The CRM stores a reviewable draft and does not treat enrichment as unquestionable truth.
  • Tests pass with no external network dependency.

The valuable part of this feature is not merely turning a website into fields. It is creating a trustworthy boundary between an external enrichment service and the CRM: strict enough to fail visibly when assumptions break, restrained enough not to leak sensitive data, and quick enough to make a salesperson’s next action easier. With that boundary in place, entering one company website becomes the beginning of a useful lead rather than the beginning of a research chore.

Blog author portrait

Mihajlo

I’m Mihajlo — a developer driven by curiosity, discipline, and the constant urge to create something meaningful. I share insights, tutorials, and free services to help others simplify their work and grow in the ever-evolving world of software and AI.

Search

Categories

Recent Posts

Rethink Your APIs: Make Them Work Like a Human Would
Cultivate Ownership Beyond the AI Blueprint
Beyond the Draft: Integrating AI Agents into Your Software Workflow
API Gateway Intelligence: Architecting for Real-Time Traffic Intelligence
Symfony: AI Drafts Inbox Replies, You Keep the Final Say