Tutorials

Reproducible Linux Server Provisioning: Scripting, Validation, and Audit Trails

Reproducible Linux Server Provisioning: Scripting, Validation, and Audit Trails

A server that can be rebuilt is more valuable than a server that has merely survived. Manual fixes accumulate invisible assumptions: a package installed during an incident, a permission changed without a record, or a configuration copied from another host. Months later, the machine works, but nobody can explain why.

This tutorial builds a small, production-oriented provisioning system for a dedicated Debian 12 or Ubuntu 24.04 server. It installs Nginx, deploys a versioned health endpoint on 127.0.0.1:8080, validates the result, records an audit trail, and supports rollback. Repeated runs converge on the same state without restarting healthy services unnecessarily.

Prerequisites and operating boundaries

Use a test virtual machine before targeting a real host. The scripts deliberately manage Nginx’s default site, so the target must be a dedicated server without existing Nginx workloads.

  • A systemd-based Debian 12 or Ubuntu 24.04 host
  • Root access through sudo
  • Working package repositories and DNS
  • Outbound HTTP or HTTPS access for package installation
  • An existing, independently tested SSH path

All commands below are host commands, not container commands. The endpoint binds only to loopback, so it requires no inbound firewall opening. Do not expose it by changing the listen address until authentication, TLS, rate limits, and explicit firewall rules are designed.

Architecture and trade-offs

The system separates immutable releases from mutable activation state. Each release lives beneath /opt/repro-node/releases. The /opt/repro-node/current symlink selects the active release, allowing an atomic switch with a same-filesystem rename.

Every run creates a transaction under /var/lib/reprovision/transactions. It records the old and new release targets plus backups of configuration changed during that run. A non-blocking flock prevents concurrent provisioners from racing over the active symlink.

Rollback covers the application release and managed Nginx configuration. It intentionally does not uninstall packages: removing a package during recovery can trigger maintainer scripts, dependency removal, and unrelated service disruption. Package installation is therefore a durable host-level change, while release activation remains reversible.

The audit file is append-only by convention, not cryptographically tamper-proof. Root can modify it. For stronger evidence, forward the matching journal events to a remote log system with restricted retention controls.

Project structure

Create this project on an administration workstation or directly in a root-owned staging directory:

reprovision/
├── provision.sh
├── rollback.sh
├── validate.sh
└── files/
    ├── nginx-repro-node.conf
    └── public/
        └── health.json

The health document is deliberately static. Provisioning validation should not depend on a database, external API, or runtime that introduces unrelated failure modes.

{"status":"ok","service":"repro-node"}

Define the Nginx service

Save the following as files/nginx-repro-node.conf. The loopback binding is an important security boundary, not merely a convenient default.

server {
    listen 127.0.0.1:8080;
    server_name _;
    server_tokens off;

    root /opt/repro-node/current/public;

    access_log /var/log/nginx/repro-node.access.log;
    error_log  /var/log/nginx/repro-node.error.log warn;

    location = /healthz {
        default_type application/json;
        add_header Cache-Control "no-store" always;
        add_header X-Content-Type-Options "nosniff" always;
        try_files /health.json =503;
    }

    location / {
        return 404;
    }
}

try_files returns 503 if activation points at an incomplete release. Requests to every other path receive 404, reducing accidental exposure.

Implement idempotent provisioning

Save this script as provision.sh. Its release identifier is derived from the managed files, so unchanged input reuses the same immutable directory. Package metadata is refreshed only when a required package is missing.

#!/usr/bin/env bash
set -Eeuo pipefail

[[ $EUID -eq 0 ]] || { echo "Run as root" >&2; exit 1; }

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
BASE=/opt/repro-node
RELEASES=$BASE/releases
CURRENT=$BASE/current
TX_ROOT=/var/lib/reprovision/transactions
CONF=/etc/nginx/conf.d/repro-node.conf
DEFAULT_SITE=/etc/nginx/sites-enabled/default
AUDIT=/var/log/reprovision/audit.tsv
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-$$"
TX=$TX_ROOT/$RUN_ID
ARMED=0
STAGE=""

exec 9>/run/lock/reprovision.lock
flock -n 9 || { echo "Another provisioning run is active" >&2; exit 1; }

install -d -m 0755 "$BASE" "$RELEASES"
install -d -m 0700 "$TX_ROOT" "$TX"
install -d -m 0750 -o root -g adm "$(dirname "$AUDIT")"
touch "$AUDIT"
chown root:adm "$AUDIT"
chmod 0640 "$AUDIT"

audit() {
    printf '%s\t%s\t%s\t%s\t%s\n' \
        "$(date -u +%FT%TZ)" "$RUN_ID" "$1" "$2" "$3" >>"$AUDIT"
    logger -t reprovision -- "run=$RUN_ID action=$1 status=$2 detail=$3"
}

on_error() {
    local rc=$1
    trap - ERR
    set +e
    audit provision failure "exit-$rc"
    if [[ $ARMED -eq 1 ]]; then
        REPROVISION_LOCK_HELD=1 "$SCRIPT_DIR/rollback.sh" "$RUN_ID" --automatic
    fi
    if [[ -n $STAGE && $STAGE == "$RELEASES"/.staging.* ]]; then
        rm -rf -- "$STAGE"
    fi
    exit "$rc"
}
trap 'on_error $?' ERR

audit provision start requested

missing=()
for package in nginx curl; do
    dpkg-query -W -f='${Status}' "$package" 2>/dev/null |
        grep -qx 'install ok installed' || missing+=("$package")
done

if ((${#missing[@]})); then
    apt-get update
    DEBIAN_FRONTEND=noninteractive apt-get install -y \
        --no-install-recommends "${missing[@]}"
    audit packages changed "${missing[*]}"
else
    audit packages unchanged present
fi

RELEASE_ID="$(
    cd "$SCRIPT_DIR/files"
    sha256sum public/health.json nginx-repro-node.conf |
        sha256sum | awk '{print $1}'
)"
DEST=$RELEASES/$RELEASE_ID

if [[ ! -d $DEST ]]; then
    STAGE="$(mktemp -d "$RELEASES/.staging.XXXXXX")"
    install -d -m 0755 "$STAGE/public"
    install -m 0644 "$SCRIPT_DIR/files/public/health.json" \
        "$STAGE/public/health.json"
    mv -- "$STAGE" "$DEST"
    STAGE=""
    audit release created "$RELEASE_ID"
else
    audit release unchanged "$RELEASE_ID"
fi

if [[ -e $CURRENT && ! -L $CURRENT ]]; then
    echo "$CURRENT exists but is not a symlink" >&2
    exit 1
fi

previous="$(readlink -f "$CURRENT" 2>/dev/null || true)"
if [[ -n $previous && $previous != "$RELEASES"/* ]]; then
    echo "Refusing unmanaged current target: $previous" >&2
    exit 1
fi

printf '%s\n' "$previous" >"$TX/previous_target"
printf '%s\n' "$DEST" >"$TX/new_target"

if [[ -e $CONF ]]; then
    if cmp -s "$SCRIPT_DIR/files/nginx-repro-node.conf" "$CONF"; then
        touch "$TX/config.unchanged"
    else
        cp -a -- "$CONF" "$TX/nginx.conf"
        touch "$TX/config.changed"
    fi
else
    touch "$TX/config.absent"
fi

if [[ -L $DEFAULT_SITE ]]; then
    cp -a --no-dereference "$DEFAULT_SITE" "$TX/default-site"
    touch "$TX/default.changed"
elif [[ -e $DEFAULT_SITE ]]; then
    echo "Refusing to remove non-symlink $DEFAULT_SITE" >&2
    exit 1
fi

ARMED=1
install -m 0644 "$SCRIPT_DIR/files/nginx-repro-node.conf" "$CONF"

if [[ -f $TX/default.changed ]]; then
    rm -- "$DEFAULT_SITE"
fi

link_tmp=$BASE/.current-$RUN_ID
ln -s "$DEST" "$link_tmp"
mv -Tf -- "$link_tmp" "$CURRENT"
touch "$TX/activation.done"

nginx -t
if systemctl is-active --quiet nginx; then
    systemctl reload nginx
else
    systemctl enable --now nginx
fi

"$SCRIPT_DIR/validate.sh"
touch "$TX/success"
audit provision success "$RELEASE_ID"
trap - ERR
printf 'Provisioned release %s with transaction %s\n' "$RELEASE_ID" "$RUN_ID"

The error trap is armed only after the transaction has enough information to recover. Failures during package installation or release construction leave the previously active service untouched.

Add controlled rollback

Save this as rollback.sh. Manual rollback is permitted only when the active release still matches the transaction’s deployed release. That guard prevents an old transaction from silently overwriting a newer deployment.

#!/usr/bin/env bash
set -Eeuo pipefail

[[ $EUID -eq 0 ]] || { echo "Run as root" >&2; exit 1; }
[[ $# -ge 1 ]] || { echo "Usage: rollback.sh RUN_ID" >&2; exit 2; }

RUN_ID=$1
[[ $RUN_ID != *[^A-Za-z0-9._-]* ]] || { echo "Invalid run ID" >&2; exit 2; }

BASE=/opt/repro-node
CURRENT=$BASE/current
RELEASES=$BASE/releases
TX=/var/lib/reprovision/transactions/$RUN_ID
CONF=/etc/nginx/conf.d/repro-node.conf
DEFAULT_SITE=/etc/nginx/sites-enabled/default
AUDIT=/var/log/reprovision/audit.tsv

[[ -d $TX ]] || { echo "Unknown transaction: $RUN_ID" >&2; exit 1; }

if [[ ${REPROVISION_LOCK_HELD:-0} != 1 ]]; then
    exec 9>/run/lock/reprovision.lock
    flock -n 9 || { echo "Provisioning is active" >&2; exit 1; }
fi

previous="$(cat "$TX/previous_target")"
deployed="$(cat "$TX/new_target")"
current="$(readlink -f "$CURRENT" 2>/dev/null || true)"

if [[ -f $TX/activation.done ]]; then
    [[ $current == "$deployed" ]] || {
        echo "Active release no longer matches transaction" >&2
        exit 1
    }

    if [[ -n $previous ]]; then
        [[ $previous == "$RELEASES"/* && -d $previous ]] || {
            echo "Previous release is unavailable" >&2
            exit 1
        }
        tmp=$BASE/.rollback-$RUN_ID
        ln -s "$previous" "$tmp"
        mv -Tf -- "$tmp" "$CURRENT"
    else
        rm -- "$CURRENT"
    fi
fi

if [[ -f $TX/config.changed ]]; then
    install -m 0644 "$TX/nginx.conf" "$CONF"
elif [[ -f $TX/config.absent ]]; then
    rm -f -- "$CONF"
fi

if [[ -f $TX/default.changed ]]; then
    rm -f -- "$DEFAULT_SITE"
    cp -a --no-dereference "$TX/default-site" "$DEFAULT_SITE"
fi

nginx -t
if systemctl is-active --quiet nginx; then
    systemctl reload nginx
fi

printf '%s\t%s\trollback\tsuccess\t%s\n' \
    "$(date -u +%FT%TZ)" "$RUN_ID" "${previous:-no-release}" >>"$AUDIT"
logger -t reprovision -- "run=$RUN_ID action=rollback status=success"
printf 'Rolled back transaction %s\n' "$RUN_ID"

Nginx configuration is tested before reload. A failed test leaves the running Nginx process on its previous in-memory configuration, although the filesystem must still be investigated. Keep transaction directories root-owned and never accept transaction archives from untrusted sources.

Validate the converged state

Save this as validate.sh. Validation checks configuration syntax, service state, release containment, ownership, permissions, and the actual HTTP response.

#!/usr/bin/env bash
set -Eeuo pipefail

CURRENT=/opt/repro-node/current
expected='{"status":"ok","service":"repro-node"}'

[[ -L $CURRENT ]] || { echo "Current release is not a symlink" >&2; exit 1; }
target="$(readlink -f "$CURRENT")"
[[ $target == /opt/repro-node/releases/* ]] || {
    echo "Current target escapes the releases directory" >&2
    exit 1
}
[[ -f $target/public/health.json ]] || {
    echo "Health document is missing" >&2
    exit 1
}

[[ "$(stat -c '%U:%G:%a' "$target/public/health.json")" == "root:root:644" ]] || {
    echo "Unexpected health document ownership or mode" >&2
    exit 1
}

nginx -t
systemctl is-active --quiet nginx

body="$(curl --fail --silent --show-error \
    --connect-timeout 2 --max-time 5 \
    http://127.0.0.1:8080/healthz)"

[[ $body == "$expected" ]] || {
    echo "Unexpected health response: $body" >&2
    exit 1
}

printf 'Validation passed for %s\n' "$target"

The connection timeout bounds TCP establishment; --max-time independently bounds the complete request. One does not substitute for the other.

Deploy and test

From the project directory on the host, install the files into a root-owned location and run the provisioner:

sudo install -d -m 0755 /srv/reprovision/files/public
sudo install -m 0755 provision.sh rollback.sh validate.sh /srv/reprovision/
sudo install -m 0644 files/nginx-repro-node.conf /srv/reprovision/files/
sudo install -m 0644 files/public/health.json /srv/reprovision/files/public/

sudo /srv/reprovision/provision.sh
curl --fail --silent --show-error http://127.0.0.1:8080/healthz
sudo /srv/reprovision/provision.sh
sudo tail -n 20 /var/log/reprovision/audit.tsv
sudo journalctl -t reprovision --since today

The second run should report an unchanged package set and release while still performing validation. That is a useful idempotency test: success means convergence, not “nothing executed.”

Test rollback on a disposable VM by changing health.json, provisioning again, and passing the resulting transaction identifier to sudo /srv/reprovision/rollback.sh RUN_ID. Then run validate.sh. Do not simulate failure first on a production host.

Security, observability, and performance

Keep /srv/reprovision writable only by root or a tightly controlled deployment account. Anyone who can alter these scripts can obtain root-level effects during the next run. Review changes before execution and consider signing deployment artifacts in a larger system.

The loopback listener keeps the endpoint outside the network firewall. If external access becomes necessary, add a narrowly scoped firewall rule without flushing the host’s existing ruleset. Preserve established connections and confirm an SSH allow rule before enabling a default-drop policy. Prefer exposing Nginx through an authenticated reverse proxy rather than opening port 8080 directly.

Nginx access and error logs provide request-level visibility; the TSV file and journal provide provisioning events. Alert on failed provision runs, repeated rollbacks, Nginx validation failures, and health-check latency. Forward logs off-host because local evidence disappears with the server.

Content-addressed releases avoid recopying unchanged artifacts, and package checks avoid unnecessary repository refreshes. The global lock serializes the rare mutating path. Release cleanup should be a separate, reviewed retention job that never removes the current target or any release referenced by retained transactions.

Common failure modes

  • Package repositories are unreachable: the run stops before activation, leaving the current release intact.
  • Nginx already hosts applications: stop and adapt the configuration ownership model; this tutorial assumes a dedicated host.
  • The current path is not a symlink: the script refuses to replace an unmanaged directory.
  • Validation returns connection refused: inspect systemctl status nginx and the Nginx error log.
  • Rollback rejects a transaction: a newer deployment is active, or the previous release was removed.
  • Permissions drift: validation fails before declaring success, making the drift visible in the audit trail.

Final verification checklist

  • The provisioner succeeds twice with the same inputs.
  • /opt/repro-node/current points inside the releases directory.
  • nginx -t succeeds and Nginx is active.
  • The health endpoint responds only on 127.0.0.1:8080.
  • The response matches the expected JSON exactly.
  • The audit TSV and journal contain start, change, success, and rollback events.
  • A tested transaction can restore the preceding release.
  • Project files, transactions, and deployment privileges are restricted.
  • Firewall and SSH access have been reviewed independently.

Reproducibility is not achieved by putting commands into a shell file. It comes from explicit ownership, deterministic inputs, atomic activation, bounded validation, honest rollback boundaries, and evidence of every transition. When those properties are designed together, rebuilding a server stops being an act of archaeology and becomes a routine, inspectable operation.

Blog author portrait

Mihajlo

I’m Mihajlo — a developer driven by curiosity, discipline, and the constant urge to create something meaningful. I share insights, tutorials, and free services to help others simplify their work and grow in the ever-evolving world of software and AI.

Search

Categories

Recent Posts

Beyond Code Completion: How AI Agents Master Your Development Workflow
System Architecture: The Case for Boring, Maintainable Code
Beyond Features: Shipping Products Rooted in True Ownership
AI Agents: From Smart Assistants to Indispensable Software Architects
Refactoring for Resilience: When and How to Rebuild Your PHP Systems