Nginx Powerhouse: Automate TLS, Secure APIs, and Achieve Zero-Downtime with This Stack
A reverse proxy becomes infrastructure the moment clients depend on it. At that point, “Nginx forwards requests” is not enough. Certificates must renew unattended, abusive clients must be contained, failed application instances must stop receiving traffic, and configuration changes must not interrupt active requests.
This tutorial builds that production stack on a Debian or Ubuntu host: two small Go API instances bound to loopback, open-source Nginx as the public edge, Certbot using the ACME webroot flow, passive and active health checks, rate limits, structured logs, and validated zero-downtime reloads.
Architecture and operating assumptions
Replace api.example.com and [email protected] throughout. The hostname must already resolve to the server’s public address. Only ports 80 and 443 are exposed; the API listens on 127.0.0.1:9001 and 127.0.0.1:9002.
- Nginx terminates TLS, applies limits, logs requests, and balances traffic.
- Two systemd-managed API processes provide redundancy during deployments.
- Nginx performs passive health detection from real traffic.
- A systemd timer actively probes both instances and reports failures to the journal.
- Certbot renews certificates through a webroot that remains reachable over HTTP.
Open-source Nginx does not provide the configurable active upstream health checks found in Nginx Plus. Passive failure handling protects client traffic, while the timer supplies proactive detection. The timer deliberately does not rewrite Nginx configuration: automatic removal based on a single probe can amplify transient failures.
Prerequisites and project layout
Use a current supported Debian or Ubuntu release with systemd, Nginx, Go 1.22 or newer, and root access through sudo. Before enabling a host firewall, preserve administrative access by allowing the actual SSH port from trusted networks. Then permit inbound TCP 80 and 443 in both the host firewall and any provider security group. Do not expose 9001 or 9002.
Install the distribution-supported packages and create explicit directories:
sudo apt update
sudo apt install nginx certbot curl
go version
nginx -v
sudo install -d -m 0755 /opt/power-api/src
sudo install -d -m 0755 /opt/power-api/current
sudo install -d -o www-data -g www-data -m 0755 /srv/www/acme
sudo install -d -m 0755 /etc/power-api
sudo useradd --system --home-dir /nonexistent \
--shell /usr/sbin/nologin power-apiIf the account already exists, skip useradd. The resulting structure is:
/opt/power-api/src/main.go
/opt/power-api/current/api
/etc/power-api/1.env
/etc/power-api/2.env
/etc/systemd/system/[email protected]
/etc/nginx/conf.d/00-power-api-global.conf
/etc/nginx/sites-available/api.example.com
/srv/www/acme/.well-known/acme-challenge/Build a bounded, shutdown-aware API
Create /opt/power-api/src/main.go with sudoedit. The server exposes a health endpoint and one example API route. Its connection and shutdown budgets are finite, and it stops advertising readiness as soon as termination begins.
package main
import (
"context"
"encoding/json"
"flag"
"log"
"net/http"
"os"
"os/signal"
"sync/atomic"
"syscall"
"time"
)
func main() {
listen := flag.String("listen", "127.0.0.1:9001", "listen address")
flag.Parse()
var ready atomic.Bool
ready.Store(true)
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
if !ready.Load() {
http.Error(w, "draining", http.StatusServiceUnavailable)
return
}
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok\n"))
})
mux.HandleFunc("GET /v1/time", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{
"time": time.Now().UTC().Format(time.RFC3339),
})
})
server := &http.Server{
Addr: *listen,
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 15 * time.Second,
IdleTimeout: 60 * time.Second,
}
go func() {
log.Printf("listening on %s", *listen)
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatal(err)
}
}()
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGINT, syscall.SIGTERM)
<-signals
ready.Store(false)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := server.Shutdown(ctx); err != nil {
log.Printf("graceful shutdown failed: %v", err)
_ = server.Close()
}
}Build the binary, then create the two environment files using sudoedit. Their contents are LISTEN=127.0.0.1:9001 and LISTEN=127.0.0.1:9002, respectively.
sudo go build -trimpath \
-o /opt/power-api/current/api \
/opt/power-api/src/main.go
sudo chown root:root /opt/power-api/current/api
sudo chmod 0755 /opt/power-api/current/apiRun both instances under systemd
Create /etc/systemd/system/[email protected]:
[Unit]
Description=Power API instance %i
After=network.target
StartLimitIntervalSec=60
StartLimitBurst=5
[Service]
Type=simple
User=power-api
Group=power-api
EnvironmentFile=/etc/power-api/%i.env
ExecStart=/opt/power-api/current/api -listen ${LISTEN}
Restart=on-failure
RestartSec=2
TimeoutStopSec=15
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
[Install]
WantedBy=multi-user.targetLoad, enable, and verify the services:
sudo systemctl daemon-reload
sudo systemctl enable --now power-api@1 power-api@2
curl --fail --max-time 2 http://127.0.0.1:9001/healthz
curl --fail --max-time 2 http://127.0.0.1:9002/healthz
sudo systemctl status power-api@1 power-api@2Configure limits, logging, and upstream behavior
Create /etc/nginx/conf.d/00-power-api-global.conf. Files under conf.d are included inside Nginx’s http context on the target distributions.
server_tokens off;
limit_req_zone $binary_remote_addr zone=api_per_ip:10m rate=10r/s;
limit_conn_zone $binary_remote_addr zone=connections_per_ip:10m;
log_format api_json escape=json
'{"time":"$time_iso8601",'
'"request_id":"$request_id",'
'"remote_addr":"$remote_addr",'
'"method":"$request_method",'
'"uri":"$request_uri",'
'"status":"$status",'
'"bytes":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"upstream_addr":"$upstream_addr",'
'"upstream_status":"$upstream_status",'
'"upstream_time":"$upstream_response_time"}';The limits use the directly connected client address. If a trusted load balancer or CDN is later placed in front of Nginx, configure the real-IP module with that provider’s exact, maintained address ranges. Never trust arbitrary X-Forwarded-For input, or clients can evade limits and falsify logs.
Bootstrap HTTP and obtain the certificate
Before referencing certificate files that do not exist, create a temporary HTTP-only virtual host at /etc/nginx/sites-available/api.example.com:
server {
listen 80;
listen [::]:80;
server_name api.example.com;
location ^~ /.well-known/acme-challenge/ {
root /srv/www/acme;
default_type text/plain;
try_files $uri =404;
}
location / {
return 503;
}
}Enable it only after checking that an existing site does not claim the same hostname. Creating the symlink is non-destructive; if the target already exists, inspect it rather than replacing it blindly.
sudo ln -s /etc/nginx/sites-available/api.example.com \
/etc/nginx/sites-enabled/api.example.com
sudo nginx -t
sudo systemctl reload nginx
sudo certbot certonly --webroot \
--webroot-path /srv/www/acme \
--domain api.example.com \
--email [email protected] \
--agree-tos --no-eff-emailActivate the production reverse proxy
After issuance succeeds, edit the same site file. The upstream uses least-connections balancing, persistent upstream connections, passive failure accounting, and bounded retry time. Nginx will not retry non-idempotent requests after sending them upstream because non_idempotent is intentionally absent.
upstream power_api {
zone power_api 64k;
least_conn;
server 127.0.0.1:9001 max_fails=3 fail_timeout=10s;
server 127.0.0.1:9002 max_fails=3 fail_timeout=10s;
keepalive 32;
}
server {
listen 80;
listen [::]:80;
server_name api.example.com;
location ^~ /.well-known/acme-challenge/ {
root /srv/www/acme;
default_type text/plain;
try_files $uri =404;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name api.example.com;
ssl_certificate /etc/letsencrypt/live/api.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_cache shared:TLS:20m;
ssl_session_timeout 1d;
ssl_session_tickets off;
access_log /var/log/nginx/power-api.access.log api_json;
error_log /var/log/nginx/power-api.error.log warn;
client_max_body_size 1m;
limit_req_status 429;
limit_conn_status 429;
location = /healthz {
access_log off;
proxy_pass http://power_api/healthz;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 2s;
proxy_read_timeout 3s;
}
location /v1/ {
limit_req zone=api_per_ip burst=40 nodelay;
limit_conn connections_per_ip 20;
proxy_pass http://power_api;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Request-ID $request_id;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
add_header X-Request-ID $request_id always;
proxy_connect_timeout 2s;
proxy_send_timeout 10s;
proxy_read_timeout 15s;
proxy_next_upstream error timeout invalid_header
http_502 http_503 http_504;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 3s;
}
location / {
return 404;
}
}Validate before every reload. On reload, the Nginx master starts workers with the new configuration and asks old workers to drain existing connections, avoiding the connection drop caused by a stop-and-start cycle.
sudo nginx -t
sudo systemctl reload nginx
curl --fail --max-time 5 https://api.example.com/healthz
curl --fail --max-time 5 https://api.example.com/v1/timeAutomate renewal and active health checks
Create /etc/letsencrypt/renewal-hooks/deploy/reload-nginx with the following content, then make it executable. Deploy hooks run after a successful renewal, not when no certificate changed.
#!/bin/sh
set -eu
/usr/sbin/nginx -t
/usr/bin/systemctl reload nginxsudo chmod 0755 \
/etc/letsencrypt/renewal-hooks/deploy/reload-nginx
sudo systemctl enable --now certbot.timer
sudo certbot renew --dry-runConfirm executable paths with command -v nginx systemctl if the distribution installs them elsewhere.
For active checks, create /usr/local/sbin/check-power-api and mark it executable:
#!/bin/sh
set -eu
failed=0
for port in 9001 9002; do
if ! /usr/bin/curl --fail --silent --show-error \
--max-time 2 "http://127.0.0.1:${port}/healthz"; then
/usr/bin/logger -t power-api-health \
"health check failed on port ${port}"
failed=1
fi
done
exit "$failed"Create power-api-health.service and power-api-health.timer under /etc/systemd/system:
[Unit]
Description=Check local Power API instances
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/check-power-api
User=nobody
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
# power-api-health.timer
[Unit]
Description=Run Power API health checks
[Timer]
OnBootSec=30s
OnUnitActiveSec=15s
AccuracySec=2s
[Install]
WantedBy=timers.targetThe comment separates the two files; place the second [Unit] section onward in the timer file. Enable it with sudo systemctl daemon-reload followed by sudo systemctl enable --now power-api-health.timer.
Test failure paths and deployment behavior
Stop one instance and make several requests. Requests should continue through the survivor, while the access log’s upstream_addr and upstream_status fields reveal failure and retry behavior.
sudo systemctl stop power-api@1
for n in 1 2 3 4 5; do
curl --fail --max-time 5 https://api.example.com/v1/time
done
sudo systemctl start power-api@1
sudo journalctl -u power-api-health.service --since "10 minutes ago"
sudo tail -n 20 /var/log/nginx/power-api.access.logTest rate limiting with a controlled burst and expect some 429 responses. Run load tests from an authorized system only. A shared NAT address represents many users, so per-IP limits must reflect the client population; authenticated APIs often benefit from a second limit keyed by a validated API identity.
Deploy API changes one instance at a time: replace the binary using an atomic release mechanism, restart power-api@1, wait until its direct health check succeeds, and only then restart instance 2. If the new build fails, the untouched instance continues serving. For Nginx changes, always use nginx -t followed by systemctl reload nginx.
Security, performance, and observability notes
- Keep loopback services private and deny public access to their ports at every network layer.
- Protect
/etc/letsencryptand its private keys with root-only administration. Never copy keys into application directories. - Add HTTP Strict Transport Security only after confirming that the hostname and its operational recovery path are permanently HTTPS-ready.
- Keep request bodies, connection counts, and timeouts bounded. Enlarging every timeout usually converts brief upstream trouble into resource exhaustion.
- Ship the JSON access log and systemd journal to external storage. Alert on sustained 5xx responses, health-unit failures, renewal failures, high latency, and unexpected 429 rates.
- Watch file descriptors, worker connections, CPU saturation, memory, and upstream response time before tuning worker counts or keepalive pools.
Common production failures
A certificate issuance failure usually means DNS points elsewhere, port 80 is blocked, or another virtual host captures the challenge. Test a file beneath /srv/www/acme/.well-known/acme-challenge/ from outside the server.
A 502 indicates that Nginx cannot connect to an upstream, the process exited, or a mandatory access-control system denied the connection. Check the API journal and Nginx error log before increasing timeouts. Persistent old Nginx workers after reload usually indicate long-lived connections; inspect them rather than killing workers and breaking clients.
Unexpected 429 responses often expose a poor limiting key, not insufficient capacity. Conversely, an apparently ineffective limit may mean Nginx trusts a spoofable forwarding header.
Final verification checklist
- Both loopback health endpoints return 200 and are unreachable publicly.
- HTTP redirects normally while the ACME challenge path remains available.
- TLS 1.2 and TLS 1.3 clients can connect with the expected certificate chain.
- A stopped API instance does not take the public endpoint offline.
- Rate-limit bursts produce controlled 429 responses.
certbot renew --dry-runsucceeds and validates Nginx before reload.- Structured logs contain request IDs, upstream addresses, statuses, and timings.
- Every configuration change passes
nginx -tbefore a reload.
The real power of this stack is not any single directive. It is the sequence of guarded transitions: bounded requests, observable failures, redundant upstreams, certificates renewed before expiry, and configurations validated before old workers surrender traffic. That turns Nginx from a convenient proxy into a dependable production boundary.
