Go Reverse Proxy: Circuit Breakers, Load Shedding, and Structured Logging in Practice

Mihajlo
Aug 11, 2026
Tutorials

A reverse proxy usually fails long before it stops accepting connections. A slow upstream consumes every available request slot, retries multiply the pressure, logs become an unsearchable stream, and a technically “healthy” proxy turns into a very efficient outage distributor.

This tutorial builds a production-oriented Go proxy that behaves differently. It limits concurrency, isolates failing backends with per-backend circuit breakers, performs active health checks, and emits structured access logs. The implementation uses only the Go standard library and deliberately avoids automatic retries, preserving safe behavior for non-idempotent requests.

Prerequisites and architecture

You need Go 1.22 or newer and two HTTP backends reachable from the proxy host. The example uses these addresses:

127.0.0.1:8080: reverse proxy
127.0.0.1:9001 and 127.0.0.1:9002: upstream services
/healthz: required upstream health endpoint
/livez and /readyz: proxy lifecycle endpoints

Requests first pass through access logging and a non-blocking concurrency gate. The proxy then selects a healthy backend whose circuit breaker permits traffic. Transport errors and upstream 5xx responses count as failures; client cancellations do not.

There are two intentional trade-offs. Selection is round-robin rather than latency-aware, which makes behavior understandable under pressure. Requests are also attempted once. Retrying a POST after an ambiguous network failure can duplicate a side effect, so retries belong in an explicitly idempotent layer.

Project structure

edgeproxy/
├── go.mod
├── cmd/
│   ├── edgeproxy/main.go
│   └── testbackend/main.go
└── deploy/
    └── edgeproxy.service

Create the directories with your editor or mkdir -p edgeproxy/cmd/{edgeproxy,testbackend} edgeproxy/deploy, then add the following module file:

module example.com/edgeproxy

go 1.22

Implement the proxy

The circuit breaker uses generation-numbered permits. When a breaker changes state, late results from the previous generation are ignored. This prevents an old in-flight request from accidentally closing a newer circuit.

package main

import (
	"context"
	"crypto/rand"
	"encoding/hex"
	"flag"
	"io"
	"log/slog"
	"net"
	"net/http"
	"net/http/httputil"
	"net/url"
	"os"
	"os/signal"
	"strings"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
)

type circuitState uint8

const (
	closed circuitState = iota
	open
	halfOpen
)

type breaker struct {
	mu            sync.Mutex
	state         circuitState
	failures      int
	threshold     int
	cooldown      time.Duration
	openedAt      time.Time
	probeInFlight bool
	generation    uint64
}

func (b *breaker) allow(now time.Time) (uint64, bool) {
	b.mu.Lock()
	defer b.mu.Unlock()

	switch b.state {
	case closed:
		return b.generation, true
	case open:
		if now.Sub(b.openedAt) < b.cooldown {
			return 0, false
		}
		b.state = halfOpen
		b.probeInFlight = true
		b.generation++
		return b.generation, true
	case halfOpen:
		if b.probeInFlight {
			return 0, false
		}
		b.probeInFlight = true
		return b.generation, true
	default:
		return 0, false
	}
}

func (b *breaker) done(ticket uint64, success bool) {
	b.mu.Lock()
	defer b.mu.Unlock()

	if ticket != b.generation {
		return
	}

	switch b.state {
	case closed:
		if success {
			b.failures = 0
			return
		}
		b.failures++
		if b.failures >= b.threshold {
			b.state = open
			b.openedAt = time.Now()
			b.generation++
		}
	case halfOpen:
		b.probeInFlight = false
		b.generation++
		if success {
			b.state = closed
			b.failures = 0
		} else {
			b.state = open
			b.openedAt = time.Now()
		}
	}
}

func (b *breaker) cancel(ticket uint64) {
	b.mu.Lock()
	defer b.mu.Unlock()

	if ticket == b.generation && b.state == halfOpen {
		b.probeInFlight = false
		b.state = open
		b.openedAt = time.Now()
		b.generation++
	}
}

func (b *breaker) ready(now time.Time) bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	return b.state == closed ||
		(b.state == open && now.Sub(b.openedAt) >= b.cooldown)
}

type backend struct {
	url     *url.URL
	healthy atomic.Bool
	breaker breaker
	proxy   *httputil.ReverseProxy
}

type permit struct {
	backend *backend
	ticket  uint64
}

type contextKey uint8

const (
	permitKey contextKey = iota
	metaKey
)

type requestMeta struct {
	backend string
}

type selector struct {
	backends []*backend
	next     atomic.Uint64
}

func (s *selector) choose() (*backend, uint64, bool) {
	n := len(s.backends)
	start := int(s.next.Add(1)-1) % n

	for i := 0; i < n; i++ {
		b := s.backends[(start+i)%n]
		if !b.healthy.Load() {
			continue
		}
		if ticket, ok := b.breaker.allow(time.Now()); ok {
			return b, ticket, true
		}
	}
	return nil, 0, false
}

type responseRecorder struct {
	http.ResponseWriter
	status int
	bytes  int
}

func (w *responseRecorder) WriteHeader(status int) {
	if w.status == 0 {
		w.status = status
		w.ResponseWriter.WriteHeader(status)
	}
}

func (w *responseRecorder) Write(p []byte) (int, error) {
	if w.status == 0 {
		w.WriteHeader(http.StatusOK)
	}
	n, err := w.ResponseWriter.Write(p)
	w.bytes += n
	return n, err
}

func (w *responseRecorder) Unwrap() http.ResponseWriter {
	return w.ResponseWriter
}

func newRequestID() string {
	var value [16]byte
	if _, err := rand.Read(value[:]); err == nil {
		return hex.EncodeToString(value[:])
	}
	return hex.EncodeToString([]byte(time.Now().UTC().Format(time.RFC3339Nano)))
}

func clientAddress(remote string) string {
	host, _, err := net.SplitHostPort(remote)
	if err == nil {
		return host
	}
	return remote
}

func accessLog(log *slog.Logger, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		id := newRequestID()
		meta := &requestMeta{}
		r = r.WithContext(context.WithValue(r.Context(), metaKey, meta))
		r.Header.Set("X-Request-ID", id)

		rec := &responseRecorder{ResponseWriter: w}
		next.ServeHTTP(rec, r)
		status := rec.status
		if status == 0 {
			status = http.StatusOK
		}

		log.Info("access",
			"request_id", id,
			"method", r.Method,
			"path", r.URL.Path,
			"status", status,
			"bytes", rec.bytes,
			"duration_ms", time.Since(started).Milliseconds(),
			"client_ip", clientAddress(r.RemoteAddr),
			"backend", meta.backend,
		)
	})
}

func shed(limit int, next http.Handler) http.Handler {
	slots := make(chan struct{}, limit)

	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		select {
		case slots <- struct{}{}:
			defer func() { <-slots }()
			next.ServeHTTP(w, r)
		default:
			w.Header().Set("Retry-After", "1")
			http.Error(w, "proxy capacity exhausted", http.StatusServiceUnavailable)
		}
	})
}

func checkBackend(ctx context.Context, client *http.Client, b *backend) bool {
	u := *b.url
	u.Path = "/healthz"
	u.RawQuery = ""

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
	if err != nil {
		return false
	}
	resp, err := client.Do(req)
	if err != nil {
		return false
	}
	defer resp.Body.Close()
	_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1024))
	return resp.StatusCode >= 200 && resp.StatusCode < 300
}

func healthLoop(
	ctx context.Context,
	log *slog.Logger,
	client *http.Client,
	backends []*backend,
) {
	run := func() {
		for _, b := range backends {
			ok := checkBackend(ctx, client, b)
			previous := b.healthy.Swap(ok)
			if previous != ok {
				log.Info("backend_health_changed",
					"backend", b.url.String(),
					"healthy", ok,
				)
			}
		}
	}

	run()
	ticker := time.NewTicker(5 * time.Second)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			run()
		}
	}
}

func main() {
	listen := flag.String("listen", "127.0.0.1:8080", "proxy listen address")
	csv := flag.String(
		"backends",
		"http://127.0.0.1:9001,http://127.0.0.1:9002",
		"comma-separated backend URLs",
	)
	maxInFlight := flag.Int("max-in-flight", 256, "concurrent request limit")
	flag.Parse()

	log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
	if *maxInFlight < 1 {
		log.Error("max-in-flight must be positive")
		os.Exit(2)
	}

	transport := &http.Transport{
		Proxy:                 http.ProxyFromEnvironment,
		DialContext:           (&net.Dialer{Timeout: 2 * time.Second}).DialContext,
		ForceAttemptHTTP2:     true,
		MaxIdleConns:          256,
		MaxIdleConnsPerHost:   64,
		IdleConnTimeout:       60 * time.Second,
		TLSHandshakeTimeout:   3 * time.Second,
		ResponseHeaderTimeout: 5 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
	}

	var backends []*backend
	for _, raw := range strings.Split(*csv, ",") {
		u, err := url.Parse(strings.TrimSpace(raw))
		if err != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") {
			log.Error("invalid backend URL", "value", raw)
			os.Exit(2)
		}

		b := &backend{
			url: u,
			breaker: breaker{
				threshold: 5,
				cooldown:  15 * time.Second,
			},
		}

		b.proxy = &httputil.ReverseProxy{
			Transport: transport,
			Rewrite: func(pr *httputil.ProxyRequest) {
				pr.SetURL(b.url)
				pr.Out.Host = b.url.Host
				pr.SetXForwarded()
				pr.Out.Header.Set("X-Request-ID", pr.In.Header.Get("X-Request-ID"))
			},
			ModifyResponse: func(resp *http.Response) error {
				p := resp.Request.Context().Value(permitKey).(permit)
				p.backend.breaker.done(p.ticket, resp.StatusCode < 500)
				return nil
			},
			ErrorHandler: func(w http.ResponseWriter, r *http.Request, err error) {
				p := r.Context().Value(permitKey).(permit)
				if r.Context().Err() != nil {
					p.backend.breaker.cancel(p.ticket)
					w.WriteHeader(499)
					return
				}
				p.backend.breaker.done(p.ticket, false)
				log.Warn("upstream_error", "backend", b.url.String(), "error", err)
				http.Error(w, "bad gateway", http.StatusBadGateway)
			},
			ErrorLog: slog.NewLogLogger(log.Handler(), slog.LevelError),
		}
		backends = append(backends, b)
	}

	sel := &selector{backends: backends}
	proxyHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		b, ticket, ok := sel.choose()
		if !ok {
			w.Header().Set("Retry-After", "1")
			http.Error(w, "no backend available", http.StatusServiceUnavailable)
			return
		}

		if meta, ok := r.Context().Value(metaKey).(*requestMeta); ok {
			meta.backend = b.url.Host
		}
		ctx := context.WithValue(r.Context(), permitKey, permit{b, ticket})
		b.proxy.ServeHTTP(w, r.WithContext(ctx))
	})

	healthCtx, stopHealth := context.WithCancel(context.Background())
	healthClient := &http.Client{Timeout: time.Second}
	go healthLoop(healthCtx, log, healthClient, backends)

	mux := http.NewServeMux()
	mux.HandleFunc("/livez", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusNoContent)
	})
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) {
		for _, b := range backends {
			if b.healthy.Load() && b.breaker.ready(time.Now()) {
				w.WriteHeader(http.StatusNoContent)
				return
			}
		}
		http.Error(w, "no backend ready", http.StatusServiceUnavailable)
	})
	mux.Handle("/", accessLog(log, shed(*maxInFlight, proxyHandler)))

	server := &http.Server{
		Addr:              *listen,
		Handler:           mux,
		ReadHeaderTimeout: 5 * time.Second,
		ReadTimeout:       30 * time.Second,
		WriteTimeout:      30 * time.Second,
		IdleTimeout:       60 * time.Second,
		MaxHeaderBytes:    1 << 20,
	}

	signalCtx, stopSignals := signal.NotifyContext(
		context.Background(), syscall.SIGINT, syscall.SIGTERM,
	)
	defer stopSignals()

	errs := make(chan error, 1)
	go func() {
		log.Info("proxy_started", "listen", *listen)
		errs <- server.ListenAndServe()
	}()

	select {
	case <-signalCtx.Done():
		log.Info("shutdown_started")
	case err := <-errs:
		if err != nil && err != http.ErrServerClosed {
			log.Error("server_failed", "error", err)
		}
	}

	stopHealth()
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	if err := server.Shutdown(shutdownCtx); err != nil {
		log.Error("shutdown_incomplete", "error", err)
	}
	transport.CloseIdleConnections()
}

The active health result and circuit state remain separate. A backend may answer health checks while its application requests fail; the circuit breaker still removes it temporarily. Conversely, a failed health check immediately prevents selection without corrupting circuit history.

Create controlled test backends

This small server provides a healthy endpoint and an optional, deterministic 503 response. Save it as cmd/testbackend/main.go.

package main

import (
	"flag"
	"fmt"
	"log"
	"net/http"
	"sync/atomic"
)

func main() {
	listen := flag.String("listen", "127.0.0.1:9001", "listen address")
	name := flag.String("name", "backend-1", "response name")
	failEvery := flag.Uint64("fail-every", 0, "return 503 every N requests")
	flag.Parse()

	var requests atomic.Uint64
	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusNoContent)
	})
	mux.HandleFunc("/", func(w http.ResponseWriter, _ *http.Request) {
		n := requests.Add(1)
		if *failEvery > 0 && n%*failEvery == 0 {
			http.Error(w, "simulated failure", http.StatusServiceUnavailable)
			return
		}
		fmt.Fprintf(w, "%s request=%d\n", *name, n)
	})

	log.Printf("test backend listening on %s", *listen)
	log.Fatal(http.ListenAndServe(*listen, mux))
}

From three terminals, run:

go run ./cmd/testbackend -listen=127.0.0.1:9001 -name=backend-1
go run ./cmd/testbackend -listen=127.0.0.1:9002 -name=backend-2 -fail-every=3
go run ./cmd/edgeproxy -listen=127.0.0.1:8080 \
  -backends=http://127.0.0.1:9001,http://127.0.0.1:9002 \
  -max-in-flight=64

Verify failure behavior

First confirm lifecycle endpoints and round-robin routing:

curl --fail --silent --show-error -o /dev/null \
  http://127.0.0.1:8080/readyz

for n in 1 2 3 4 5 6 7 8; do
  curl --silent --show-error -i http://127.0.0.1:8080/demo
done

Stop one test backend. Its health status changes after the next check, while requests continue through the remaining backend. Restart it and allow one health interval for readmission.

To exercise load shedding, lower -max-in-flight and point the proxy at a backend handler that deliberately blocks. Excess requests should receive 503 immediately with Retry-After: 1; they must not wait in an unbounded in-process queue.

The breaker opens after five relevant failures in one generation. It remains open for 15 seconds, permits a single half-open probe, and closes only if that probe succeeds. Because every automatic retry consumes additional backend capacity, clients should apply bounded retries with jitter and retry only operations they know are safe.

Security, performance, and observability

Keep the proxy on a private address when TLS terminates at a trusted frontend. If it must accept public traffic directly, add TLS, restrict allowed methods and body sizes, and place administrative endpoints on a separate protected listener. A host firewall should permit the public frontend port while limiting backend ports to the proxy host or private network.

ProxyRequest.SetXForwarded is important: Go removes untrusted forwarding headers before the rewrite function runs, then generates controlled values. Do not copy arbitrary inbound X-Forwarded-For headers yourself.

The transport bounds connection establishment, TLS negotiation, and response-header waiting separately. A dial timeout does not bound a slow response. Server read and write budgets are 30 seconds, so this configuration targets conventional API traffic. Long-lived streams and large uploads need route-specific budgets rather than globally disabling protection.

JSON access records include request ID, selected backend, status, response bytes, client address, and duration. Health transitions and transport failures are separate events. In a larger system, derive counters and latency histograms from these events or instrument the same boundaries with your metrics stack. Avoid logging authorization headers, cookies, query secrets, or bodies.

Deploy with systemd

Build as an unprivileged user, inspect the destination before replacing an existing binary, and then install the explicit artifact:

mkdir -p ./bin
go test ./...
go build -trimpath -o ./bin/edgeproxy ./cmd/edgeproxy
sudo install -m 0755 ./bin/edgeproxy /usr/local/bin/edgeproxy

Save this unit as /etc/systemd/system/edgeproxy.service. Creating that file and managing the service require root privileges.

[Unit]
Description=Go edge reverse proxy
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=edgeproxy
Group=edgeproxy
ExecStart=/usr/local/bin/edgeproxy -listen=127.0.0.1:8080 -backends=http://127.0.0.1:9001,http://127.0.0.1:9002 -max-in-flight=256
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
CapabilityBoundingSet=
AmbientCapabilities=

[Install]
WantedBy=multi-user.target

Create the dedicated system user according to local account policy, validate the unit with systemd-analyze verify /etc/systemd/system/edgeproxy.service, then use systemctl daemon-reload, systemctl enable --now edgeproxy, and journalctl -u edgeproxy. If upstream DNS, service discovery, or Unix sockets are required, adjust the address-family and network assumptions deliberately.

Common production failures

Every backend starts unavailable: this is intentional until the first active health check succeeds. Make readiness probes tolerate startup time.
Healthy endpoints mask broken application paths: make /healthz verify only dependencies required to serve traffic, while relying on breaker outcomes for request-path failures.
503 responses appear during traffic spikes: determine whether they came from load shedding, backend selection, or an upstream response. The message and backend log field distinguish them.
Streaming responses terminate at 30 seconds: the global write timeout is unsuitable for WebSockets, server-sent events, and indefinite downloads. Give such traffic a separate server or explicit policy.
Shutdown exceeds ten seconds: an accepted request is still blocked. Align upstream response limits, server timeouts, frontend drain time, and the systemd stop budget.

Final verification checklist

Both upstream health URLs return a bounded 2xx response.
Normal requests alternate across healthy backends.
A stopped backend disappears without stopping the proxy.
Repeated 5xx or transport failures open only that backend’s circuit.
Capacity exhaustion returns immediate 503 responses.
Logs contain request IDs, durations, statuses, and backend identities without secrets.
SIGTERM drains accepted requests within the shutdown budget.
Only the intended frontend can reach proxy and backend ports.

A resilient proxy is not defined by how many requests it can accept. It is defined by how carefully it refuses work, how narrowly it contains failure, and how clearly it explains every decision afterward. Those properties turn a thin networking component into a dependable production boundary.