Production-Ready Go REST API: Graceful Shutdown, Validation, and Integration Testing

Mihajlo
Aug 8, 2026
Tutorials

A REST API is not production-ready merely because it returns JSON. The difficult behavior appears at the edges: malformed bodies, slow clients, concurrent requests, deployment signals, overloaded dependencies, and tests that must exercise the real HTTP stack rather than call handlers directly.

This tutorial builds a compact task API using Go 1.22 and the standard library. It provides strict validation, bounded HTTP timeouts, structured logs, health and readiness probes, lightweight metrics, graceful shutdown, panic recovery, and integration tests over real TCP connections.

Prerequisites and design boundaries

You need Go 1.22 or newer, a Unix-like shell, and permission to bind a local TCP port. The application stores tasks in memory so the example remains focused on HTTP reliability. Its process behavior is production-oriented, but its data is intentionally ephemeral: restarting the service loses tasks, and multiple replicas do not share state. A durable deployment should replace the map with a database-backed repository.

The server exposes these endpoints:

POST /v1/tasks validates and creates a task.
GET /v1/tasks/{id} returns one task.
GET /healthz reports whether the process can serve HTTP.
GET /readyz returns failure as soon as shutdown begins.
GET /metrics exposes process-local request counters.

Readiness and liveness are deliberately separate. A terminating process may remain alive while draining requests, but it should leave the load balancer before that drain begins.

Create the project

mkdir -p production-api/cmd/api
cd production-api
go mod init example.com/production-api

The resulting structure stays small enough to audit:

production-api/
├── go.mod
└── cmd/
    └── api/
        ├── main.go
        └── main_test.go

Implement the HTTP service

Place the following code in cmd/api/main.go. Go 1.22 method-aware routing gives us automatic method rejection without an external router.

package main

import (
	"context"
	"encoding/json"
	"errors"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"strconv"
	"strings"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
	"unicode/utf8"
)

type config struct {
	addr            string
	shutdownTimeout time.Duration
}

type task struct {
	ID        int64     `json:"id"`
	Title     string    `json:"title"`
	CreatedAt time.Time `json:"created_at"`
}

type metrics struct {
	requests atomic.Uint64
	failures atomic.Uint64
	inFlight atomic.Int64
	sequence atomic.Uint64
}

type app struct {
	logger       *slog.Logger
	mu           sync.RWMutex
	tasks        map[int64]task
	nextID       int64
	shuttingDown atomic.Bool
	metrics      metrics
}

type responseRecorder struct {
	http.ResponseWriter
	status int
	bytes  int
}

func (r *responseRecorder) WriteHeader(status int) {
	if r.status != 0 {
		return
	}
	r.status = status
	r.ResponseWriter.WriteHeader(status)
}

func (r *responseRecorder) Write(body []byte) (int, error) {
	if r.status == 0 {
		r.WriteHeader(http.StatusOK)
	}
	n, err := r.ResponseWriter.Write(body)
	r.bytes += n
	return n, err
}

func newApp(logger *slog.Logger) *app {
	return &app{
		logger: logger,
		tasks:  make(map[int64]task),
	}
}

func (a *app) routes() http.Handler {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /healthz", a.health)
	mux.HandleFunc("GET /readyz", a.ready)
	mux.HandleFunc("GET /metrics", a.serveMetrics)
	mux.HandleFunc("POST /v1/tasks", a.createTask)
	mux.HandleFunc("GET /v1/tasks/{id}", a.getTask)
	return a.observe(mux)
}

func (a *app) observe(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		started := time.Now()
		requestID := strconv.FormatUint(a.metrics.sequence.Add(1), 10)
		w.Header().Set("X-Request-ID", requestID)

		rec := &responseRecorder{ResponseWriter: w}
		a.metrics.inFlight.Add(1)

		defer func() {
			if value := recover(); value != nil {
				a.logger.Error("request panic",
					"request_id", requestID,
					"panic", fmt.Sprint(value))
				if rec.status == 0 {
					writeError(rec, http.StatusInternalServerError, "internal server error")
				}
			}

			if rec.status == 0 {
				rec.status = http.StatusOK
			}
			a.metrics.inFlight.Add(-1)
			a.metrics.requests.Add(1)
			if rec.status >= 500 {
				a.metrics.failures.Add(1)
			}

			a.logger.Info("request completed",
				"request_id", requestID,
				"method", r.Method,
				"path", r.URL.Path,
				"status", rec.status,
				"bytes", rec.bytes,
				"duration_ms", time.Since(started).Milliseconds(),
				"remote_addr", r.RemoteAddr)
		}()

		next.ServeHTTP(rec, r)
	})
}

func (a *app) health(w http.ResponseWriter, _ *http.Request) {
	writeJSON(w, http.StatusOK, map[string]string{"status": "up"})
}

func (a *app) ready(w http.ResponseWriter, _ *http.Request) {
	if a.shuttingDown.Load() {
		writeError(w, http.StatusServiceUnavailable, "server is shutting down")
		return
	}
	writeJSON(w, http.StatusOK, map[string]string{"status": "ready"})
}

func (a *app) serveMetrics(w http.ResponseWriter, _ *http.Request) {
	w.Header().Set("Content-Type", "text/plain; charset=utf-8")
	w.Header().Set("Cache-Control", "no-store")
	fmt.Fprintf(w, "api_http_requests_total %d\n", a.metrics.requests.Load())
	fmt.Fprintf(w, "api_http_failures_total %d\n", a.metrics.failures.Load())
	fmt.Fprintf(w, "api_http_requests_in_flight %d\n", a.metrics.inFlight.Load())
}

func (a *app) createTask(w http.ResponseWriter, r *http.Request) {
	r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
	defer r.Body.Close()

	var input struct {
		Title string `json:"title"`
	}

	decoder := json.NewDecoder(r.Body)
	decoder.DisallowUnknownFields()

	if err := decoder.Decode(&input); err != nil {
		writeError(w, http.StatusBadRequest, "body must contain one valid JSON object")
		return
	}
	if err := decoder.Decode(&struct{}{}); !errors.Is(err, io.EOF) {
		writeError(w, http.StatusBadRequest, "body must contain exactly one JSON object")
		return
	}

	input.Title = strings.TrimSpace(input.Title)
	length := utf8.RuneCountInString(input.Title)
	if length == 0 || length > 120 {
		writeError(w, http.StatusUnprocessableEntity,
			"title must contain between 1 and 120 characters")
		return
	}

	a.mu.Lock()
	a.nextID++
	created := task{
		ID:        a.nextID,
		Title:     input.Title,
		CreatedAt: time.Now().UTC(),
	}
	a.tasks[created.ID] = created
	a.mu.Unlock()

	w.Header().Set("Location", fmt.Sprintf("/v1/tasks/%d", created.ID))
	writeJSON(w, http.StatusCreated, created)
}

func (a *app) getTask(w http.ResponseWriter, r *http.Request) {
	id, err := strconv.ParseInt(r.PathValue("id"), 10, 64)
	if err != nil || id < 1 {
		writeError(w, http.StatusBadRequest, "task id must be a positive integer")
		return
	}

	a.mu.RLock()
	found, ok := a.tasks[id]
	a.mu.RUnlock()

	if !ok {
		writeError(w, http.StatusNotFound, "task not found")
		return
	}
	writeJSON(w, http.StatusOK, found)
}

func writeError(w http.ResponseWriter, status int, message string) {
	writeJSON(w, status, map[string]string{"error": message})
}

func writeJSON(w http.ResponseWriter, status int, value any) {
	w.Header().Set("Content-Type", "application/json; charset=utf-8")
	w.Header().Set("Cache-Control", "no-store")
	w.WriteHeader(status)
	_ = json.NewEncoder(w).Encode(value)
}

func durationFromEnv(name string, fallback time.Duration) (time.Duration, error) {
	value := os.Getenv(name)
	if value == "" {
		return fallback, nil
	}
	parsed, err := time.ParseDuration(value)
	if err != nil || parsed <= 0 {
		return 0, fmt.Errorf("%s must be a positive duration", name)
	}
	return parsed, nil
}

func loadConfig() (config, error) {
	cfg := config{addr: os.Getenv("ADDR")}
	if cfg.addr == "" {
		cfg.addr = ":8080"
	}

	var err error
	cfg.shutdownTimeout, err = durationFromEnv("SHUTDOWN_TIMEOUT", 10*time.Second)
	return cfg, err
}

func run() error {
	cfg, err := loadConfig()
	if err != nil {
		return err
	}

	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
	application := newApp(logger)

	server := &http.Server{
		Addr:              cfg.addr,
		Handler:           application.routes(),
		ReadHeaderTimeout: 2 * time.Second,
		ReadTimeout:       5 * time.Second,
		WriteTimeout:      10 * time.Second,
		IdleTimeout:       60 * time.Second,
		MaxHeaderBytes:    1 << 20,
		ErrorLog:          slog.NewLogLogger(logger.Handler(), slog.LevelError),
	}

	signalContext, stop := signal.NotifyContext(
		context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	serverErrors := make(chan error, 1)
	go func() {
		logger.Info("server starting", "addr", cfg.addr)
		serverErrors <- server.ListenAndServe()
	}()

	select {
	case err := <-serverErrors:
		if errors.Is(err, http.ErrServerClosed) {
			return nil
		}
		return fmt.Errorf("serve HTTP: %w", err)

	case <-signalContext.Done():
		stop()
		application.shuttingDown.Store(true)
		logger.Info("shutdown started",
			"timeout", cfg.shutdownTimeout.String())

		shutdownContext, cancel := context.WithTimeout(
			context.Background(), cfg.shutdownTimeout)
		defer cancel()

		if err := server.Shutdown(shutdownContext); err != nil {
			closeErr := server.Close()
			if closeErr != nil {
				return fmt.Errorf("shutdown: %v; force close: %w", err, closeErr)
			}
			return fmt.Errorf("graceful shutdown: %w", err)
		}

		err := <-serverErrors
		if !errors.Is(err, http.ErrServerClosed) {
			return fmt.Errorf("server stopped unexpectedly: %w", err)
		}
		logger.Info("shutdown completed")
		return nil
	}
}

func main() {
	if err := run(); err != nil {
		slog.Error("application stopped", "error", err)
		os.Exit(1)
	}
}

Why these boundaries matter

ReadHeaderTimeout limits slow header attacks, while ReadTimeout separately bounds reading the complete request. A connection timeout would not bound later reads or database queries. The one-megabyte body limit protects memory before JSON decoding, and DisallowUnknownFields catches client spelling mistakes instead of silently discarding data.

The write and shutdown budgets are also distinct. A handler gets at most ten seconds to write under this configuration, and shutdown grants the server ten seconds to drain active handlers. If the drain expires, Close terminates remaining connections and the process exits with an error, making the degraded shutdown visible to an orchestrator.

The map lock protects only mutation and lookup. JSON encoding happens after the lock is released, avoiding a slow client turning a short critical section into global contention.

Exercise the API manually

go run ./cmd/api

curl --fail-with-body http://127.0.0.1:8080/readyz

curl --fail-with-body \
  -H 'Content-Type: application/json' \
  -d '{"title":"Review shutdown alerts"}' \
  http://127.0.0.1:8080/v1/tasks

curl --fail-with-body http://127.0.0.1:8080/v1/tasks/1
curl --fail-with-body http://127.0.0.1:8080/metrics

Try an unknown field or a blank title. The former returns 400 Bad Request because the representation is malformed for this API; the latter returns 422 Unprocessable Entity because the JSON is valid but violates a business constraint.

Add integration tests

Handler-only tests can miss routing, connection, and shutdown behavior. Save this as cmd/api/main_test.go. The first test uses an HTTP test server; the second opens a real listener and proves that shutdown waits for an active request.

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log/slog"
	"net"
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
	"time"
)

func testApp() *app {
	return newApp(slog.New(slog.NewTextHandler(io.Discard, nil)))
}

func TestCreateAndFetchTask(t *testing.T) {
	server := httptest.NewServer(testApp().routes())
	defer server.Close()

	client := &http.Client{Timeout: 2 * time.Second}
	response, err := client.Post(
		server.URL+"/v1/tasks",
		"application/json",
		bytes.NewBufferString(`{"title":"  Ship safely  "}`),
	)
	if err != nil {
		t.Fatal(err)
	}
	defer response.Body.Close()

	if response.StatusCode != http.StatusCreated {
		t.Fatalf("create status: got %d", response.StatusCode)
	}

	var created task
	if err := json.NewDecoder(response.Body).Decode(&created); err != nil {
		t.Fatal(err)
	}
	if created.Title != "Ship safely" || created.ID != 1 {
		t.Fatalf("unexpected task: %+v", created)
	}

	response, err = client.Get(server.URL + response.Header.Get("Location"))
	if err != nil {
		t.Fatal(err)
	}
	defer response.Body.Close()

	if response.StatusCode != http.StatusOK {
		t.Fatalf("fetch status: got %d", response.StatusCode)
	}
}

func TestRejectsUnknownFields(t *testing.T) {
	server := httptest.NewServer(testApp().routes())
	defer server.Close()

	response, err := server.Client().Post(
		server.URL+"/v1/tasks",
		"application/json",
		strings.NewReader(`{"title":"valid","priority":1}`),
	)
	if err != nil {
		t.Fatal(err)
	}
	defer response.Body.Close()

	if response.StatusCode != http.StatusBadRequest {
		t.Fatalf("got %d, want 400", response.StatusCode)
	}
}

func TestShutdownDrainsActiveRequest(t *testing.T) {
	application := testApp()
	started := make(chan struct{})
	release := make(chan struct{})

	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		close(started)
		<-release
		application.routes().ServeHTTP(w, r)
	})

	listener, err := net.Listen("tcp", "127.0.0.1:0")
	if err != nil {
		t.Fatal(err)
	}

	server := &http.Server{Handler: handler}
	go func() { _ = server.Serve(listener) }()

	requestDone := make(chan error, 1)
	go func() {
		response, err := http.Get("http://" + listener.Addr().String() + "/healthz")
		if err == nil {
			defer response.Body.Close()
			if response.StatusCode != http.StatusOK {
				err = fmt.Errorf("unexpected status %d", response.StatusCode)
			}
		}
		requestDone <- err
	}()

	<-started
	shutdownDone := make(chan error, 1)
	go func() {
		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
		defer cancel()
		shutdownDone <- server.Shutdown(ctx)
	}()

	select {
	case err := <-shutdownDone:
		t.Fatalf("shutdown returned before request drained: %v", err)
	case <-time.After(50 * time.Millisecond):
	}

	close(release)

	if err := <-requestDone; err != nil {
		t.Fatal(err)
	}
	if err := <-shutdownDone; err != nil {
		t.Fatal(err)
	}
}

Run formatting, static analysis, tests, and the race detector:

gofmt -w cmd/api/main.go cmd/api/main_test.go
go vet ./...
go test -race ./...

Deploy with systemd

Build on the target architecture, inspect any existing destination before replacing it, and install with root privileges:

mkdir -p dist
go build -trimpath -ldflags='-s -w' -o ./dist/task-api ./cmd/api
sudo install -o root -g root -m 0755 ./dist/task-api /usr/local/bin/task-api

Create /etc/systemd/system/task-api.service as root:

[Unit]
Description=Production task API
After=network.target

[Service]
Type=simple
DynamicUser=yes
ExecStart=/usr/local/bin/task-api
Environment=ADDR=127.0.0.1:8080
Environment=SHUTDOWN_TIMEOUT=10s
Restart=on-failure
RestartSec=2s
KillSignal=SIGTERM
TimeoutStopSec=15s
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
RestrictAddressFamilies=AF_INET AF_INET6
MemoryDenyWriteExecute=yes

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now task-api.service
sudo systemctl status task-api.service
sudo journalctl -u task-api.service --since today

The service binds only to loopback. Put a TLS-terminating reverse proxy on the same host or a private ingress in front of it. Do not expose the plaintext listener or unauthenticated /metrics endpoint to the public Internet. Configure the host firewall to permit only the intended proxy or management paths.

Observability, security, and performance

Structured logs carry request IDs, latency, response size, status, and peer address. The server generates its own monotonically increasing request identifier instead of trusting arbitrary client input. Across multiple replicas, let the ingress attach a globally unique trace identifier and validate it before propagation.

The metrics are deliberately dependency-free. A larger service should expose a monitoring system’s standard format and add bounded labels such as method and route. Never label metrics with raw paths, task IDs, request IDs, or user input; those values create unbounded cardinality.

Authentication and authorization are outside this small domain model, but they belong before business handlers. Terminate TLS at a controlled proxy, limit request rates there, redact secrets from logs, and pass authenticated identity through a validated mechanism. If a database replaces the map, apply separate connection and query deadlines; establishing a connection does not limit query execution.

The in-memory map also grows without bound. That is acceptable for this operational skeleton, not for an unlimited production workload. A durable repository should add pagination, storage quotas, database indexes, and explicit query timeouts. Load testing should measure tail latency and memory under representative body sizes rather than celebrate a single requests-per-second number.

Common failure modes

Readiness remains successful during termination: the load balancer continues sending new work while old requests drain. Set the shutdown flag before calling Shutdown.
Shutdown never completes: a handler ignores cancellation or a dependency lacks a deadline. Give each outbound call a timeout shorter than the overall shutdown budget.
Validation accepts extra JSON: decoding only once permits trailing objects. Perform the second decode and require io.EOF.
Timeouts are copied blindly: large uploads or streaming responses may legitimately exceed these limits. Choose budgets from endpoint behavior, not convention.
Metrics disappear on restart: these counters are process-local. A monitoring system must scrape and retain them externally.
Deployments lose tasks: the example repository is memory-backed. Use durable storage before treating task data as persistent.

Final verification checklist

go vet ./... and go test -race ./... pass.
Valid creation returns 201, a Location header, and normalized JSON.
Malformed, oversized, unknown-field, and semantically invalid bodies are rejected.
Health, readiness, logs, request IDs, and metrics behave as documented.
A SIGTERM removes readiness, drains active requests, and exits within the configured budget.
The listener is private or protected by TLS, authentication, rate limits, and firewall policy.
Deployment and proxy stop budgets exceed the application’s ten-second graceful-shutdown budget.

Production reliability is rarely one dramatic feature. It is the accumulation of small, explicit boundaries: one JSON object, one body limit, one timeout for each phase, one honest readiness state, and one tested shutdown contract. When those boundaries are visible in code and executable in tests, an API stops being merely functional and starts becoming dependable.