Secure Go Services: mTLS, Key Rotation, and Zero-Trust Authorization in Practice
A service can encrypt every byte on the wire and still trust the wrong caller. Production security begins when transport identity and application authorization are treated as separate controls: mutual TLS proves which certificate a peer owns, while authorization decides what that identity may do.
This tutorial builds a Go 1.22 service that requires client certificates, authorizes an exact URI identity, reloads server certificates and private keys without downtime, and shuts down cleanly. The companion client uses the same trust model and deliberately avoids insecure shortcuts such as InsecureSkipVerify.
Architecture and security boundaries
The example has one private certificate authority, an HTTPS server on 127.0.0.1:8443, and a client identified by the URI SAN spiffe://workshop.internal/service/catalog-client. The URI resembles a SPIFFE identifier, but this tutorial does not claim conformance with the SPIFFE workload API.
The TLS layer performs three jobs:
- It encrypts traffic with TLS 1.3.
- It verifies that the server certificate represents
localhost. - It requires a client certificate issued by the configured CA.
After the handshake, the HTTP middleware applies an explicit allowlist. A certificate from the correct CA is necessary but not sufficient: an authenticated yet unauthorized workload receives 403 Forbidden.
The private CA used below is suitable for a contained exercise, not for distributing production credentials. A real deployment should use an internal CA, short-lived workload certificates, protected signing keys, and an automated issuance system. Leaf rotation and CA rotation are also different operations. Leaf rotation can be immediate; CA rotation normally needs an overlap period in which both old and new roots are trusted.
Prerequisites and project layout
You need Go 1.22 or newer, OpenSSL 3, and a Unix-like shell. The service listens on an unprivileged port, so neither compilation nor execution requires root access.
secure-go-service/
├── go.mod
├── main.go
└── pki/
├── ca.crt
├── ca.key
├── server.crt
├── server.key
├── client.crt
└── client.keyCreate a fresh working directory, then add this module file:
module example.com/secure-go-service
go 1.22Issue narrowly scoped certificates
The following commands create a private CA and 30-day leaf certificates. Run them only in a new project directory: mkdir pki intentionally fails if that path already exists, preventing accidental reuse or replacement of an existing PKI directory.
umask 077
mkdir -m 0700 pki
openssl genpkey -algorithm EC \
-pkeyopt ec_paramgen_curve:P-256 \
-out pki/ca.key
openssl req -x509 -new -sha256 -days 3650 \
-key pki/ca.key \
-subj "/CN=Workshop Root CA" \
-addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-out pki/ca.crt
openssl req -new -newkey ec \
-pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout pki/server.key \
-subj "/CN=inventory.internal" \
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
-addext "basicConstraints=critical,CA:FALSE" \
-addext "keyUsage=critical,digitalSignature" \
-addext "extendedKeyUsage=serverAuth" \
-out pki/server.csr
openssl x509 -req -sha256 -days 30 \
-in pki/server.csr \
-CA pki/ca.crt -CAkey pki/ca.key -CAcreateserial \
-copy_extensions copy \
-out pki/server.crt
openssl req -new -newkey ec \
-pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout pki/client.key \
-subj "/CN=catalog-client" \
-addext "subjectAltName=URI:spiffe://workshop.internal/service/catalog-client" \
-addext "basicConstraints=critical,CA:FALSE" \
-addext "keyUsage=critical,digitalSignature" \
-addext "extendedKeyUsage=clientAuth" \
-out pki/client.csr
openssl x509 -req -sha256 -days 30 \
-in pki/client.csr \
-CA pki/ca.crt -CAkey pki/ca.key -CAserial pki/ca.srl \
-copy_extensions copy \
-out pki/client.crt
openssl verify -CAfile pki/ca.crt -purpose sslserver pki/server.crt
openssl verify -CAfile pki/ca.crt -purpose sslclient pki/client.crt
rm pki/server.csr pki/client.csrThe server certificate has DNS and IP SANs because hostname validation does not fall back to the common name. The client identity lives in a URI SAN rather than a mutable HTTP header.
Implement the service and client
Place the following program in main.go. Server reloads are transactional at the process level: files are parsed and validated into a new immutable tls.Config, then an atomic pointer is swapped. A malformed replacement leaves the previous configuration active.
package main
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"flag"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"os"
"os/signal"
"strings"
"sync/atomic"
"syscall"
"time"
)
const allowedIdentity = "spiffe://workshop.internal/service/catalog-client"
type identityKey struct{}
func loadServerTLS(certFile, keyFile, caFile string) (*tls.Config, error) {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, fmt.Errorf("load server key pair: %w", err)
}
pem, err := os.ReadFile(caFile)
if err != nil {
return nil, fmt.Errorf("read client CA: %w", err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, errors.New("client CA contains no certificates")
}
return &tls.Config{
MinVersion: tls.VersionTLS13,
Certificates: []tls.Certificate{cert},
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: pool,
}, nil
}
func loadClientTLS(certFile, keyFile, caFile string) (*tls.Config, error) {
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, fmt.Errorf("load client key pair: %w", err)
}
pem, err := os.ReadFile(caFile)
if err != nil {
return nil, fmt.Errorf("read server CA: %w", err)
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, errors.New("server CA contains no certificates")
}
return &tls.Config{
MinVersion: tls.VersionTLS13,
Certificates: []tls.Certificate{cert},
RootCAs: pool,
}, nil
}
func authorize(next http.Handler, logger *slog.Logger) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
http.Error(w, "client certificate required", http.StatusUnauthorized)
return
}
identity := ""
for _, uri := range r.TLS.PeerCertificates[0].URIs {
if uri.String() == allowedIdentity {
identity = uri.String()
break
}
}
if identity == "" {
logger.Warn("authorization denied",
"remote", r.RemoteAddr, "path", r.URL.Path)
http.Error(w, "forbidden", http.StatusForbidden)
return
}
ctx := context.WithValue(r.Context(), identityKey{}, identity)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
func observe(next http.Handler, logger *slog.Logger) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
next.ServeHTTP(w, r)
logger.Info("request completed",
"method", r.Method,
"path", r.URL.Path,
"identity", r.Context().Value(identityKey{}),
"duration_ms", time.Since(start).Milliseconds())
})
}
func runServer(ctx context.Context, addr, certFile, keyFile, caFile string) error {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
initial, err := loadServerTLS(certFile, keyFile, caFile)
if err != nil {
return err
}
var active atomic.Pointer[tls.Config]
active.Store(initial)
dispatcher := &tls.Config{
MinVersion: tls.VersionTLS13,
GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
return active.Load(), nil
},
}
mux := http.NewServeMux()
mux.HandleFunc("GET /v1/inventory", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
_, _ = io.WriteString(w, `{"service":"inventory","status":"ready"}`+"\n")
})
server := &http.Server{
Addr: addr,
Handler: authorize(observe(mux, logger), logger),
TLSConfig: dispatcher,
ReadHeaderTimeout: 3 * time.Second,
ReadTimeout: 5 * time.Second,
WriteTimeout: 5 * time.Second,
IdleTimeout: 30 * time.Second,
MaxHeaderBytes: 1 << 20,
ErrorLog: slog.NewLogLogger(logger.Handler(), slog.LevelWarn),
}
hup := make(chan os.Signal, 1)
signal.Notify(hup, syscall.SIGHUP)
defer signal.Stop(hup)
go func() {
for {
select {
case <-ctx.Done():
return
case <-hup:
replacement, err := loadServerTLS(certFile, keyFile, caFile)
if err != nil {
logger.Error("TLS reload rejected", "error", err)
continue
}
active.Store(replacement)
logger.Info("TLS configuration reloaded")
}
}
}()
errCh := make(chan error, 1)
go func() {
logger.Info("server starting", "address", addr)
errCh <- server.ListenAndServeTLS("", "")
}()
select {
case err := <-errCh:
if errors.Is(err, http.ErrServerClosed) {
return nil
}
return err
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
if err := server.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("graceful shutdown: %w", err)
}
err := <-errCh
if errors.Is(err, http.ErrServerClosed) {
return nil
}
return err
}
}
func runClient(url, certFile, keyFile, caFile string) error {
tlsConfig, err := loadClientTLS(certFile, keyFile, caFile)
if err != nil {
return err
}
transport := &http.Transport{
TLSClientConfig: tlsConfig,
TLSHandshakeTimeout: 5 * time.Second,
ResponseHeaderTimeout: 5 * time.Second,
IdleConnTimeout: 30 * time.Second,
DialContext: (&net.Dialer{
Timeout: 3 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
}
defer transport.CloseIdleConnections()
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("request failed: %w", err)
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if err != nil {
return err
}
if resp.StatusCode/100 != 2 {
return fmt.Errorf("HTTP %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
fmt.Print(string(body))
return nil
}
func main() {
mode := flag.String("mode", "", "server or client")
addr := flag.String("listen", "127.0.0.1:8443", "server listen address")
url := flag.String("url", "https://localhost:8443/v1/inventory", "client URL")
cert := flag.String("cert", "", "leaf certificate path")
key := flag.String("key", "", "private key path")
ca := flag.String("ca", "", "CA bundle path")
flag.Parse()
if *cert == "" || *key == "" || *ca == "" {
fmt.Fprintln(os.Stderr, "-cert, -key, and -ca are required")
os.Exit(2)
}
ctx, stop := signal.NotifyContext(
context.Background(), os.Interrupt, syscall.SIGTERM,
)
defer stop()
var err error
switch *mode {
case "server":
err = runServer(ctx, *addr, *cert, *key, *ca)
case "client":
err = runClient(*url, *cert, *key, *ca)
default:
err = errors.New("-mode must be server or client")
}
if err != nil {
slog.Error("exiting", "error", err)
os.Exit(1)
}
}Each TLS handshake receives the configuration currently stored in active. Existing connections remain encrypted with their negotiated session until they close; rotation affects new handshakes. That behavior is desirable because terminating every connection during routine rotation creates avoidable disruption.
Build and verify the happy path
mkdir -m 0755 bin
go build -trimpath -o bin/secure-go-service .
./bin/secure-go-service \
-mode server \
-listen 127.0.0.1:8443 \
-cert pki/server.crt \
-key pki/server.key \
-ca pki/ca.crtIn a second terminal, run the client:
./bin/secure-go-service \
-mode client \
-url https://localhost:8443/v1/inventory \
-cert pki/client.crt \
-key pki/client.key \
-ca pki/ca.crtThe response should be {"service":"inventory","status":"ready"}. Test the transport boundary by omitting the client certificate with openssl s_client; the handshake should not produce a usable HTTP session:
openssl s_client \
-connect 127.0.0.1:8443 \
-servername localhost \
-CAfile pki/ca.crt </dev/nullAlso issue a client certificate with a different URI SAN and run the client with it. TLS authentication should succeed because the CA is trusted, but the HTTP request should return 403 Forbidden. This distinction is an important regression test: it proves authorization is not accidentally equivalent to “signed by our CA.”
Rotate the server key without downtime
Create a new private key and certificate in a separate directory. Validate them before touching the active paths.
umask 077
mkdir -m 0700 pki/rotation-1
openssl req -new -newkey ec \
-pkeyopt ec_paramgen_curve:P-256 -nodes \
-keyout pki/rotation-1/server.key \
-subj "/CN=inventory.internal" \
-addext "subjectAltName=DNS:localhost,IP:127.0.0.1" \
-addext "basicConstraints=critical,CA:FALSE" \
-addext "keyUsage=critical,digitalSignature" \
-addext "extendedKeyUsage=serverAuth" \
-out pki/rotation-1/server.csr
openssl x509 -req -sha256 -days 30 \
-in pki/rotation-1/server.csr \
-CA pki/ca.crt -CAkey pki/ca.key -CAserial pki/ca.srl \
-copy_extensions copy \
-out pki/rotation-1/server.crt
openssl verify -CAfile pki/ca.crt \
-purpose sslserver pki/rotation-1/server.crt
openssl x509 -checkend 300 -noout \
-in pki/rotation-1/server.crtAfter validation, retain a recoverable copy of the current pair, install both replacements, and only then send SIGHUP. These moves replace the active files, so execute them from the verified project directory and use a unique backup directory for each rotation.
mkdir -m 0700 pki/backup-rotation-1
cp -p pki/server.crt pki/server.key pki/backup-rotation-1/
install -m 0600 pki/rotation-1/server.key pki/server.key.new
install -m 0600 pki/rotation-1/server.crt pki/server.crt.new
mv pki/server.key.new pki/server.key
mv pki/server.crt.new pki/server.crt
kill -HUP "$(pgrep -n -x secure-go-service)"For a systemd deployment, prefer systemctl reload secure-go with an ExecReload rule instead of process discovery. If loading fails, the log records TLS reload rejected and the old in-memory configuration continues serving new connections.
The one-shot client reads its certificate on every execution, so replacing client.crt and client.key changes the next request. A long-lived client should atomically replace its transport on reload and call CloseIdleConnections; otherwise existing pooled connections can continue using the old authenticated session.
Deployment hardening and operations
Run the binary as a dedicated unprivileged account. The account needs read access to its leaf key and CA bundle, but it should never have access to the CA signing key. Keep ca.key offline or in the issuing system; it does not belong on the service host.
A compact systemd unit can enforce useful boundaries:
[Unit]
Description=Secure Go inventory service
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=securego
Group=securego
ExecStart=/opt/secure-go/bin/secure-go-service -mode server -listen 0.0.0.0:8443 -cert /etc/secure-go/server.crt -key /etc/secure-go/server.key -ca /etc/secure-go/client-ca.crt
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2s
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
ReadOnlyPaths=/etc/secure-go
[Install]
WantedBy=multi-user.targetProvision the account, binary, certificates, ownership, and firewall policy through your normal configuration system. Permit port 8443 only from the workload networks that require it. mTLS limits who can complete a handshake, but network segmentation still reduces scanning, resource exhaustion, and exposure to implementation defects.
Observability that preserves secrets
The program emits structured request, authorization, reload, and TLS error logs. Do not log certificates, private keys, authorization headers, or complete request bodies. Export counters and latency histograms for handshake failures, authorization denials, reload failures, request duration, and active connections.
Certificate expiry deserves a gauge or scheduled alert based on NotAfter. An expiry alert should fire well before the rotation deadline. Remember that HTTP middleware cannot observe failed TLS handshakes because those failures occur before an HTTP request exists; collect them from the server error log or a fronting proxy.
Performance considerations
TLS handshakes consume more CPU than reused HTTP connections, so enable keep-alives and avoid gratuitous reconnects. Conversely, unlimited connection lifetime delays certificate uptake and revocation response. Set an operational maximum connection age in a proxy or client transport when rotation speed matters.
The configured read, write, header, dial, handshake, and overall request timeouts bound different phases. A dial timeout does not bound TLS negotiation or response processing. Measure before tuning cipher or session behavior; TLS 1.3 defaults in Go are generally safer than a hand-maintained cipher list.
Common failure modes
- Hostname mismatch: connecting to a name absent from the server SAN fails even when the issuing CA is trusted.
- Wrong extended key usage: a server-only certificate cannot be used as a client certificate, and vice versa.
- Trusted but unauthorized: a valid certificate with the wrong URI reaches HTTP but receives
403. - Partial rotation: replacing only the key or certificate makes the pair invalid. Stage and validate both before signaling.
- Expired connections masking rotation: established keep-alive connections do not perform a new handshake merely because files changed.
- Root replacement without overlap: switching issuers before every peer trusts the new CA causes an outage. Distribute a bundle containing both roots, rotate leaves, then remove the old root.
- Overbroad identity matching: substring or prefix checks can authorize unintended identities. Compare a canonical, exact URI and keep policy separate from certificate-chain validation.
Final verification checklist
- The server starts with TLS 1.3 and refuses clients without a trusted certificate.
- The expected URI identity receives
200 OK; another CA-signed identity receives403 Forbidden. - Hostname verification succeeds without
InsecureSkipVerify. - Malformed replacement files produce a reload error while the previous configuration remains active.
- A validated new key pair becomes visible on new handshakes after
SIGHUP. - Private leaf keys are readable only by the service account, and the CA signing key is absent from the host.
- Timeouts, graceful shutdown, firewall scope, expiry alerts, and TLS failure logs are configured and tested.
Zero trust is not a product toggle. It is the disciplined composition of narrow network reachability, cryptographic identity, explicit authorization, short-lived credentials, observable failure, and safe rotation. When each boundary can fail closed without turning routine key maintenance into an outage, mTLS becomes an operational control rather than decorative encryption.
