A production shell is not merely a remote terminal. It is a control plane for deployments, secrets, processes, and data. Protecting it with a long-lived public key solves only one problem: proving possession of a file that may have been copied years ago.

A stronger design uses short-lived SSH certificates for identity, a separate one-time password for multifactor authentication, narrowly scoped administrative commands, and audit records that connect a login to its privileged effects. The result is not invulnerability. It is a system with bounded credentials, explicit authority, and enough evidence to investigate mistakes or abuse.

This tutorial builds that system on an Ubuntu Server host named srv01.internal. Administrators reach it through the trusted VPN subnet 10.20.0.0/24. The examples use the account alice, UID 1101, and an application service named myapp.service.

Architecture and security boundaries

An encrypted, offline Ed25519 certificate-authority key signs administrators’ public keys. The server stores only the CA public key and accepts certificates containing the principal prod-admin. Certificates last eight hours, so ordinary access naturally expires without editing every server.

OpenSSH then requires a second authentication method through PAM. A time-based one-time password is useful here because it is independent of the SSH key, although it is not phishing-resistant. The TOTP seed also resides on the server, so a complete host compromise defeats that factor. For higher-assurance environments, place the MFA boundary on a hardened access gateway or use a security-key-backed design.

Authorization remains local. Membership in prod-admins permits login, while sudo allows only three exact operational commands. OpenSSH logs authentication, Linux Audit records executed programs, and sudo captures terminal input and output for privileged commands.

Certificates answer who may connect and for how long. MFA raises the cost of stolen credentials. Sudo determines what the authenticated person may do. Auditing records what actually happened. None of these layers substitutes for another.

Prerequisites and project structure

You need an Ubuntu Server host with console or existing administrative access, working time synchronization, a trusted VPN or management network, and a separate Linux administration workstation. Keep the current SSH session open throughout deployment. Do not close it until a new session has passed every verification step.

The completed host uses these files:

• /etc/ssh/ca/user_ca.pub — trusted CA public key
• /etc/ssh/auth_principals/alice — certificate principals accepted for Alice
• /etc/ssh/sshd_config.d/40-production-admin.conf — SSH hardening policy
• /etc/pam.d/sshd — dedicated SSH OTP authentication stack
• /etc/sudoers.d/alice-myapp — least-privilege commands
• /etc/audit/rules.d/50-admin-session.rules — executable audit rules
• /etc/systemd/journald.conf.d/40-persistent-audit.conf — bounded persistent journal storage

On srv01.internal, install the required distribution packages:

sudo apt update
sudo apt install openssh-server sudo auditd audispd-plugins \
  libpam-google-authenticator ufw

sudo systemctl enable --now ssh
sudo systemctl enable --now auditd
timedatectl status

Correct substantial clock drift before continuing. TOTP validation and certificate validity both depend on time.

Create and protect the SSH certificate authority

Run this section on the administration workstation, not on the server. The CA private key should live on encrypted removable storage kept offline except while issuing or revoking certificates. Its passphrase protects a stolen copy; it does not protect a key left mounted on an unlocked workstation.

umask 077
sudo install -d -m 0700 /media/secure-ssh-ca
sudo chown "$(id -u):$(id -g)" /media/secure-ssh-ca

ssh-keygen -t ed25519 -a 100 \
  -f /media/secure-ssh-ca/user_ca \
  -C "production SSH user CA"

Record certificate serial numbers and identities in a durable issuance ledger. A serial is not secret, but it gives incident responders a stable identifier.

Transfer only user_ca.pub to /root/bootstrap/user_ca.pub on the server through the existing trusted administrative channel. Never copy the private file named user_ca to a managed host.

Prepare the account and trust policy

Back on srv01.internal, first confirm that UID 1101 is unused. The first command should produce no output:

getent passwd 1101
sudo addgroup --system prod-admins
sudo adduser --uid 1101 --disabled-password --gecos "" alice
sudo adduser alice prod-admins

sudo install -d -o root -g root -m 0755 /etc/ssh/ca
sudo install -d -o root -g root -m 0755 /etc/ssh/auth_principals
sudo install -o root -g root -m 0644 \
  /root/bootstrap/user_ca.pub /etc/ssh/ca/user_ca.pub

If the UID lookup returns an account, allocate a different fixed UID and use it consistently in the audit rules below.

Create /etc/ssh/auth_principals/alice with this single line:

prod-admin

Set it to root ownership and mode 0644. Separating the certificate principal from the Unix username lets the CA express a role while each server decides which local account may assume it.

Enroll the independent OTP factor

Run enrollment from the trusted console or existing administrative session:

sudo -H -u alice google-authenticator \
  -t -d -f -r 3 -R 30 -w 3

sudo chown alice:alice /home/alice/.google_authenticator
sudo chmod 0400 /home/alice/.google_authenticator

Alice must capture the displayed secret in her authenticator and store the emergency codes in a separate protected location. The options select time-based tokens, reject token reuse, limit attempts, and tolerate a small clock window.

Before changing PAM, preserve a root-owned backup appropriate to this deployment. Then make /etc/pam.d/sshd contain:

auth required pam_google_authenticator.so

account required pam_nologin.so
@include common-account

session required pam_loginuid.so
@include common-session

This intentionally excludes common-auth. Keyboard-interactive authentication asks only for the OTP; it does not quietly re-enable account passwords. Account and session policy still use Ubuntu’s standard PAM stacks.

Require certificates and MFA in OpenSSH

Create /etc/ssh/sshd_config.d/40-production-admin.conf as root:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication yes
PubkeyAuthentication yes
UsePAM yes

TrustedUserCAKeys /etc/ssh/ca/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
AuthenticationMethods publickey,keyboard-interactive:pam
AllowGroups prod-admins

DisableForwarding yes
PermitTunnel no
X11Forwarding no
LoginGraceTime 30
MaxAuthTries 3
MaxSessions 2
MaxStartups 10:30:30
ClientAliveInterval 300
ClientAliveCountMax 2
LogLevel VERBOSE

DisableForwarding closes agent, TCP, Unix-socket, and X11 forwarding paths. That is desirable for an administrative shell but unsuitable if this host is deliberately used as a bastion or tunnel endpoint. In that case, expose only the required forwarding capability and restrict destinations separately.

Validate both syntax and the effective policy before reloading:

sudo sshd -t
sudo sshd -T -C user=alice,host=srv01.internal,addr=10.20.0.10 \
  | grep -E 'authenticationmethods|trustedusercakeys|authorizedprincipalsfile|disableforwarding'

sudo systemctl reload ssh
sudo systemctl is-active ssh

A failed sshd -t is a stop condition. Fix the reported file and line; do not reload experimentally.

Issue a short-lived administrator certificate

On Alice’s workstation, create a dedicated private key:

umask 077
ssh-keygen -t ed25519 -a 100 \
  -f "$HOME/.ssh/id_ed25519_prod" \
  -C "alice production access"

Move only id_ed25519_prod.pub to the offline signing workstation. After verifying the requester and fingerprint through your organization’s enrollment process, mount the CA and sign it:

CERT_SERIAL=10001

ssh-keygen -s /media/secure-ssh-ca/user_ca \
  -I "alice-$(date -u +%Y%m%dT%H%M%SZ)" \
  -z "$CERT_SERIAL" \
  -n prod-admin \
  -V -5m:+8h \
  -O clear \
  -O permit-pty \
  id_ed25519_prod.pub

ssh-keygen -Lf id_ed25519_prod-cert.pub

The five-minute backward allowance absorbs small clock differences. -O clear removes default certificate permissions, after which only terminal allocation is restored. Server-side forwarding restrictions remain authoritative.

Return id_ed25519_prod-cert.pub to Alice as ~/.ssh/id_ed25519_prod-cert.pub. The private key never leaves her workstation.

Apply least privilege with auditable sudo

Alice’s account has no password, so unrestricted sudo is neither useful nor intended. Create /etc/sudoers.d/alice-myapp with exact commands:

Defaults:alice use_pty,log_input,log_output
Defaults:alice iolog_dir="/var/log/sudo-io"

alice ALL=(root) NOPASSWD: /usr/bin/systemctl reload myapp.service
alice ALL=(root) NOPASSWD: /usr/bin/systemctl restart myapp.service
alice ALL=(root) NOPASSWD: /usr/bin/journalctl --unit=myapp.service --since=-30min --no-pager

Set mode 0440, then run sudo visudo -cf /etc/sudoers.d/alice-myapp. Argument matching is exact: changing the unit, option order, or time range is denied. Avoid granting interpreters, editors, shells, unrestricted journalctl, or broadly parameterized service commands; they commonly become indirect root shells.

Sudo I/O logs may contain sensitive application output or typed input. Restrict root access, define retention, and never ask operators to enter production secrets into recorded terminal programs.

Build the audit trail

Create /etc/audit/rules.d/50-admin-session.rules:

-a always,exit -F arch=b64 -S execve -F auid=1101 -k admin-exec
-a always,exit -F arch=b32 -S execve -F auid=1101 -k admin-exec

Load and inspect the rules:

sudo augenrules --load
sudo auditctl -l
sudo auditctl -s
sudo ausearch -k admin-exec -i

The login UID, or auid, follows Alice through sudo, connecting privileged processes to the originating login. Audit captures execve calls and arguments, not shell built-ins, terminal output, or every file mutation. Sudo I/O logging fills part of that gap for privileged commands.

For persistent SSH and service logs, create /etc/systemd/journald.conf.d/40-persistent-audit.conf:

[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=1G
MaxRetentionSec=30day

Create /var/log/journal with sudo systemd-tmpfiles --create --prefix /var/log/journal, then restart systemd-journald. Local logs help with operations, but an attacker who gains root may alter them. Production deployments should forward SSH, sudo, and audit events to a separately administered collector.

Firewall and staged deployment

Confirm that 10.20.0.0/24 is truly the VPN subnet for this deployment. Then stage the firewall rule:

sudo ufw allow from 10.20.0.0/24 to any port 22 proto tcp
sudo ufw status numbered

If UFW is not already active, enable it only from console access after accounting for every other required service. Enabling a default-deny firewall remotely without complete rules can disconnect both SSH and the application.

Roll out to one canary host first. Keep the bootstrap session open, authenticate through the VPN in a second terminal, exercise sudo, inspect all three audit layers, and only then repeat the configuration through versioned automation.

Test authentication, authorization, and evidence

From Alice’s workstation:

ssh -o IdentitiesOnly=yes \
  -i "$HOME/.ssh/id_ed25519_prod" \
  [email protected]

sudo -l
sudo /usr/bin/systemctl reload myapp.service
sudo /usr/bin/journalctl --unit=myapp.service --since=-30min --no-pager
sudo /bin/bash

The connection must require both the private key and OTP. The first three sudo commands should succeed where applicable; the shell must be denied. Also test negative paths: omit the certificate, enter an invalid OTP, connect outside the VPN, and use a certificate with the wrong principal. Every case should fail closed.

From the retained administrative session, correlate the resulting records:

sudo journalctl -u ssh --since "15 minutes ago"
sudo ausearch -k admin-exec -ua 1101 -i
sudo sudoreplay -l user alice
sudo ls -l /var/log/sudo-io

Common failures and operational trade-offs

• Certificate rejected: compare ssh-keygen -Lf output with the server’s accepted principal, validity window, and trusted CA fingerprint.
• No OTP prompt: confirm the client actually offered the certificate and that AuthenticationMethods appears in effective sshd -T output.
• Every OTP fails: inspect time synchronization and the ownership and permissions of .google_authenticator.
• Sudo command denied: compare the executable path and complete argument sequence with sudo -l. Exact matching is deliberate.
• Missing audit events: verify Alice’s UID, loaded audit rules, and the audit daemon’s status. Existing sessions retain their original login UID.
• Excessive log volume: keep audit rules scoped to administrative identities, monitor disk use, and enforce retention. System-wide execve capture can be expensive on busy hosts.

Short certificates reduce exposure but introduce an availability dependency on the issuance process. Keep the CA offline, document an emergency issuance procedure, and maintain revocation instructions for incidents that cannot wait for expiry. Test revocation artifacts on a canary: an unreadable file configured through RevokedKeys can cause public-key authentication to fail broadly.

Final verification checklist

• The CA private key exists only on encrypted offline media.
• The server trusts only the CA public key and the intended principal.
• Certificates have unique identities, recorded serials, and short validity.
• Root login, passwords, and unnecessary forwarding are disabled.
• SSH requires both a valid certificate and OTP.
• Only the VPN subnet can reach port 22.
• Sudo grants exact operational commands rather than a general shell.
• SSH journal entries, audit events, and sudo replays are retrievable.
• Retention and off-host forwarding match the sensitivity of recorded data.
• Positive and negative tests pass before the bootstrap session is closed.

Robust remote administration comes from making authority temporary, narrow, and visible. A copied key should expire quickly. A stolen factor should be insufficient. A routine deployment should not imply a root shell. And when something goes wrong, the evidence should describe the path from identity to action. That is the difference between merely allowing SSH and operating a defensible production control plane.