A healthy filesystem can still lose data. A clean SMART report can precede an application-level disaster. An encrypted disk can protect stolen hardware while doing nothing for accidental deletion. Storage reliability comes from layers whose failure modes overlap without pretending to replace one another.

This tutorial builds a production-oriented storage stack around a dedicated 128 GiB data disk: LUKS2 encryption, LVM allocation, an ext4 application volume, short-lived snapshots, off-host backups, SMART monitoring, and routine restore tests. The commands use /dev/vdb in a disposable virtual machine. On physical hardware, use the stable path reported under /dev/disk/by-id/.

Warning: formatting commands below destroy data on /dev/vdb. Rehearse in a virtual machine first. Never copy a device path from an article into production without independently verifying its model, serial number, size, and current mounts.

Architecture and trade-offs

The resulting path is deliberately simple:

physical disk /dev/vdb
└── LUKS2 container: cryptdata
    └── LVM physical volume
        └── volume group: vg_secure
            ├── logical volume: appdata, 64 GiB
            ├── temporary snapshot: appdata_snap, 16 GiB
            └── temporary restore-test volume: restore_test, 48 GiB

Encryption sits below LVM, so logical-volume names and allocation metadata are hidden while the machine is off. LVM provides flexible capacity and snapshots. Ext4 supplies predictable recovery tooling. A separately encrypted, independently mounted backup target at /mnt/backup holds archives after snapshots are removed.

LVM snapshots use copy-on-write storage. They are not backups: they share the original disk, consume capacity as blocks change, and become invalid if their allocated exception space fills. Their job here is to present a stable, short-lived view while an archive is copied elsewhere.

The snapshot is crash-consistent, not automatically application-consistent. Filesystems remain structurally recoverable, but a database may need its own backup command, checkpoint, or brief write quiescence. Do not freeze a busy filesystem for the duration of a large archive.

Prerequisites and project structure

Run host commands with an account that can use sudo. Install the distribution packages providing cryptsetup, LVM2, ext4 tools, GNU tar, gzip, util-linux, and smartmontools. Confirm that /mnt/backup is a different filesystem backed by separate or remote storage.

sudo apt-get update
sudo apt-get install cryptsetup lvm2 e2fsprogs smartmontools acl
sudo install -d -m 0750 /srv/appdata
sudo install -d -m 0750 /mnt/backup
sudo install -d -m 0750 /mnt/restore-test
sudo install -d -m 0750 /usr/local/lib/storage
sudo install -d -m 0750 /var/log/storage

lsblk -o NAME,PATH,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIAL
findmnt --target /mnt/backup

The finished system uses these operational paths:

• /srv/appdata for live application data
• /mnt/backup/appdata for independently protected archives
• /usr/local/lib/storage/backup-appdata for the backup job
• /mnt/restore-test for disposable recovery drills
• /etc/crypttab and /etc/fstab for boot activation

On RPM-based systems, use the corresponding package manager and package names. Package installation is the only distribution-specific part of the core design.

Build the encrypted LVM stack

Prove the target is safe

In the laboratory VM, require the disk to be exactly the expected unmounted block device. These checks are intentionally separate from the destructive command so their output can be reviewed.

DATA_DISK=/dev/vdb

test -b "$DATA_DISK"
lsblk -o NAME,PATH,SIZE,TYPE,FSTYPE,MOUNTPOINTS,MODEL,SERIAL "$DATA_DISK"
sudo wipefs --no-act "$DATA_DISK"
sudo blkid "$DATA_DISK" || true
findmnt --source "$DATA_DISK" || true

Stop if the device contains signatures, partitions, mounted filesystems, or an unexpected size. A production disk should be referenced by its verified /dev/disk/by-id/ symlink, not a potentially changing /dev/sdX name.

Create LUKS2, LVM, and ext4

The next block is destructive. cryptsetup luksFormat asks for explicit confirmation and a passphrase. The mapping remains open at /dev/mapper/cryptdata until closed or the machine shuts down.

DATA_DISK=/dev/vdb

sudo cryptsetup luksFormat --type luks2 "$DATA_DISK"
sudo cryptsetup open "$DATA_DISK" cryptdata

sudo pvcreate /dev/mapper/cryptdata
sudo vgcreate vg_secure /dev/mapper/cryptdata
sudo lvcreate --size 64G --name appdata vg_secure

sudo mkfs.ext4 -L appdata /dev/vg_secure/appdata
sudo mount /dev/vg_secure/appdata /srv/appdata
sudo install -d -m 0750 /srv/appdata/health
printf '%s\n' 'storage-restore-sentinel-v1' |
  sudo tee /srv/appdata/health/restore-sentinel.txt >/dev/null

sudo pvs
sudo vgs
sudo lvs -o lv_name,vg_name,lv_size,origin,data_percent
findmnt /srv/appdata

Only 64 GiB is allocated initially. The remaining extents are operational headroom for snapshots, restore drills, and controlled growth. Allocating the entire volume group on day one removes much of LVM’s value.

Configure boot activation

Use UUIDs so device-name changes do not break boot. The following appends entries only if their UUIDs are absent. Review both files before rebooting.

DATA_DISK=/dev/vdb
CRYPT_UUID=$(sudo cryptsetup luksUUID "$DATA_DISK")
FS_UUID=$(sudo blkid -s UUID -o value /dev/vg_secure/appdata)

sudo grep -q "UUID=$CRYPT_UUID" /etc/crypttab ||
  printf 'cryptdata UUID=%s none luks\n' "$CRYPT_UUID" |
  sudo tee -a /etc/crypttab

sudo grep -q "UUID=$FS_UUID" /etc/fstab ||
  printf 'UUID=%s /srv/appdata ext4 defaults,nodev,nosuid 0 2\n' "$FS_UUID" |
  sudo tee -a /etc/fstab

sudo cryptsetup luksDump "$DATA_DISK"
sudo findmnt --verify
sudo systemctl daemon-reload

The nodev and nosuid options reduce risk for application data. Add noexec only if the workload never executes binaries or scripts from this volume. The configuration uses an interactive boot passphrase; unattended servers should use an organization-approved TPM, network-bound, or hardware-backed unlock design with a documented recovery key.

Create consistent snapshot backups

The backup target must already be mounted. The script refuses to write into the root filesystem if that mount disappears, takes an exclusive lock, creates one snapshot, mounts it without replaying the ext4 journal, writes a compressed archive atomically, and always cleans up.

#!/usr/bin/env bash
set -Eeuo pipefail

BACKUP_ROOT=/mnt/backup/appdata
SNAP_MOUNT=/run/appdata-snapshot
SNAP_LV=/dev/vg_secure/appdata_snap
SOURCE_LV=/dev/vg_secure/appdata
STAMP=$(date -u +%Y%m%dT%H%M%SZ)
FINAL="$BACKUP_ROOT/appdata-$STAMP.tar.gz"
PARTIAL="$FINAL.partial"

exec 9>/run/lock/appdata-backup.lock
flock -n 9 || { echo "backup already running" >&2; exit 1; }

mountpoint -q /mnt/backup ||
  { echo "/mnt/backup is not mounted" >&2; exit 1; }
test "$(findmnt -n -o SOURCE /mnt/backup)" != "/dev/vg_secure/appdata" ||
  { echo "backup target is the source volume" >&2; exit 1; }

install -d -m 0750 "$BACKUP_ROOT" "$SNAP_MOUNT"

cleanup() {
  mountpoint -q "$SNAP_MOUNT" && umount "$SNAP_MOUNT" || true
  lvs vg_secure/appdata_snap >/dev/null 2>&1 &&
    lvremove --yes "$SNAP_LV" || true
  rm -f "$PARTIAL"
}
trap cleanup EXIT

lvcreate --snapshot --size 16G --name appdata_snap "$SOURCE_LV"
mount -o ro,noload "$SNAP_LV" "$SNAP_MOUNT"

tar --acls --xattrs --numeric-owner --one-file-system \
  -C "$SNAP_MOUNT" -cf - . | gzip -1 >"$PARTIAL"
gzip -t "$PARTIAL"
mv "$PARTIAL" "$FINAL"
sha256sum "$FINAL" >"$FINAL.sha256"

echo "created $FINAL"

Install that content as /usr/local/lib/storage/backup-appdata, owned by root with mode 0750. Run it from a systemd timer or an existing scheduler as root. Do not place a LUKS passphrase in the script.

A 16 GiB snapshot can preserve up to roughly that amount of changed block data while the archive runs; archive size is irrelevant. Watch Data% during early production runs. If write churn approaches the snapshot capacity, increase it, shorten the job, use application-native incremental backups, or move backups to a replica.

Monitor media, capacity, and snapshot pressure

SMART observes the physical device below encryption. First verify that the controller exposes meaningful data:

sudo smartctl --health /dev/vdb
sudo smartctl --all /dev/vdb
sudo systemctl enable --now smartd
sudo journalctl -u smartd --since today

sudo vgs -o vg_name,vg_size,vg_free
sudo lvs -a -o lv_name,lv_size,origin,data_percent,metadata_percent
df -h /srv/appdata /mnt/backup
sudo cryptsetup status cryptdata

Virtio disks may not expose host SMART telemetry to a guest; monitor the physical device on the hypervisor instead. Hardware RAID controllers may require controller-specific device selection. A successful command is not evidence that the displayed counters describe the underlying media.

Configure smartd using the stable physical-device path on a real host. Keep its default logging to the system journal, and connect journal alerts to the monitoring system already responsible for paging. Monitor at least filesystem utilization, volume-group free extents, backup age, archive verification failures, kernel I/O errors, SMART health changes, and snapshot Data%.

Capacity alerts need time to act. Treat exhaustion of the volume group and backup destination as separate risks. LVM cannot create a restore-test volume merely because the filesystem reports free space; it needs unallocated extents in the volume group.

Perform a real restore drill

An archive is only a claim until it has been extracted and inspected. Run this after the backup script has removed its snapshot. The 48 GiB test volume fits beside the 64 GiB live volume in the example 128 GiB disk, allowing for encryption and LVM metadata.

BACKUP_ROOT=/mnt/backup/appdata
ARCHIVE=$(find "$BACKUP_ROOT" -maxdepth 1 -type f \
  -name 'appdata-*.tar.gz' -printf '%T@ %p\n' |
  sort -n | tail -1 | cut -d' ' -f2-)

test -n "$ARCHIVE"
cd "$BACKUP_ROOT"
sha256sum --check "$(basename "$ARCHIVE").sha256"
gzip -t "$ARCHIVE"

sudo lvcreate --size 48G --name restore_test vg_secure
sudo mkfs.ext4 -L restore_test /dev/vg_secure/restore_test
sudo mount -o nodev,nosuid /dev/vg_secure/restore_test /mnt/restore-test

sudo tar --acls --xattrs --numeric-owner \
  -xzf "$ARCHIVE" -C /mnt/restore-test

sudo test "$(sudo cat /mnt/restore-test/health/restore-sentinel.txt)" = \
  "storage-restore-sentinel-v1"
sudo find /mnt/restore-test -xdev -type f | wc -l
sudo du -sh /mnt/restore-test
findmnt /mnt/restore-test

Replace the sentinel-only check with workload-aware validation: open restored repositories, validate media, run a database engine against its supported logical backup, or start an isolated application instance with outbound network access blocked. Never point a restore test at production credentials, queues, webhooks, or email systems.

After recording the result, remove the disposable volume safely:

sudo umount /mnt/restore-test
sudo e2fsck -f /dev/vg_secure/restore_test
sudo lvremove --yes /dev/vg_secure/restore_test
sudo lvs
sudo vgs

e2fsck runs only while the filesystem is unmounted. Its success confirms filesystem structure, while application checks confirm that the restored contents are useful. Both matter.

Security, performance, and deployment discipline

LUKS protects data at rest, not data on an unlocked machine. Restrict root access, protect recovery keys separately, patch the kernel and storage tools, and keep backup encryption independent from the source disk. If both source and backup unlock automatically with the same host, theft or compromise can defeat both layers.

Do not expose storage-management services through the firewall. The design opens no network port. If /mnt/backup uses NFS, SSH, or object-storage synchronization, permit only the required destination, authenticate with a narrowly scoped identity, and prevent the application account from deleting historical backups.

Compression level one intentionally favors a shorter snapshot lifetime over maximum compression. Measure archive duration, write churn, CPU load, and destination throughput before changing it. SSD discard can improve long-term behavior but leaks allocation patterns through encryption; enable it only after evaluating the device, threat model, and sanitization expectations.

Deploy in stages: rehearse destruction and recovery in a VM, build the production volume using a verified stable device path, run a manual backup, inspect snapshot consumption, restore into an isolated LV, reboot during a maintenance window, and verify automatic unlock and mounting. Schedule recurring restore drills, not merely recurring backups.

Common failures worth planning for

• The snapshot reaches 100%: it becomes unusable. Remove it, preserve the original volume, increase headroom, and rerun the backup.
• The backup mount disappears: without the mountpoint check, archives could fill the root filesystem. Keep the refusal behavior.
• Boot enters emergency mode: inspect /etc/crypttab, UUIDs, key availability, and /etc/fstab ordering from recovery media.
• SMART reports unsupported: query the physical host or configure the correct controller-specific device type rather than treating missing telemetry as good health.
• A database restores inconsistently: use its documented logical or physical backup mechanism; a crash-consistent filesystem snapshot cannot manufacture transactional guarantees the application does not provide.
• The restore LV cannot be created: inspect vgs. Free space inside appdata is not free space in vg_secure.

Final verification checklist

• The production device is identified by verified model, serial number, size, and stable path.
• LUKS recovery material is stored separately and has been tested.
• /srv/appdata mounts after a controlled reboot with the intended hardening options.
• The volume group retains enough free extents for snapshots and restore drills.
• The backup target is independent, encrypted, capacity-monitored, and protected from application deletion.
• The latest archive passes its checksum and compression tests.
• A restored copy passes filesystem and workload-specific validation.
• SMART, kernel I/O errors, capacity, snapshot pressure, and backup age reach the operational alerting system.
• Restore drills have an owner, schedule, recorded duration, and documented recovery procedure.

The memorable unit of storage reliability is not the disk, volume, snapshot, or archive. It is the verified recovery path. Encryption limits exposure, LVM creates room to maneuver, SMART provides imperfect early signals, and backups preserve history. Only a tested restore turns those useful mechanisms into confidence.