Disk Encryption, RAID, and Backups


One purpose of this page is to write down what I did so that I can remember it later.
Another purpose of this page is to share what I did with other people who may find it useful.
 

Fundamentals and Theory

Feel free to skip to the How-To section if you are looking for implementation details, including relevant shell commands with example arguments and flags

Backups are as easy as 3-2-1

While the concept of an on-site backup and an off-site backup likely predate it, the "3-2-1" backup strategy is a catchy name for the concept that originated in the DAM Book on Digital Asset Management (DAM). The rule expands like so: at least three copies of the data (a primary copy plus two backups), at least two independent storage media, and at least one copy off-site.

There are all kinds of disasters that could wipe out the primary copy of your data plus any and all on-site copies. A backup kept at your workplace, or kept by a trusted friend or family member will ensure that any such disaster won't take your data along with it.

Independent storage media are important because the lifespan of the same models or even kinds of storage media tend to be correlated. If you have two drives of the same make and model, when one drive fails, the other is sure to follow.

Even if they come from different manufacturers, if you have two hard drives, their expected lifespan is likely to be similar. The same goes for other storage media like solid state drives, SD cards, DVDs, and so on. Solid state drives, SD cards, and USB drives all tend to be backed by the same flash memory technology.

Since the primary wear factor for flash memory is data written, these different media may be rated for different lifespans (as measured by bytes written) even after adjusting for the difference in typical storage capacities. Unlike flash memory, most other storage media, including hard disks and DVDs, involve moving parts, which has its own drawbacks and factors for wear.

The redundancy of three copies of your data ensures that you are protected in case disaster strikes twice. If your backup system ever leaves you with only one (primary) copy of your data, then it has failed because it is no longer providing you with any backups of your data.

RAID is not a backup

A redundant array of independent disks presents itself to a computer as a single logical disk.

How-To and Application

Base Debian Install with Disk Encryption

My servers run Debian. The Debian installer has a few different options for guided partitioning. The simplest one is a root partition and a swap partition. In addition to "classic" partitioning, the installer supports Logical Volume Management (LVM) and "encrypted LVM". When using encrypted LVM, a separate boot partition is needed as the debian installer currently lacks native support for an encrypted /boot partition.

Most modern computers rely on EFI (Extensible Firmware Interface) rather than a BIOS. In that case, the debian installer will also create a separate partition for /boot/efi. So a freshly installed system has a EFI partition, a boot partition, and a third encrypted partition. The encrypted partition is configured as a physical volume in LVM, with two logical volumes: the root partition and a swap partition.

A few different configuration files are important for properly mounting the encrypted logical volumes at boot time: /etc/fstab, /etc/crypttab, and the contents of /etc/lvm/.

Encrypting and Adding a Data Disk

With the hierarchy of encrypted physical volume -> LVM2 volume group -> LVM2 logical volumes, the obvious first step for adding a data disk is to add encryption to it:

cryptsetup luksFormat /dev/sdb

Substitute /dev/sdb for /dev/sda or /dev/sdc or even /dev/nvme1n1 as applicable. Once the drive is encrypted, you need to "open" it (i.e. load the encryption key into memory) for reading and writing:

cryptsetup luksOpen /dev/sdb sdb_crypt

Once that is loaded, /dev/mapper/sdb_crypt can act as a physical volume. LVM needs to learn about it so that it can create logical volumes on top of it:

pvcreate /dev/mapper/sdb_crypt

Then, it is necessary to add the physical volume (PV) to a volume group (VG) so that LVM can use the physical volume as backing storage for the volume group's logical volumes:

vgextend myserver-vg /dev/mapper/sdb_crypt

Obviously, myserver-vg is simply an example name for the default volume group created by the debian installer on a computer whose hostname is myserver. When actually running this command, "myserver-vg" should be replaced with the actual name of the existing volume group.

RAID1 for root partition

See above for explanation of RAID. For the purposes of this section, you may read "RAID1" as "mirror one disk to another". In this case, the goal is to mirror the encrypted portion of the OS disk to the encrypted data disk. This relies on the data disk being at least as big as the OS disk itself, down to the byte.

The default "segment type" in LVM is "linear", which basically means all data in the volume in a single bucket without any striping nor mirroring of blocks. There is a deprecated "mirror" segment type which is deprecated in favour of "raid1".

The existing state of the OS disk here is that it is the root logical volume in the myserver-vg volume group, and that it is a linear volume backed by /dev/sda3_crypt

The following incantation will convert that logical volume (root in myserver-vg) into a raid1 volume across /dev/sda3_crypt and /dev/sdb_crypt:

lvconvert --type raid1 /dev/mapper/myserver--vg-root -m1 /dev/sda3_crypt /dev/sdb_crypt

This command should return back to a shell right away, but it will kick off the process to mirror all the data from the existing physical volume to the new physical volume in the background. This process may take a while. The server should keep serving things to its clients while the data operation runs in the background, but this is a good point to step away from the terminal, take a break, and come back to this later.

It is probably a bad idea to reboot the server until the mirroring process completes and the encrypted data disk is added to /etc/crypttab.

Automatically Mount Encrypted Data Disk

The debian installer should generate an existing entry in /etc/crypttab for the OS disk. This entry should serve as a good base for adding additional entries for additional data disks. It is possible to find UUIDs for the encrypted block storage devices through the blkid (block id) tool.

Recover From OS Disk Failure

⚠️ This section is untested ⚠️

Ideally, at least one data disk has a backup of the /boot partition and a backup of the /boot/efi partition. First things first, slap in a new OS disk and boot into a debian installer/live/rescue environment.

In the live environment, start by partitioning the new OS disk: first, 500MB vFAT for /boot/efi, second, 475MB ext2 for /boot, and the rest of the disk for an encrypted partition (similar to the setup for the encrypted data disk). Mount the first two partitions and setup LUKS on the third partition.

luksOpen the data disk for recovery, and the same on the third partition to setup LVM inside. Special care may need to be taken to match up with old/existing mappings. Setup a new swap logical volume on the new OS disk. Repair the root RAID1 with the remaining space on the physical volume.

Wait on RAID1 mirroring to complete.

Mount the new boot and new EFI partitions to recover from the backups on the data disk. Update the UUIDs and GUIDs in /etc/crypttab and in /etc/fstab.

Hopefully all of that works and is enough to get up and running again.

Recover From Single Data Disk Failure

⚠️ This section is untested ⚠️

Hopefully it still boots without the data disk. Slap in a new data disk and follow the setup steps again. The arguments to lvconvert will likely involve the --repair flag.