Just a few days ago I finally got a new server to replace a good old friend of mine which has been keeping my data safe since 2005. I was literally dying to get it up and running and move my data over when I realized it had been 8 years since I last setup dmcrypt on a server I only had ssh access to, and had no idea of what best current practices are.
So, let me start first by describing the environment. Like my previous server, this new machine is setup in a datacenter somewhere in Europe. I don't have any physical access to this machine, I can only ssh into it. I don't have a serial port I can connect to over the network, I don't have IPMI, nor something like intel kvm, but I really want to keep my data encrypted.
Having a laptop or desktop with your whole disk encrypted is pretty straightforward with modern linux systems. Your distro will boot up, kernel will be started, your scripts in the initrd will detect the encrypted partition, stop the boot process, ask you for a passphrase, decrypt your disk, and happily continue with the boot process.
But when your passphrase is asked, your network is not up yet, there is no ssh access. Either you sit in front of the monitor and type your passphrase, or there is really not that much you can do from a few thousand miles away.
To have encrypted partitions you can manage remotely, you pretty much need:
Sounds easy, doesn't it? I spent some time looking around to see if I could find some pre-baked solutions, like a simple package to install that would tweak my initrd and add ssh and the needed scripts, or some suggestion on how to do it in a smart way. In the end, I baked my own solution, exactly like 8 years ago.
So, here it is...
Creating an initrd or a tiny partition to do the initial boot did not seem very attractive: for one, I do not want to keep the whole root encrypted. Root contains only tools and scripts downloaded from the Debian repositories. Configs really contain no sensitive data, and the kind of logs I care about do not end up in /var/log. In second instance, my experience with initrd is that it changes quite a bit over time: you need to generate a new initrd for every new kernel and set of drivers, which is tricky by itself, the tools have changed significantly over time, you need to compute (and install) the dependencies for any tool you need from the minimal root, and hook your stuff well enough with the generator so it keeps working over time. And well, it's hard to test unless you reboot your system, which is really not the time I want to have surprises.
Creating a minimal root outside of the initrd did not seem very attractive either: not fancying having two root partitions to keep up to date in terms of kernel, grub, updates, and so on. And again, I did not need to encrypt root.
The solution I used is pretty simple: have your root and boot in clear, boot from there as normal. From rc*.d, disable all the services that require my encrypted data (like mysql, apache, or my repositories), remove my encrypted partitions from fstab and crypttab (or mark them noauto in both). Basically, have your normal system boot as usual, with ssh but no other service that needs your encrypted data.
Once the system boots, I have a script I can run manually that, in order:
Let's start from the encrypted partitions. I've been using LVM and dmcrypt pretty much since they existed. I don't have hybrid systems, always linux only, and I like LVM much more than managing partition tables manually.
One common solution to have multiple volumes encrypted is to create an encrypted volume with LVM, decrypt it, and then use it as a physical volume for another volume group. For example, a system volume group, system/encrypted as a logical volume, and rather than have a simple file system in there, use it as a physical volume for another volume group with multiple encrypted sub volumes.
I am not quite fond of this solution as it makes it hard to borrow space from encrypted space to clear text space and vice versa, and generally makes things more confusing.
What I tend to do is just have a single volume group, containing some encrypted and clear text logical volumes. This however means that each logical volume has to be decrypted independently, and most of the tools will ask you for a passphrase for each volume, multiple times, which is annoying. Some wikis suggest you to keep keyfiles on disk, which is roughly what I do: I create an encrypted logical volume, called keys, with a strong passphrase, that contains truly random keys. To mount the other volumes I first decrypt this key volume, and then immediately unmount it.
Using this mechanism, the "decrypt all the partitions" step I was talking about becomes:
As an additional requirement, I want those steps to be idempotent: if a partition is already mounted, it should be skipped. If I run the script multiple times, it should just complete the work that wasn't done before.
Back in 2005, together with a few friends with whom I was sharing the server, we wrote a small script to maintain those volumes and implement the steps above. The script is now checked in on github, you can find it here.
To get it running, you first need to install the tools and create the volume where to store the keys:
# Install sys-scripts. mkdir -p /opt/{scripts,conf} git clone https://github.com/ccontavalli/sys-scripts.git /opt/scripts # Install the tools that are needed. apt-get install cryptsetup lvm2 # Create a volume "encrypted-keys" in group "system", this would # be "vg0" unless you changed the default. lvcreate -L 20M -n encrypted-keys system # Encrypt the partition and open it. cryptsetup luksFormat /dev/system/encrypted-keys \ --cipher=aes-cbc-essiv:sha256 --key-size=256 --verify-passphrase cryptsetup luksOpen /dev/system/encrypted-keys cleartext-keys # Put a file system on that partition. mkfs.ext4 /dev/mapper/cleartext-keys
Note that if your volume is not called system but vg0, you will need to edit /opt/scripts/ac-dmcrypt-manage and change cfg_key_volume to look like:
cfg_key_volume=${cfg_key_volume-vg0/encrypted-keys}
or remember to always call ac-dmcrypt-manage with the volume passed, like:
cfg_key_volume=vg0/encrypted-keys ac-dmcrypt-manage ...
Now you are ready to create volumes. All you have to do is something like:
/opt/scripts/ac-dmcrypt-manage create-volume sytem \ mysql 20G /opt/mysql ext4
for example, and follow the prompts. You can create as many volumes as you like. If you want to mount that volume, you can then run:
/opt/scripts/ac-dmcrypt-manage start
If you want to inspect the generated keys:
/opt/scripts/ac-dmcrypt-manage mount-keys
Just remember to umount them after a whlie, by using umount-keys.
You can also change the mount options of your partition by editing /opt/conf/ac-fstab, which has been generated automatically by create-volume.
Let's say now you want to mark mysql as a process that cannot be started until the encrypted partitions are mounted. What you have to do is:
/opt/scripts/ac-system-boot add mysql
The script will disable mysql from the normal boot, by running something like update-rc.d mysql disable.
All you have to do is ssh on the system, and then run:
/opt/scripts/ac-system-boot start
This set of scripts has served me well for several years. I will probably stick with them until I find a better mechanism for this kind of setup. Systems like ecryptfs or encfs look like viable alternatives for home directories or private data for individual users. But from what I have read so far dmcrypt still looks like the best option to keep system partitions encrypted, on a server.
Before using ac-system-boot, we tried using runlevels. Isn't this what they were meant for? The idea was to have a minimal network runlevel, and another runlevel with the system daemons to boot once the partitions are available, and use telinit to switch between them. But between the various alternatives to SysV init that popped up in the last few years, the attention to speeding up the boot process, and various distribution scripts fiddling with rc*.d or assuming one setup or another, this did not work well.
Do you have better proposals? alternatives? let me know.