Debian Sid on encrypted ZFS

This guide is for an advanced Debian GNU/Linux installation using the ZFS storage system with an encrypted root volume for security and privacy. It will also be upgraded from the current Stable release (Bullseye) to the rolling-release Unstable version (Sid).

ZFS has long been considered the last word on advanced storage developments. With its advanced safety, efficiency, and performance mechanisms it’s easy to see why it’s popular in the storage world, DIY and enterprise alike.

However, there’s a great deal of fear and uncertainty surrounding its use on PCs. Some believe that without ECC memory you’re more likely to experience catastrophic data loss with ZFS than with a typical filesystem such as EXT4 or XFS. Others will insist that without error correcting memory it is dangerous to perform a scrub or integrity check, naming the phenomenon the scrub of death. This has largely been disproven in the years since, most notibly by one of the co-founders of ZFS.

Anecdotally, I will say that having used ZFS on non-ECC systems for the better part of a decade, it’s absolutely safe and reliable. If you have the choice between a filesystem that has the capacity for integrity checks and one that doesn’t, take the one that will prevent data rot.

While some operating systems and distributions have toyed with integrating ZFS, it still remains slightly difficult to implement on your own. This post aims to present a recipe for installing Debian Sid on a ZFS filesystem. Is it much more hands-on than a typical install, but the end result will have numerous advantages over a similar system without copy-on-write capabilities. For example, ZFS allows point-in-time data snapshots that allow the administrator to “roll-back” the system to a known state, such as undoing a problematic system update. It also allows for very fast and efficient incremental system-wide backups which significantly lower overhead than Rsync or other tools.

Once complete, this system will have a fully functional Debian Sid (unstable) install on an encrypted ZFS storage system. Note that the /boot partition will be on an unencrypted ext2 partition to hold the kernel and initramfs for more reliable bootloader operation, but at the cost of leaving a small number of system files unencrypted.

Prepare bootable media

The first step will be to prepare Debian install media. For this procedure we must use the live installer rather than the usual netinstall ISO, since it requires running a shell and installing tools on the installer itself.

The ISO image can be downloaded from the official Debian site here:

https://cdimage.debian.org/cdimage/release/current-live/amd64/iso-hybrid/

The non-free firmware installer can be used instead if the device in use has driver issues with the regular image. This is recommended if you have an Intel or Realtek wireless card onboard.

https://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/current-live/amd64/iso-hybrid/

Configure the live environment

Once the device has booted into the live installer system, a few modifications can be made.

SSH server (optional)

If needed, the ssh server can be installed and started. This is useful for remote access.

Install and run ssh server

sudo apt install -y openssh-server
sudo systemctl start ssh
mkdir ~/.ssh 
curl https://github.com/myusername.keys > .ssh/authorized_keys

ZFS setup tools

By default, Debian will only include free software in the repositories. Typically this only includes software with GPL, MIT, or Apache based licenses. Since ZFS is released under the CDDL which is not compatible with Debian’s licenses, the system must be reconfigured to allow installing from the contrib repositories.

On the installer, enable non-free contrib sources by editing /etc/apt/sources.list

deb http://deb.debian.org/debian/ bullseye main contrib

Update the system’s sources.

apt update

Install the ZFS utilities package:

apt install -y zfsutils-linux

This will take a moment, since it must build DKMS binaries for the system after installation.

Prepare the disks

Install the tools needed to bootstrap the system:

apt install -y debootstrap gdisk 

To ensure the correct path is always used, we will use the disk’s ID rather than the usual device path (such as sda or nvme0n1).

Find the disk’s ID

$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Jul 22 17:58 ata-Samsung_SSD_860_EVO_500GB_A1B2C3D4E5 -> ../../sda

Set the $DISK variable to the ID of your drive:

DISK="/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_A1B2C3D4E5"

Partition the disk

Warning: this will erase any data on the disk. Please be careful!

Clear the partition table:

sgdisk --zap-all $DISK

Then, create the partitions according to how your system boots up.

In both cases, the disk will use GPT.

  • For Legacy boot devices, a small 8MB unformatted partition will sit at the beginning of the disk to prevent corrupting the MBR where GRUB2 will be installed.
  • For EFI boot systems, a 512MB partition will sit at the beginning to hold the ESP partition.

For both styles, a 1GB /boot partition will sit next, then the remainder will be dedicated to the ZFS volume.

Legacy Boot

parted -a optimal $DISK

mklabel GPT

mkpart primary 2048s 8M
set 1 bios_grub on

mkpart primary 9 1023
name 2 boot

mkpart primary 1024 100%
name 3 zfs

EFI boot

parted -a optimal $DISK

mklabel GPT

mkpart primary 1 512
name 1 efi
set 1 boot on 

mkpart primary 512 1536
name 2 boot

mkpart primary 1536 100%
name 3 zfs

Once the partitions are created, the filesystems can be configured.

ZFS root filesystem

Create the zpool:

zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -O encryption=on -O keylocation=prompt -O keyformat=passphrase \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/ -R /mnt \
    zroot ${DISK}-part3

Create the ZFS datasets for the system partitions:

zfs create -o canmount=off -o mountpoint=none zroot/ROOT
zfs create -o canmount=noauto -o mountpoint=/ zroot/ROOT/debian
zfs mount zroot/ROOT/debian

zfs create                     zroot/home
zfs create -o mountpoint=/root zroot/home/root
chmod 700 /mnt/root
zfs create -o canmount=off     zroot/var
zfs create -o canmount=off     zroot/var/lib
zfs create                     zroot/var/log
zfs create                     zroot/var/spool
zfs create -o canmount=off zroot/usr
zfs create                 zroot/usr/local

Boot filesystem

The boot partition should be formatted as ext2 or ext4.

mkfs.ext2 $DISK-part2

Then, it can be mounted at /boot.

mkdir /mnt/boot
mount $DISK-part2 /mnt/boot

Bootstrap the Debian system

mkdir /mnt/run
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock

Create the minimal system install. This will take 2-5 minutes.

debootstrap bullseye /mnt

Copy in ZFS zpool.cache into the chroot:

mkdir /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/

Set the hostname:

hostname gablogian
hostname > /mnt/etc/hostname
echo "127.0.1.1    gablogian" > /mnt/etc/hosts

Configure software sources

Configure the file /mnt/etc/apt/sources.list

deb http://deb.debian.org/debian bullseye main contrib
deb-src http://deb.debian.org/debian bullseye main contrib

deb http://deb.debian.org/debian-security bullseye-security main contrib
deb-src http://deb.debian.org/debian-security bullseye-security main contrib

deb http://deb.debian.org/debian bullseye-updates main contrib
deb-src http://deb.debian.org/debian bullseye-updates main contrib

Note the inclusion of contrib on each line, as mentioned earlier.

Enter the chroot

Mount the ephemeral system devices to pass through the host hardware to the chroot:

mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys

Then, create a shell into the new install.

chroot /mnt /usr/bin/env DISK=$DISK bash --login

Once inside the chroot, copy the mounted device information from the host into the system.

ln -s /proc/self/mounts /etc/mtab

Configure locales

apt update
apt install --yes console-setup locales

Set up the locale, timezone, keyboard, and console config.

Note: en_US.UTF-8 should be included regardless of which locale and language you intend to use.

dpkg-reconfigure locales tzdata keyboard-configuration console-setup

Configure the /boot mount

Only the /boot partition must be set up in the /etc/fstab file, since the ZFS kernel driver will configure mont points for all of the zfs datasets.

echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \
/boot ext2 defaults 0 0 >> /etc/fstab

Install ZFS modules

Install ZFS in the chroot:

apt install --yes dpkg-dev linux-headers-generic linux-image-generic

apt install --yes zfs-initramfs

echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf

Install GRUB

GRUB will be used as the system’s bootloader.

apt install --yes grub-pc

Set the root password

passwd

The root account can be disabled afterwords for security purposes.

tmpfs

/tmp should be a very small ramdisk.

cp /usr/share/systemd/tmp.mount /etc/systemd/system/
systemctl enable tmp.mount

Optionally, reconfigure the mountpoint to include noexec to reduce the attack surface.

Install other utilities

apt install openssh-server
systemctl enable ssh.service

Configure GRUB

Though the grub package has been installed, the bootloader itself must be installed to the MBR or EFI of the selected drive.

grub-probe /boot

update-initramfs -c -k all

Configure /etc/default/grub to point to the correct ZFS dataset for the root volume:

GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/debian"

Update the grub config and install the bootloader to EFI/MBR:

update-grub
grub-install $DISK

Configure filesystem mount ordering:

mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/zroot
zed -F &

cat /etc/zfs/zfs-list.cache/zroot
fg
^C

Fix the paths to remove /mnt/

sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*

Take a snapshot

This will take a full system snaspshot of all datasets:

zfs snapshot -r zroot/ROOT/debian@install

Create a user account

username=ongo

zfs create zroot/home/$username
adduser $username

cp -a /etc/skel/. /home/$username
chown -R $username:$username /home/$username
usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username

Install software

First, update the system to the latest stable version. Then, the tasksel tool can be used to select a desktop environment to use if desired.

apt dist-upgrade --yes
tasksel --new-install

Reboot

Exit from the chroot

exit

Unmount all ZFS mounts:

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a

Reboot the computer into the installed environment and remove the installer USB.

If mounting fails during bootup, retry importing the pool in the initramfs shell:

zpool import -f zroot
exit

Configure the system

After booted up into the newly installed system, some additional setup tasks will be required.

Non-Free firmware & Drivers

It’s likely that the hardware will need non-free drivers to work correctly. Commonly, realtek and Intel wireless cards will need non-free firmware to function correctly under Linux.

Add non-free to the end of every line in /etc/apt/sources.list, then install the packages.

sudo apt update
sudo apt install firmware-iwlwifi firmware-linux

Install Plymouth (optional)

Plymouth is likely already installed, but needs to be configured.

sudo apt install plymouth plymouth-themes

Optionally, set a theme if you don’t like the default Debian one:

sudo plymouth-set-default-theme -R spinner

Modify /etc/default/grub to set the screen resolution:

GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/debian"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_GFXMODE=1280x800

Finally, update the GRUB config:

sudo update-grub2

Upgrade to Sid / Unstable

The system was installed using Debian stable. For many people this is the way to go, but some prefer to have a cutting-edge system with much newer versions of all packages.

Before starting the upgrade, take a snapshot of the system:

sudo zfs snapshot -R zroot@upgrade

Install some tools to ease the updating process:

sudo apt install apt-listbugs apt-listchanges

Update the sources.list file:

deb https://deb.debian.org/debian/ unstable main contrib non-free
deb-src https://deb.debian.org/debian/ unstable main contrib non-free

Update sources:

sudo apt update

Perform the full upgrade:

sudo apt dist-upgrade

This may take a few attempts. After all the packages are installed, reboot and load the new kernel and libraries.

Important: Do not reboot until the upgrade is finished. You may end up with an unbootable system if the kernel does not correctly install the ZFS modules into the initramfs.

If there are issues, try running these commands to fix any package dependance problems:

apt install -f
dpkg --configure --pending

Cleanup the system

Remove a few default programs, if you prefer.

sudo apt remove gnome-games gnome-software gnome-software-common

Snapshots

A simple script to take a snapshot of the root volume with the current date & time:

#!/bin/bash
# Take a snapshot marked current time
sudo zfs snapshot -r zroot@"`date '+%Y%m%d-%H%M'`"

Install non-LTS packages

Debian Stable comes with the ESR (Extended Support Release) version of Firefox, but it can be replaced by the mainline version.

sudo apt remove firefox-esr
sudo apt install firefox

Rescue mode

The process for emergency-booting this install is slightly different than a normal debian install. Rather than simply booting up and chrooting into the root volume, we must first install the ZFS utilities, import and decrypt the zpool, then mount the datasets correctly.

First make sure the boot OS has zfs tools available:

sudo apt install -y zfsutils-linux

Mount the root volume for chroot

zpool import -f -R /mnt -d /dev/sda3 zroot -l

The root dataset may need to be unmounted, since the actual / path is mounted from zroot/ROOT/debian

zfs set canmount=off zroot

Make sure the /boot volume is also mounted:

mount /dev/sda2 /mnt/boot

Enter the chroot:

mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login

References: