Debian Sid on encrypted ZFS
This guide is for an advanced Debian GNU/Linux installation using the ZFS storage system with an encrypted root volume for security and privacy. It will also be upgraded from the current Stable release (Bullseye) to the rolling-release Unstable version (Sid).
ZFS has long been considered the last word on advanced storage developments. With its advanced safety, efficiency, and performance mechanisms it’s easy to see why it’s popular in the storage world, DIY and enterprise alike.
However, there’s a great deal of fear and uncertainty surrounding its use on PCs. Some believe that without ECC memory you’re more likely to experience catastrophic data loss with ZFS than with a typical filesystem such as EXT4 or XFS. Others will insist that without error correcting memory it is dangerous to perform a scrub or integrity check, naming the phenomenon the scrub of death. This has largely been disproven in the years since, most notibly by one of the co-founders of ZFS.
Anecdotally, I will say that having used ZFS on non-ECC systems for the better part of a decade, it’s absolutely safe and reliable. If you have the choice between a filesystem that has the capacity for integrity checks and one that doesn’t, take the one that will prevent data rot.
While some operating systems and distributions have toyed with integrating ZFS, it still remains slightly difficult to implement on your own. This post aims to present a recipe for installing Debian Sid on a ZFS filesystem. Is it much more hands-on than a typical install, but the end result will have numerous advantages over a similar system without copy-on-write capabilities. For example, ZFS allows point-in-time data snapshots that allow the administrator to “roll-back” the system to a known state, such as undoing a problematic system update. It also allows for very fast and efficient incremental system-wide backups which significantly lower overhead than Rsync or other tools.
Once complete, this system will have a fully functional Debian Sid (unstable) install on an encrypted ZFS storage system.
Note that the /boot partition will be on an unencrypted ext2 partition to hold the kernel and initramfs for more reliable bootloader operation, but at the cost of leaving a small number of system files unencrypted.
Prepare bootable media
The first step will be to prepare Debian install media. For this procedure we must use the live installer rather than the usual netinstall ISO, since it requires running a shell and installing tools on the installer itself.
The ISO image can be downloaded from the official Debian site here:
https://cdimage.debian.org/cdimage/release/current-live/amd64/iso-hybrid/
The non-free firmware installer can be used instead if the device in use has driver issues with the regular image. This is recommended if you have an Intel or Realtek wireless card onboard.
Configure the live environment
Once the device has booted into the live installer system, a few modifications can be made.
SSH server (optional)
If needed, the ssh server can be installed and started. This is useful for remote access.
Install and run ssh server
sudo apt install -y openssh-server
sudo systemctl start ssh
mkdir ~/.ssh
curl https://github.com/myusername.keys > .ssh/authorized_keys
ZFS setup tools
By default, Debian will only include free software in the repositories. Typically this only includes software with GPL, MIT, or Apache based licenses. Since ZFS is released under the CDDL which is not compatible with Debian’s licenses, the system must be reconfigured to allow installing from the contrib repositories.
On the installer, enable non-free contrib sources by editing /etc/apt/sources.list
deb http://deb.debian.org/debian/ bullseye main contrib
Update the system’s sources.
apt update
Install the ZFS utilities package:
apt install -y zfsutils-linux
This will take a moment, since it must build DKMS binaries for the system after installation.
Prepare the disks
Install the tools needed to bootstrap the system:
apt install -y debootstrap gdisk
To ensure the correct path is always used, we will use the disk’s ID rather than the usual device path (such as sda or nvme0n1).
Find the disk’s ID
$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root 9 Jul 22 17:58 ata-Samsung_SSD_860_EVO_500GB_A1B2C3D4E5 -> ../../sda
Set the $DISK variable to the ID of your drive:
DISK="/dev/disk/by-id/ata-Samsung_SSD_860_EVO_500GB_A1B2C3D4E5"
Partition the disk
Warning: this will erase any data on the disk. Please be careful!
Clear the partition table:
sgdisk --zap-all $DISK
Then, create the partitions according to how your system boots up.
In both cases, the disk will use GPT.
- For Legacy boot devices, a small 8MB unformatted partition will sit at the beginning of the disk to prevent corrupting the MBR where GRUB2 will be installed.
- For EFI boot systems, a 512MB partition will sit at the beginning to hold the ESP partition.
For both styles, a 1GB /boot partition will sit next, then the remainder will be dedicated to the ZFS volume.
Legacy Boot
parted -a optimal $DISK
mklabel GPT
mkpart primary 2048s 8M
set 1 bios_grub on
mkpart primary 9 1023
name 2 boot
mkpart primary 1024 100%
name 3 zfs
EFI boot
parted -a optimal $DISK
mklabel GPT
mkpart primary 1 512
name 1 efi
set 1 boot on
mkpart primary 512 1536
name 2 boot
mkpart primary 1536 100%
name 3 zfs
Once the partitions are created, the filesystems can be configured.
ZFS root filesystem
Create the zpool:
zpool create \
-o ashift=12 \
-o autotrim=on \
-O encryption=on -O keylocation=prompt -O keyformat=passphrase \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto \
-O compression=lz4 \
-O normalization=formD \
-O relatime=on \
-O canmount=off -O mountpoint=/ -R /mnt \
zroot ${DISK}-part3
Create the ZFS datasets for the system partitions:
zfs create -o canmount=off -o mountpoint=none zroot/ROOT
zfs create -o canmount=noauto -o mountpoint=/ zroot/ROOT/debian
zfs mount zroot/ROOT/debian
zfs create zroot/home
zfs create -o mountpoint=/root zroot/home/root
chmod 700 /mnt/root
zfs create -o canmount=off zroot/var
zfs create -o canmount=off zroot/var/lib
zfs create zroot/var/log
zfs create zroot/var/spool
zfs create -o canmount=off zroot/usr
zfs create zroot/usr/local
Boot filesystem
The boot partition should be formatted as ext2 or ext4.
mkfs.ext2 $DISK-part2
Then, it can be mounted at /boot.
mkdir /mnt/boot
mount $DISK-part2 /mnt/boot
Bootstrap the Debian system
mkdir /mnt/run
mount -t tmpfs tmpfs /mnt/run
mkdir /mnt/run/lock
Create the minimal system install. This will take 2-5 minutes.
debootstrap bullseye /mnt
Copy in ZFS zpool.cache into the chroot:
mkdir /mnt/etc/zfs
cp /etc/zfs/zpool.cache /mnt/etc/zfs/
Set the hostname:
hostname gablogian
hostname > /mnt/etc/hostname
echo "127.0.1.1 gablogian" > /mnt/etc/hosts
Configure software sources
Configure the file /mnt/etc/apt/sources.list
deb http://deb.debian.org/debian bullseye main contrib
deb-src http://deb.debian.org/debian bullseye main contrib
deb http://deb.debian.org/debian-security bullseye-security main contrib
deb-src http://deb.debian.org/debian-security bullseye-security main contrib
deb http://deb.debian.org/debian bullseye-updates main contrib
deb-src http://deb.debian.org/debian bullseye-updates main contrib
Note the inclusion of contrib on each line, as mentioned earlier.
Enter the chroot
Mount the ephemeral system devices to pass through the host hardware to the chroot:
mount --make-private --rbind /dev /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys /mnt/sys
Then, create a shell into the new install.
chroot /mnt /usr/bin/env DISK=$DISK bash --login
Once inside the chroot, copy the mounted device information from the host into the system.
ln -s /proc/self/mounts /etc/mtab
Configure locales
apt update
apt install --yes console-setup locales
Set up the locale, timezone, keyboard, and console config.
Note: en_US.UTF-8 should be included regardless of which locale and language you intend to use.
dpkg-reconfigure locales tzdata keyboard-configuration console-setup
Configure the /boot mount
Only the /boot partition must be set up in the /etc/fstab file, since the ZFS kernel driver will configure mont points for all of the zfs datasets.
echo /dev/disk/by-uuid/$(blkid -s UUID -o value ${DISK}-part2) \
/boot ext2 defaults 0 0 >> /etc/fstab
Install ZFS modules
Install ZFS in the chroot:
apt install --yes dpkg-dev linux-headers-generic linux-image-generic
apt install --yes zfs-initramfs
echo REMAKE_INITRD=yes > /etc/dkms/zfs.conf
Install GRUB
GRUB will be used as the system’s bootloader.
apt install --yes grub-pc
Set the root password
passwd
The root account can be disabled afterwords for security purposes.
tmpfs
/tmp should be a very small ramdisk.
cp /usr/share/systemd/tmp.mount /etc/systemd/system/
systemctl enable tmp.mount
Optionally, reconfigure the mountpoint to include noexec to reduce the attack surface.
Install other utilities
apt install openssh-server
systemctl enable ssh.service
Configure GRUB
Though the grub package has been installed, the bootloader itself must be installed to the MBR or EFI of the selected drive.
grub-probe /boot
update-initramfs -c -k all
Configure /etc/default/grub to point to the correct ZFS dataset for the root volume:
GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/debian"
Update the grub config and install the bootloader to EFI/MBR:
update-grub
grub-install $DISK
Configure filesystem mount ordering:
mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/zroot
zed -F &
cat /etc/zfs/zfs-list.cache/zroot
fg
^C
Fix the paths to remove /mnt/
sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*
Take a snapshot
This will take a full system snaspshot of all datasets:
zfs snapshot -r zroot/ROOT/debian@install
Create a user account
username=ongo
zfs create zroot/home/$username
adduser $username
cp -a /etc/skel/. /home/$username
chown -R $username:$username /home/$username
usermod -a -G audio,cdrom,dip,floppy,netdev,plugdev,sudo,video $username
Install software
First, update the system to the latest stable version. Then, the tasksel tool can be used to select a desktop environment to use if desired.
apt dist-upgrade --yes
tasksel --new-install
Reboot
Exit from the chroot
exit
Unmount all ZFS mounts:
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
xargs -i{} umount -lf {}
zpool export -a
Reboot the computer into the installed environment and remove the installer USB.
If mounting fails during bootup, retry importing the pool in the initramfs shell:
zpool import -f zroot
exit
Configure the system
After booted up into the newly installed system, some additional setup tasks will be required.
Non-Free firmware & Drivers
It’s likely that the hardware will need non-free drivers to work correctly. Commonly, realtek and Intel wireless cards will need non-free firmware to function correctly under Linux.
Add non-free to the end of every line in /etc/apt/sources.list, then install the packages.
sudo apt update
sudo apt install firmware-iwlwifi firmware-linux
Install Plymouth (optional)
Plymouth is likely already installed, but needs to be configured.
sudo apt install plymouth plymouth-themes
Optionally, set a theme if you don’t like the default Debian one:
sudo plymouth-set-default-theme -R spinner
Modify /etc/default/grub to set the screen resolution:
GRUB_CMDLINE_LINUX="root=ZFS=zroot/ROOT/debian"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_GFXMODE=1280x800
Finally, update the GRUB config:
sudo update-grub2
Upgrade to Sid / Unstable
The system was installed using Debian stable. For many people this is the way to go, but some prefer to have a cutting-edge system with much newer versions of all packages.
Before starting the upgrade, take a snapshot of the system:
sudo zfs snapshot -R zroot@upgrade
Install some tools to ease the updating process:
sudo apt install apt-listbugs apt-listchanges
Update the sources.list file:
deb https://deb.debian.org/debian/ unstable main contrib non-free
deb-src https://deb.debian.org/debian/ unstable main contrib non-free
Update sources:
sudo apt update
Perform the full upgrade:
sudo apt dist-upgrade
This may take a few attempts. After all the packages are installed, reboot and load the new kernel and libraries.
Important: Do not reboot until the upgrade is finished. You may end up with an unbootable system if the kernel does not correctly install the ZFS modules into the initramfs.
If there are issues, try running these commands to fix any package dependance problems:
apt install -f
dpkg --configure --pending
Cleanup the system
Remove a few default programs, if you prefer.
sudo apt remove gnome-games gnome-software gnome-software-common
Snapshots
A simple script to take a snapshot of the root volume with the current date & time:
#!/bin/bash
# Take a snapshot marked current time
sudo zfs snapshot -r zroot@"`date '+%Y%m%d-%H%M'`"
Install non-LTS packages
Debian Stable comes with the ESR (Extended Support Release) version of Firefox, but it can be replaced by the mainline version.
sudo apt remove firefox-esr
sudo apt install firefox
Rescue mode
The process for emergency-booting this install is slightly different than a normal debian install. Rather than simply booting up and chrooting into the root volume, we must first install the ZFS utilities, import and decrypt the zpool, then mount the datasets correctly.
First make sure the boot OS has zfs tools available:
sudo apt install -y zfsutils-linux
Mount the root volume for chroot
zpool import -f -R /mnt -d /dev/sda3 zroot -l
The root dataset may need to be unmounted, since the actual / path is mounted from zroot/ROOT/debian
zfs set canmount=off zroot
Make sure the /boot volume is also mounted:
mount /dev/sda2 /mnt/boot
Enter the chroot:
mount --make-private --rbind /dev /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys /mnt/sys
chroot /mnt /usr/bin/env DISK=$DISK bash --login
References: