Roll back a broken Ubuntu system with ZFS snapshots

This post covers ZFS snapshots as a recovery tool on Ubuntu 22.10: creating, listing, comparing, deleting, rolling back and automating them. In January 2023 a displaylink driver install left my machine unable to boot, and a snapshot brought it back in under five minutes.

What a snapshot is

A ZFS snapshot is a read-only copy of a file system as it was at a given moment. It lives on the same pool as the original and stores only the differences from the current state, so it takes far less space than a traditional backup. Because it works at file system level, it captures system configuration, installed software and data together.

That also sets its limit: a snapshot on the same disk is not a backup. Keep backing up your important files.

ZFS is not part of the mainline Linux kernel because of licensing questions, which I will not discuss here. Ubuntu supports ZFS natively (I use 22.10). On other distributions you may need ZFS on Linux and some manual steps.

Create and roll back

To create a snapshot, run the zfs snapshot command with the file system name, an @ sign and a snapshot name:

zfs snapshot my_filesystem@snapshot_name

You can take several snapshots of the same file system under different names. To restore one:

zfs rollback my_filesystem@snapshot_name

This returns the file system to the state it had when the snapshot was taken.

List, compare and delete

To list your snapshots with their names and sizes, use zfs list -t snapshot. To see what changed between two snapshots, use zfs diff. To remove a snapshot you no longer need and free its space, use zfs destroy.

Practical habits

Take snapshots when the system is stable: a snapshot taken mid-upgrade captures a half-finished state, and rolling back to it will not help. Give each snapshot a name that says what it is for, so you can find the right one under pressure. Delete old snapshots regularly, so the pool keeps room for new ones.

Automating snapshots

I use a modified version of zfs-auto-snapshot, which takes snapshots on a schedule and keeps a set number of them. The original is at GitHub - zfsonlinux/zfs-auto-snapshot. My version is at GitHub - dzacca/zfs-auto-snapshot. The difference is that mine also creates a snapshot every time apt installs, upgrades or removes packages. That hook is what saved me in January 2023.

Restoring a system that no longer boots

After installing the displaylink driver I was asked to reboot, and the system then failed to boot. Ubuntu offered its emergency mode; I do not remember the exact name. From the recovery shell I listed the snapshots taken by the apt hook, filtered with grep to the relevant ones:

zfs list -t snapshot |grep apt|grep 2023-01-19-1453|grep ROOT
rpool/ROOT@zfs-auto-snap_apt-2023-01-19-1453                                              0B      -       96K  -
rpool/ROOT/ubuntu_8foh88@zfs-auto-snap_apt-2023-01-19-1453                              180M      -     11.2G  -
rpool/ROOT/ubuntu_8foh88/srv@zfs-auto-snap_apt-2023-01-19-1453                            0B      -       96K  -
rpool/ROOT/ubuntu_8foh88/usr@zfs-auto-snap_apt-2023-01-19-1453                            0B      -       96K  -
rpool/ROOT/ubuntu_8foh88/usr/local@zfs-auto-snap_apt-2023-01-19-1453                      8K      -     6.01G  -
rpool/ROOT/ubuntu_8foh88/var@zfs-auto-snap_apt-2023-01-19-1453                            0B      -       96K  -
rpool/ROOT/ubuntu_8foh88/var/games@zfs-auto-snap_apt-2023-01-19-1453                      0B      -       96K  -
rpool/ROOT/ubuntu_8foh88/var/lib@zfs-auto-snap_apt-2023-01-19-1453                        0B      -       96K  -
rpool/ROOT/ubuntu_8foh88/var/lib/AccountsService@zfs-auto-snap_apt-2023-01-19-1453        0B      -      364K  -
rpool/ROOT/ubuntu_8foh88/var/lib/NetworkManager@zfs-auto-snap_apt-2023-01-19-1453         0B      -      148K  -
rpool/ROOT/ubuntu_8foh88/var/lib/apt@zfs-auto-snap_apt-2023-01-19-1453                  184K      -     87.0M  -
rpool/ROOT/ubuntu_8foh88/var/lib/dpkg@zfs-auto-snap_apt-2023-01-19-1453                 616K      -     55.6M  -
rpool/ROOT/ubuntu_8foh88/var/log@zfs-auto-snap_apt-2023-01-19-1453                     2.74M      -     1.40G  -
rpool/ROOT/ubuntu_8foh88/var/mail@zfs-auto-snap_apt-2023-01-19-1453                      80K      -      248K  -
rpool/ROOT/ubuntu_8foh88/var/snap@zfs-auto-snap_apt-2023-01-19-1453                       8K      -     2.39M  -
rpool/ROOT/ubuntu_8foh88/var/spool@zfs-auto-snap_apt-2023-01-19-1453                    128K      -     4.70M  -
rpool/ROOT/ubuntu_8foh88/var/www@zfs-auto-snap_apt-2023-01-19-1453                        0B      -       96K  -

The entries that matter are those with a USED size above zero, because they hold changes made since the snapshot. I rolled back three:

rpool/ROOT/ubuntu_8foh88@zfs-auto-snap_apt-2023-01-19-1453   
rpool/ROOT/ubuntu_8foh88/var/lib/apt@zfs-auto-snap_apt-2023-01-19-1453 
rpool/ROOT/ubuntu_8foh88/var/lib/dpkg@zfs-auto-snap_apt-2023-01-19-1453 

with these commands:

zfs rollback rpool/ROOT/ubuntu_8foh88@zfs-auto-snap_apt-2023-01-19-1453   
zfs rollback rpool/ROOT/ubuntu_8foh88/var/lib/apt@zfs-auto-snap_apt-2023-01-19-1453 
zfs rollback rpool/ROOT/ubuntu_8foh88/var/lib/dpkg@zfs-auto-snap_apt-2023-01-19-1453 

I rebooted and the system was back in its previous stable state.

I left the other datasets alone because I did not need to restore the logs, and I knew my home directory was unaffected.