CachyOS Btrfs Snapshot Recovery — Conversation Log¶

Problem¶

Snapper snapshot restore failed. Could initially log in to an old snapshot, but it was read-only. Followed a guide to restore that image as the login one — afterward, the system wouldn't boot at all, throwing filesystem errors ("you are on your own"). Booted a CachyOS USB to investigate.

Step 1 — Initial diagnosis plan¶

Asked for lsblk -f output to identify the drive, then proposed:

Mount the btrfs top-level volume (subvolid=5) to see all subvolumes regardless of default.
List subvolumes with btrfs subvolume list -t.
Find the old known-good snapshot and set it as default with btrfs subvolume set-default.
If mounting subvolid=5 itself failed, run a read-only btrfs check before considering --repair.
Once mounted correctly, chroot in via arch-chroot after mounting all subvolumes and binding /proc, /sys, /dev, /run.
Regenerate initramfs and reinstall Limine inside the chroot.

Step 2 — lsblk output¶

sda
├─sda1 vfat   FAT32
└─sda2 btrfs   (auto-mounted at /run/media/liveuser/...)

The disk automounted cleanly (no mount errors) — a good sign the filesystem itself wasn't badly corrupted.

Instructed to unmount the automount, then:

sudo mount -o subvolid=5 /dev/sda2 /mnt/btrfs
ls /mnt/btrfs
sudo btrfs subvolume list -t /mnt/btrfs
sudo btrfs subvolume get-default /mnt/btrfs

Step 3 — Subvolume listing¶

Output showed subvolumes: @home, @root, @srv, @cache, @tmp, @log, @.broken (with nested numbered snapper snapshots 2475–2484), a timeshift-btrfs snapshot, and one entry (ID 1981) with a blank path.

get-default returned ID 5 (the raw top-level FS_TREE) — meaning the system wasn't booting into any real named subvolume, consistent with the automount showing a plain directory listing instead of an actual OS.

Notably: there was no subvolume named plain @ — only @.broken. This matched the classic snapshot-restore pattern: rename old @ → @.broken, then snapshot the chosen good snapshot in as the new @.

ID 1981 (blank path, top level 5, recent generation) was flagged as possibly an orphaned/half-created new @ from the restore attempt — told not to touch it yet.

Diagnostic commands given:

sudo ls -la /mnt/btrfs/
sudo btrfs inspect-internal subvolid-resolve 1981 /mnt/btrfs
for n in 2475 2476 2477 2478 2479 2480 2481 2482 2483 2484; do
  echo "== $n =="
  sudo cat /mnt/btrfs/@.broken/.snapshots/$n/info.xml 2>/dev/null
done

Step 4 — ls -la result¶

Actual result showed @ did exist, alongside @.broken, but both displayed as dr-xr-xr-x (read-only-looking permissions) versus normal subvolumes showing drwxr-xr-x.

Diagnosis: the restore guide had likely promoted a read-only snapshot directly to be @, instead of creating a new writable snapshot from it. A read-only root filesystem explains the "fs errors, you're on your own" boot message — systemd can't write to /var, create runtime files, etc.

Fix proposed:

sudo mv /mnt/btrfs/@ /mnt/btrfs/@.readonly-backup
sudo btrfs subvolume snapshot /mnt/btrfs/@.readonly-backup /mnt/btrfs/@
sudo ls -la /mnt/btrfs/ | grep '@$'

Step 5 — subvolid-resolve confirmation¶

sudo btrfs inspect-internal subvolid-resolve 1981 /mnt/btrfs
@

Confirmed ID 1981 was the read-only @ seen in the listing — not a separate orphaned subvolume. No additional corruption; just the one read-only-root issue.

Step 6 — First fix attempt didn't change anything¶

After running the mv + snapshot commands, @ still showed dr-xr-xr-x, same timestamp as before — suggesting the commands hadn't actually run, or something silently failed. Asked for full command output including any errors, and to try remount,rw if needed.

Step 7 — Snapshot creation confirmed, but still read-only-looking¶

Output showed:

Create snapshot of '/mnt/btrfs/@.readonly-backup' in '/mnt/btrfs/@'

— so the commands did work, but the new snapshot also showed dr-xr-xr-x.

Initial (incorrect) theory: btrfs-level ro property was set on the subvolume. Checked with:

sudo btrfs property get /mnt/btrfs/@ ro

Result: false — so the btrfs ro flag wasn't the cause after all.

Step 8 — Real cause: directory permission bits¶

Correct diagnosis: the dr-xr-xr-x was the actual Unix permission mode of the top-level directory (555 — read+execute only), which got carried over from the source snapshot's inode permissions into every new snapshot made from it.

Confirmed via:

sudo stat /mnt/btrfs/@

Output showed Access: (0555/dr-xr-xr-x).

Fix:

sudo chmod 755 /mnt/btrfs/@
sudo stat /mnt/btrfs/@

Result confirmed: Access: (0755/drwxr-xr-x) — fixed.

Step 9 — Confirm subvolume ID and set default¶

sudo btrfs subvolume list -t /mnt/btrfs

Showed the new @ as ID 1983 (top level 5, matching real subvolumes — as opposed to 1981/1982 which were the readonly-backup and its nested copy).

sudo btrfs subvolume set-default 1983 /mnt/btrfs
sudo btrfs subvolume get-default /mnt/btrfs

Step 10 — Mount subvolumes and chroot¶

sudo mkdir -p /mnt/root
sudo mount -o subvol=@ /dev/sda2 /mnt/root
sudo mount -o subvol=@home /dev/sda2 /mnt/root/home
sudo mount -o subvol=@log /dev/sda2 /mnt/root/var/log
sudo mount -o subvol=@cache /dev/sda2 /mnt/root/var/cache
sudo mount -o subvol=@tmp /dev/sda2 /mnt/root/tmp
sudo mount -o subvol=@srv /dev/sda2 /mnt/root/srv
sudo mount -o subvol=@root /dev/sda2 /mnt/root/root
sudo mount /dev/sda1 /mnt/root/boot/efi   # (later corrected — see below)

sudo mount -t proc /proc /mnt/root/proc
sudo mount --rbind /sys /mnt/root/sys
sudo mount --rbind /dev /mnt/root/dev
sudo mount --rbind /run /mnt/root/run

sudo arch-chroot /mnt/root

The /boot/efi mount failed ("mount point does not exist"), but arch-chroot itself succeeded regardless (prompt changed to root@CachyOS /).

Step 11 — Locating the correct ESP mount point¶

Confusion initially arose from checking /mnt/root/etc/fstab after already being inside the chroot (where /mnt/root no longer refers to anything, since chroot remaps /). Corrected by checking directly:

cat /etc/fstab
ls -la /

fstab revealed the ESP actually mounts at /boot, not /boot/efi:

UUID=6670-2BBD  /boot  vfat  defaults,umask=0077 0 2
UUID=387838e1...  /              btrfs  subvol=/@,...
UUID=387838e1...  /home          btrfs  subvol=/@home,...
UUID=387838e1...  /root          btrfs  subvol=/@root,...
UUID=387838e1...  /srv           btrfs  subvol=/@srv,...
UUID=387838e1...  /var/cache     btrfs  subvol=/@cache,...
UUID=387838e1...  /var/tmp       btrfs  subvol=/@tmp,...
UUID=387838e1...  /var/log       btrfs  subvol=/@log,...

ls -la / confirmed a proper populated root filesystem (etc, usr, var, etc. all present) — allaying the earlier fear that the snapshot might have been partial/empty.

Step 12 — Mount ESP and finish bootloader¶

mount /dev/sda1 /boot
ls /boot

Result: EFI, intel-ucode.img, limine.conf, limine.conf.old, loader — all present, correct layout confirmed.

which limine-update
mkinitcpio -P
pacman -S linux-cachyos --noconfirm   # (reinstall kernel package to retrigger Limine hooks)

Then:

exit
sudo umount -R /mnt/root
sudo reboot

Result¶

System booted successfully.

Post-recovery checks¶

Recommended:

sudo btrfs scrub start -B /
sudo btrfs filesystem usage /
journalctl -b -p err
sudo snapper list

journalctl -b -p err showed mostly harmless noise (TDX unsupported, dbus-broker duplicate name, keyring unlock issue from restored older keyring state) plus one real finding:

limine-snapper-sync[7275]: /.snapshots is not a Btrfs subvolume.

Cause: snapshotting @.readonly-backup into the new @ collapsed the nested /.snapshots subvolume into a plain empty directory (btrfs snapshots don't preserve child subvolumes as subvolumes). This meant snapper's dedicated snapshot-storage subvolume was gone, even though the actual old numbered snapshots (2475–2484) were still safely preserved under @.broken/.snapshots.

Fix:

sudo btrfs subvolume show /.snapshots   # confirms the problem
sudo rmdir /.snapshots
sudo btrfs subvolume create /.snapshots
sudo chmod 750 /.snapshots
sudo snapper -c root create-config /
sudo snapper list

snapper list afterward showed a clean baseline entry — confirming snapper was working again, just starting fresh (old history archived but disconnected, still recoverable from @.broken/.snapshots if ever needed).

Suggested (optional, no rush) cleanup of leftover recovery artifacts once stability is confirmed over a few days:

sudo btrfs subvolume delete /.readonly-backup
sudo btrfs subvolume delete /@.broken/.snapshots/2475/snapshot   # ...repeat for 2476–2484
sudo btrfs subvolume delete /@.broken/.snapshots
sudo btrfs subvolume delete /@.broken

Root cause summary¶

The original restore guide likely promoted a read-only snapshot directly into the @ position (either via manual rename or a similar approach), rather than creating a writable snapshot of it. A read-only root filesystem causes systemd to fail at boot with filesystem errors, since it can't write to /var, create runtime files, journal entries, etc.

Better approach for next time¶

Use snapper rollback instead of manual btrfs subvolume surgery:

sudo snapper -c root list          # find the snapshot number to restore to
sudo snapper -c root rollback <N>

This automatically: - Creates a writable snapshot of the target snapshot - Sets it as the new default subvolume - Preserves the current (broken) @ for comparison, rather than destroying it

Notes: - Needs to be run either from within a system that still boots enough to have /.snapshots mounted and snapper functional, or from a chroot into the broken system via live USB (same chroot process used in this recovery). - Don't mix Timeshift's snapshots with snapper's manual rollback — Timeshift (also present on this disk as a separate timeshift-btrfs subvolume) has its own restore mechanism.

Snapper config review¶

snapper -c root get-config showed a healthy config, with one notable setting:

TIMELINE_CREATE: no — automatic time-based snapshots (hourly/daily/etc.) are disabled. Snapshots currently only occur on pacman package transactions (pre/post install hooks). This is normal default behavior for CachyOS/Arch-based snapper setups, but means there's no periodic safety net for non-package changes (config edits, manual file changes, etc).

Optional improvement, to add a time-based fallback:

sudo snapper -c root set-config TIMELINE_CREATE=yes
sudo snapper -c root set-config TIMELINE_LIMIT_DAILY=7
sudo snapper -c root set-config TIMELINE_LIMIT_WEEKLY=4

sudo systemctl enable --now snapper-timeline.timer
sudo systemctl enable --now snapper-cleanup.timer