My home server is a ZimaBoard 2 running ZimaOS. The storage that matters is a RAID1 mirror, two 4 TB drives formatted BTRFS, holding a media library, backups, and personal archives, with the Docker application stack running off the same pool. It is a small setup, and for a long time it was a boring one, which is exactly what you want from storage.

Then a migration went wrong, and it taught me more than a year of everything working ever did.

What actually happened

I was moving applications and data around on ZimaOS when the system root filesystem filled completely. The migration stalled and deadlocked. That part is ordinary; a full disk is a full disk. The interesting part came after.

Once the root filesystem was saturated, the storage orchestration layer could no longer reliably attach the RAID1 pool, even though nothing was actually wrong with the array. mdadm reported both disks healthy and the mirror intact, the reassuring [UU]. But underneath that, the system was confused:

  • /dev/md* device nodes were created inconsistently between boots.
  • The array would assemble by hand but refuse to stay mounted.
  • BTRFS was detected at block level, but its subvolumes were never enumerated.
  • The ZimaOS storage service kept racing my manual mounts and undoing them.
  • In the UI, the pool looked empty, as if it had never been initialised, despite the metadata being right there.

Kernel logs told the real story: the filesystem mounted and unmounted cleanly when I did it myself, so the data was almost certainly fine. What had broken was not the disks and not BTRFS. It was the orchestration layer’s idea of the disks, which had lost coherent state when the migration was interrupted and the root filesystem ran dry.

The tempting mistake

At this point there is a fork in the road, and one path is a trap.

The trap is to keep going. The mirror is readable. mdadm is happy. You can almost taste the in-place recovery, and every reattachment attempt gets you a little closer to a pool that mounts. So you try one more thing, and one more thing after that.

The problem is that I could not answer a simple question with confidence: after a failed migration, a full root filesystem, an OS reinstall, and a dozen manual mdadm and BTRFS reattach attempts, was the long-term state actually clean? Snapshots, subvolume integrity, future upgrades, all of it now carried a question mark. The array was primary production storage for me, so “probably fine” was not good enough. What I wanted was deterministic, and a half-recovered orchestration state is the opposite of deterministic.

The decision

Before doing anything irreversible, I confirmed three things:

  1. The RAID1 mirror was readable.
  2. Each disk held a full, independent copy of the data.
  3. A complete external backup existed and had been verified, not just created.

With that in hand, the call was easy:

Wipe, reinitialise, and resynchronise the array, then restore from backup.

Not because recovery was impossible. It was probably possible. But the time cost, the uncertainty, and the operational risk of nursing a partially recovered state all outweighed the benefit once I had a verified backup sitting safely off the machine. A clean rebuild gave me a known-good baseline: fresh BTRFS, clean storage-service integration, no latent inconsistencies waiting to surface during some future upgrade.

The rebuild itself was uneventful. Data secured off-system, pool recreated, filesystem formatted, data resynced, application stack redeployed. Boring again, which was the point.

What I changed afterwards

The incident was really a lesson about a boundary I had not drawn clearly enough: the OS and the data are not the same class of thing.

  • Treat the OS as disposable and the data as authoritative. RAID1 protected the bytes. It did nothing for the orchestration state on top, and I had been quietly relying on that state being recoverable. It is not something to rely on.
  • A full, verified backup before any migration is non-negotiable. The only reason the clean-rebuild option existed was that the backup was already there and already checked.
  • Keep headroom on the system partition. Root filesystem exhaustion was the first domino. A few gigabytes of free space would have prevented the whole thing.
  • Keep application data on the pool, not the system disk, so the OS can be wiped without taking the apps’ state with it.
  • Snapshots and documented rescue procedures, so the next bad day has a runbook instead of improvisation.

RAID1 did its job perfectly. It kept every byte of my data safe through an OS-level mess. The mistake was ever expecting it to keep the system’s understanding of that data safe too. Mirror integrity and clean reintegration are different guarantees, and only one of them comes from the redundancy.