Lots of files missing from btrfs RAID after crash (power outage)

Support
1

Disclaimer
This is more about btrfs than about Manjaro. I’m however not sure where else to post because I don’t want to feed Stack Overflow anymore, and I find Reddit kind of icky. I haven’t found other suitable places, so here we are. Do feel free to point me to a better place to get helpful support.

Assuming that at least a few people will read this and think “this guy is sure living dangerous”. You are correct. I actually have a nice automated, versioned online backup using Borg. I just haven’t bothered to include this particular file system in it. Yes, I should have. I also do kind of live on the edge with that crazy RAID set-up of mine, I’m aware. The thing is: I don’t do normal. In fact, nobody who uses Manjaro does normal, so let’s just embrace that, shall we? I’m just trying to make my specific not-normal setup work.

My machine

I’ve collected a set of 10 hard drives over the years and am using them in a heterogenous btrfs RAID 10 array. I actually started out with RAID 5, but stopped that when the write-hole started causing data loss whenever something crashed, or broke. RAID 10 should actually be quite safe. I thought…

Device info from inxi:

❯ sudo inxi -Dpzv1
System:
  Kernel: 6.18.38-1-MANJARO arch: x86_64 bits: 64
  Desktop: KDE Plasma v: 6.6.6 Distro: Manjaro Linux
CPU:
  Info: 12-core AMD Ryzen 9 5900X [MT MCP] speed (MHz): avg: 1731
    min/max: 567/4955
Graphics:
  Device-1: Advanced Micro Devices [AMD/ATI] Navi 24 [Radeon RX 6400/6500
    XT/6500M] driver: amdgpu v: kernel
  Display: unspecified server: X.Org v: 21.1.24 with: Xwayland v: 24.1.13
    driver: X: loaded: amdgpu unloaded: modesetting,radeon dri: radeonsi
    gpu: amdgpu resolution: N/A
  API: OpenGL v: 4.6 vendor: amd mesa v: 26.1.4-arch1.1 renderer: AMD
    Radeon RX 6500 XT (radeonsi navi24 ACO DRM 3.64 6.18.38-1-MANJARO)
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo
    de: kscreen-console,kscreen-doctor gpu: corectrl wl: wayland-info
    x11: xdriinfo, xdpyinfo, xprop, xrandr
Drives:
  Local Storage: total: 8 TiB used: 2.18 TiB (27.3%)
  ID-1: /dev/sda vendor: Patriot model: Burst size: 447.13 GiB
  ID-2: /dev/sdb vendor: Western Digital model: WD6400AAKS-65A7B2
    size: 596.17 GiB
  ID-3: /dev/sdc vendor: Western Digital model: WD6400AACS-00G8B0
    size: 596.17 GiB
  ID-4: /dev/sdd vendor: Western Digital model: WD6400AAKS-65A7B2
    size: 596.17 GiB
  ID-5: /dev/sde vendor: Western Digital model: WD6400AAKS-00A7B0
    size: 596.17 GiB
  ID-6: /dev/sdf vendor: Toshiba model: HDWD120 size: 1.82 TiB
  ID-7: /dev/sdg vendor: Western Digital model: WD2500JD-75HBB0
    size: 232.83 GiB
  ID-8: /dev/sdh vendor: Toshiba model: HDWD120 size: 1.82 TiB
  ID-9: /dev/sdi vendor: Hitachi model: HDS7250SASUN500G 0711KSW1AH
    size: 465.76 GiB
  ID-10: /dev/sdj vendor: Hitachi model: HDS7250SASUN500G 0711KT275F
    size: 465.76 GiB
  ID-11: /dev/sdk vendor: Hitachi model: HDS7250SASUN500G 0711KT21WF
    size: 465.76 GiB
Partition:
  ID-1: / size: 446.83 GiB used: 374.45 GiB (83.8%) fs: btrfs dev: /dev/sda2
  ID-2: /boot/efi size: 299.4 MiB used: 568 KiB (0.2%) fs: vfat
    dev: /dev/sda1
  ID-3: /bulk size: 3.06 TiB used: 1.76 TiB (57.4%) fs: btrfs dev: /dev/sdd2
  ID-4: /home size: 446.83 GiB used: 374.45 GiB (83.8%) fs: btrfs
    dev: /dev/sda2
  ID-5: /tempraid size: 200 GiB used: 60.59 GiB (30.3%) fs: btrfs
    dev: /dev/sdi2
  ID-6: /var/cache size: 446.83 GiB used: 374.45 GiB (83.8%) fs: btrfs
    dev: /dev/sda2
  ID-7: /var/log size: 446.83 GiB used: 374.45 GiB (83.8%) fs: btrfs
    dev: /dev/sda2
  ID-8: swap-1 size: 10 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdf1
  ID-9: swap-2 size: 10 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdj1
  ID-10: swap-3 size: 10 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdi1
  ID-11: swap-4 size: 10 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdk1
  ID-12: swap-5 size: 10 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sdg1
Info:
  Memory: total: 64 GiB available: 62.69 GiB used: 8.3 GiB (13.2%)
  Processes: 571 Uptime: 3h 6m Shell: Sudo inxi: 3.3.41

Those 10 disks are partitioned, with one swap partition each, one small partition contributing to a RAID 0 filesystem for temporary use (not /tmp, since I’m aware that 10-year-old hard drives in a RAID 0 are just asking for trouble). Most of the space goes into a big old RAID 10 array, called bulk, and mounted to /bulk:

❯ sudo btrfs fi show /bulk
Label: 'bulk'  uuid: 82006e2d-149a-409d-bc85-7a30e43ea409
        Total devices 10 FS bytes used 1.76TiB
        devid    1 size 576.17GiB used 378.02GiB path /dev/sdd2
        devid    2 size 576.17GiB used 375.55GiB path /dev/sdb2
        devid    3 size 576.17GiB used 373.45GiB path /dev/sdc2
        devid    4 size 576.17GiB used 372.04GiB path /dev/sde2
        devid    5 size 435.76GiB used 294.82GiB path /dev/sdi3
        devid    6 size 1.20TiB used 583.56GiB path /dev/sdf3
        devid    7 size 202.83GiB used 168.08GiB path /dev/sdg3
        devid    8 size 435.76GiB used 296.70GiB path /dev/sdk3
        devid    9 size 435.76GiB used 290.97GiB path /dev/sdj3
        devid   10 size 1.20TiB used 576.56GiB path /dev/sdh3

Here’s what this looked like in February (kept a note of it because it was running full. Have deleted about 100 GiB since then, so no worries about space shortage)

❯  sudo  btrfs fi us /bulk                                                                                                                                                                                                                                           
Overall:
    Device size:                   6.12TiB
    Device allocated:              5.45TiB
    Device unallocated:          695.54GiB
    Device missing:                  0.00B
    Device slack:                 12.00KiB
    Used:                          5.43TiB
    Free (estimated):            354.55GiB      (min: 180.67GiB)
    Free (statfs, df):           350.59GiB
    Data ratio:                       2.00
    Metadata ratio:                   4.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID10: Size:2.71TiB, Used:2.71TiB (99.76%)
   /dev/sdd2     561.02GiB
   /dev/sdb2     561.02GiB
   /dev/sdc2     563.05GiB
   /dev/sde2     563.00GiB
   /dev/sdg3     425.07GiB
   /dev/sdi3     922.66GiB
   /dev/sdk3     194.78GiB
   /dev/sdh3     427.71GiB
   /dev/sdf3     425.07GiB
   /dev/sdj3     916.74GiB

Metadata,RAID1C4: Size:4.01GiB, Used:3.50GiB (87.14%)
   /dev/sdd2       4.00GiB
   /dev/sdb2       4.01GiB
   /dev/sdc2       2.00GiB
   /dev/sde2       2.01GiB
   /dev/sdi3       3.01GiB
   /dev/sdj3       1.01GiB

System,RAID1C4: Size:32.50MiB, Used:688.00KiB (2.07%)
   /dev/sdd2      32.25MiB
   /dev/sdb2     256.00KiB
   /dev/sde2      32.50MiB
   /dev/sdi3      32.50MiB
   /dev/sdj3      32.50MiB

Unallocated:
   /dev/sdd2      11.12GiB
   /dev/sdb2      11.13GiB
   /dev/sdc2      11.12GiB
   /dev/sde2      11.13GiB
   /dev/sdg3      10.69GiB
   /dev/sdi3     302.82GiB
   /dev/sdk3       8.05GiB
   /dev/sdh3       8.05GiB
   /dev/sdf3      10.69GiB
   /dev/sdj3     310.74GiB

What happened

Last week, late at night, the power went off while my PC was running. It was way too late, so I left it off and went to bed.
Yesterday, I noticed a few files missing. Because a large part of the storage is devoted to movies, I assumed that I had moved those to another place and kept going. I also have an automated scrub that runs once per week. That also went through yesterday, without issue. Something looked a bit funny, though, so I also ran a balance (`dusage=50’), for good measure. That was probably a big mistake.

Because today, I looked at some of the other contents, and found that a lot of stuff went missing.
The whole directory structure is exactly as it was, but the files are missing. Not just all the movies and videos I kept, but also a huge part of my Photos (which are backed up, mostly, but could be a lot of work to restore).

Now, it looks like this:

❯ sudo btrfs fi us /bulk
[sudo] Passwort für sascha: 
Overall:
    Device size:                   6.12TiB
    Device allocated:              3.62TiB
    Device unallocated:            2.50TiB
    Device missing:                  0.00B
    Device slack:                 12.00KiB
    Used:                          3.52TiB
    Free (estimated):              1.30TiB      (min: 691.50GiB)
    Free (statfs, df):             1.25TiB
    Data ratio:                       2.00
    Metadata ratio:                   4.00
    Global reserve:               49.88MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID10: Size:1.80TiB, Used:1.75TiB (97.24%)
<omitted details about which disk has how much or not...>

So … there’s about 2 TiB more space unallocated than there was in February, which makes for about 1 TiB of missing files.

What I’ve tried so far

I’ve unmounted the file system and ran btrfs check

❯ sudo btrfs check /dev/sdg3
Opening filesystem to check...
Checking filesystem on /dev/sdg3
UUID: 82006e2d-149a-409d-bc85-7a30e43ea409
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
[4/8] checking free space tree
Space key logical 1048576 length 4194304 has no corresponding block group
Space key logical 5242880 length 8388608 has no corresponding block group
Space key logical 13631488 length 8388608 has no corresponding block group
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 1930731331584 bytes used, error(s) found
total csum bytes: 1883122512
total tree bytes: 2411249664
total fs tree bytes: 337723392
total extent tree bytes: 51675136
btree space waste bytes: 184598204
file data blocks allocated: 1928480935936
 referenced 1921613320192

If I use the backup root (option -b), I get almost the same output but some of the numbers are different, and an error message about mismatch in superblock bytes:

❯ sudo btrfs check -b /dev/sdg3
Opening filesystem to check...
Checking filesystem on /dev/sdg3
UUID: 82006e2d-149a-409d-bc85-7a30e43ea409
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
super bytes used 1930731331584 mismatches actual used 1930731347968
ERROR: errors found in extent allocation tree or chunk allocation
[4/8] checking free space tree
Space key logical 1048576 length 4194304 has no corresponding block group
Space key logical 5242880 length 8388608 has no corresponding block group
Space key logical 13631488 length 8388608 has no corresponding block group
[5/8] checking fs roots
[6/8] checking only csums items (without verifying data)
[7/8] checking root refs
[8/8] checking quota groups skipped (not enabled on this FS)
found 1930731347968 bytes used, error(s) found
total csum bytes: 1883122512
total tree bytes: 2411266048
total fs tree bytes: 337739776
total extent tree bytes: 51675136
btree space waste bytes: 184613010
file data blocks allocated: 1928480935936
 referenced 1921613320192

I also tried repeating this with the -s option to use superblock 0, 1 and 2. For 0 and 1, I get identical output, for 2:

❯ sudo btrfs check -s 2 /dev/sdg3
using SB copy 2, bytenr 274877906944
Opening filesystem to check...
ERROR: superblock bytenr 274877906944 is larger than device size 217786089472
ERROR: cannot open file system

I tried various incantations of btrfs restore (with -D option because I currently have nothing to save 1 TiB of restored files to), which list no files at all but mention that they skipped some snapshots, although I don’t think there are any snapshots on the system, at least none that I made explicitly. If I use the -s option to include snapshots, the result is a segmentation fault:

❯ sudo btrfs restore -sD /dev/sdb2 ./btrfs-restore_fake
This is a dry-run, no files are going to be restored
zsh: segmentation fault  sudo btrfs restore -sD /dev/sdb2 ./btrfs-restore_fake

…that’s as far as I got.
I have not yet been through all the things I could try with btrfs check, btrfs rescue and btrfs restore.

My Question

The main thing I’d like to get from this forum is some helpful hint about what to try next, because at this point my idea of what went wrong is still somewhat hazy: Maybe btrfs was busy balancing or scrubbing in the background when the power went off, this caused some data to be lost, and the next scrub misinterpreted what it found and restored RAID 10 consistency by picking the wrong version of what it found.

What I don’t know is:

  • Whether that is actually plausible or I’m just guessing wrong.
  • How I can get as much data back as possible without trying every thing that might just work, in random order. I’m sure there’s a more structured approach that I’m missing
  • How I could avoid this scenario in the future (without increasing redundancy even further). I have four copies of metadata and system data, for crying out loud. btrfs should have been able to know what’s correct and what isn’t. If that’s not doing it, what is?
2
RAID 5 is a recipe for disaster with Btrfs.

The developers explicitly warn against using it in production. So, if your volume was RAID 5 at any point, that is a problem that could still manifest itself later on.
RAID 5/6 is experimental. Anyone finished with the experiment should wipe the volume and create a new one. Anything else is grossly negligent.

After a power outage, it is necessary to assess the damage.

If a Btrfs system with 10 hard drives experiences a power failure, it is possible that every single drive will contain errors afterward. Each drive has its own electronics and its own hardware write cache.

This is unlikely to cause problems if no data is being written at the time. However, if large amounts of data are being written to the RAID at that exact moment, all 10 drives could be affected.

In such a case, it is strongly recommended to initiate a manual “scrub” after restarting the device to check the state of the file system. Naturally, this is time-consuming with mechanical hard drives, as all the data must be read, but that shouldn’t deter you from doing it.

99.76% – are you serious?

What can I say to that?

I can only point out that btrfs reacts badly if you fill it up too much.

Something was already wrong there back in February.

This distribution suggests that the volume was modified at some point, but that a full balance operation was not performed afterwards (or that something was already amiss at that time). Btrfs normally tends to distribute its chunks evenly across all partitions (meaning each one ends up being filled to the same percentage).

I think you’re guessing wrong. Scrub uses the block’s checksum to identify the correct block.

If a block with a corrupted checksum is found, it is replaced by the copy with the correct checksum. This happens at the block level and works without regard to how the blocks are used within the file system.

I’m sorry. I can’t help with that.

Here are a few tips:

  • Make backups.
  • This cannot be said often enough: make regular backups.
  • Do not use experimental features (ever), or clean up thoroughly afterwards (discard the volume and set up a new one).
  • Never fill a btrfs volume to 99% capacity.
  • After a power outage, definitely perform a manual scrub.
  • Avoid using too many (potentially old) drives.
  • Do not create too many partitions per drive if you are using a large number of drives (ideally, just one partition per drive).
  • Do not spread your volumes across all available drives; instead, group only the number of drives necessary for a specific volume (though you should use at least two partitions on separate drives).

For classic RAID setups, using identical drives of the same size is recommended. For btrfs, however, I always advise—in the interest of minimizing risk—using different drives from different manufacturers and with varying capacities.

Losing data with btrfs is an art form—but certainly not impossible.

:footprints:

3

Hello @Mr_Teatime :wink:

Yes, the ‘balance’ command was probably the problem at that point because it created a new root tree and assumed that the missing files were not there. This probably explains it: https://unix.stackexchange.com/questions/340947/does-btrfs-guarantee-data-consistency-on-power-outages

From now on, I would mount /bulk as read-only to prevent anything from being overwritten in the background.

What I would try is to take an old root tree and restore the data exactly as it was.

btrfs-find-root /dev/sdXY

and then restore it by the byte number:

sudo btrfs restore -D -t <bytenr> --path-regex '^/(|Photos(|/.*)|Videos(|/.*))$' /dev/sdX ./test

This doesn’t restore anything, but it shows whether a directory has been overwritten.

This wrapper does the same thing, but in a more convenient way:

Of course, to recover files, you’ll need a second storage device.

4

That should not be possible. Both a balance and a scrub are interruptible operations and will always leave the filesystem in a consistent state.

However, as @andreas85 said, you originally had a btrfs RAID 5, and that’s an unsupported (and broken) implementation, plus that your filesystem is full.

5

To clarify: I had that RAID 5 for a few months, and switched to RAID 10 when installing Manjaro, in 2022 or so. Did a proper balance at the time, too, to make sure that everything is properly duplicated. Pretty sure that this is no longer causing issues.

6

Well, a full filesystem certainly will. Even the rather primitive ext4 already has problems with that, but on btrfs it’s exponentially worse due to the copy-on-write — too many intertwined extents, and no spare disk space for the filesystem to perform any maintenance tasks.

A balance operation is such a maintenance task requiring a certain amount of free space, with the required amount being proportional to the amount of data that needs to be processed.

9

You could, for example, say something along the lines of:

Yes, I wonder about that too. I think it had to do with the fact that two of the drives are much bigger than the others, and the RAID 10 setup somehow failed to distribute the data such that those two could find mirrors elsewhere. Which, according my understanding, it should actually have done.
However, that output is from February, and I only posted it because it’s the last output of btrfs fi us that I have on record from before this week’s accident. The issue I was dealing with in February: btrfs kept telling me I had a few hundred GiB space but the operating system thought otherwise. I likely had to do with the fact that two of the drives are much bigger than the others, and btrfs did not manage to allocate space in a way that this space is matched elsewhere. I was not able to resolve this in February and decided to just remove some data from the filesystem, which allowed me to move on.

However: That’s not the issue I’m dealing with right now. I won’t complain if someone can give me a helpful pointer about it, though that’s not the priority right now, and I don’t expect this to impact the current situation. Although you never know, of course.

The filesystem is now unmounted, and I’ll keep it like that until all is done. According to my understanding, even mounting read-only merely prevents users from writing files but still permits btrfs maintenance processes to run and make changes to how the data is stored. So even that is out for now. The only way I believe to be safe is this:
> sudo mount -o ro,rescue=all /bulk
The rescue=all option is btrfs-specific and should prevent any accidental changes, including by system-internal processes.

To make that even more explicit: I manually verified that the scheduled scrub had completed, and it did.

Look, I really tried to put most relevant information in the first post. I get that it’s a rather long one, which makes it easy to overlook stuff, and there’s some detail that’s less relevant than other detail. The problem is: If I try to make it shorter, someone’s going to complain about missing information. The longer I make it, the higher the risk that someone’s going to post uninformed replies. I don’t find it productive to quote my own original post in response to half the comments because that only increases the entropy in this thread.

Mod edit: Consecutive posts merged.
If there are no intervening replies, please edit your previous post instead to add further information. Cheers! :wink:

10

No, that is not correct, although the journal can indeed get replayed when the filesystem is mounted read-only.

11

The command we needed, especially given the number of devices involved, was:

sudo btrfs device usage /bulk

That would give us some insight to your previous question.

Pointing out the line:

Data,RAID10: Size:2.71TiB, Used:2.71TiB (99.76%)

is actually fine; I suspect that was just a slip-up?

That doesn’t change the state it is in.

(Which using parity RAID most likely would have done.)

13

That sounds like a good idea. I’ve tried btrfs restore -D before, and got nothing out of it.
However, when I run find-root, this happens:

❯ sudo btrfs-find-root /dev/sdc2
Superblock thinks the generation is 128047
Superblock thinks the level is 1
Found tree root at 2615596580864 gen 128047 level 1

It produces the same output for every device in the array, which I would expect for an intact array.
That root is probably the one that’s used if I mount the filesystem, and the output I get is very similar to what I got before…

❯ sudo btrfs restore -D -t 2615596580864 /dev/sdb2 ~/btrfs-restore_fake
This is a dry-run, no files are going to be restored
Skipping snapshot snapshot
[lots more of these...]
Skipping snapshot snapshot
ERROR: reading subvolume /home/myusername/btrfs-restore_fake/@/@shared/.snapshots/3863/snapshot failed: 18446744073709551614
ERROR: searching directory /home/myusername/btrfs-restore_fake/@/@shared/.snapshots/3863/snapshot failed: -2
ERROR: searching directory /home/myusername/btrfs-restore_fake/@/@shared/.snapshots/3863/snapshot failed: -2
ERROR: searching directory /home/myusername/btrfs-restore_fake/@/@shared/.snapshots/3863/snapshot failed: -2
ERROR: searching directory /home/myusername/btrfs-restore_fake/@/@shared/.snapshots/3863/snapshot failed: -2

…except those errors at the end don’t appear if I don’t specify the root explicitly. I’m also a bit confused by them because that path is empty, supposed to be empty, not on the btrfs filesystem, and it’s where it was supposed to pretend (option -D) to save the restored files to, not read from.

I wonder if I should try a different root, but I’ve not had to dig this deep into btrfs, so I’m not sure what makes sense here. btrfs-find-roots -a outputs a huge number of things

❯ sudo btrfs-find-root -a /dev/sdc2
[sudo] Passwort für sascha: 
Superblock thinks the generation is 128047
Superblock thinks the level is 1
Well block 2615629512704(gen: 128047 level: 1) seems good, and it matches superblock
Well block 2615596580864(gen: 128047 level: 1) seems good, and it matches superblock
Well block 2615599857664(gen: 128046 level: 1) seems good, but generation/level doesn't match, want gen: 128047 level: 1
Well block 2615637278720(gen: 128043 level: 1) seems good, but generation/level doesn't match, want gen: 128047 level: 1
Well block 2615635394560(gen: 128042 level: 1) seems good, but generation/level doesn't match, want gen: 128047 level: 1
Well block 3625356738560(gen: 128041 level: 1) seems good, but generation/level doesn't match, want gen: 128047 level: 1
Well block 3625356165120(gen: 128041 level: 1) seems good, but generation/level doesn't match, want gen: 128047 level: 1
[...continues for 600 lines...]

The previously-found root is the second in that list, so I tried the first one too, for good measure, and found that it produces the same output as before, minus the error messages.

I also tried the third:

❯ sudo btrfs restore -D -t 2615599857664 /dev/sdb2 ~/btrfs-restore_fake
parent transid verify failed on 2615599857664 wanted 128047 found 128046
parent transid verify failed on 2615599857664 wanted 128047 found 128046
parent transid verify failed on 2615599857664 wanted 128047 found 128046
parent transid verify failed on 2615599857664 wanted 128047 found 128046
parent transid verify failed on 2615599857664 wanted 128047 found 128046
Ignoring transid failure
This is a dry-run, no files are going to be restored
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot
Skipping snapshot snapshot

…and fourth:

❯ sudo btrfs restore -D -t 3625336504320 /dev/sdb2 ~/btrfs-restore_fake
parent transid verify failed on 3625336504320 wanted 128047 found 128016
parent transid verify failed on 3625336504320 wanted 128047 found 128016
parent transid verify failed on 3625336504320 wanted 128047 found 128016
parent transid verify failed on 3625336504320 wanted 128047 found 128016
parent transid verify failed on 3625336504320 wanted 128047 found 128016
Ignoring transid failure
ERROR: root [1 0] level 0 does not match 1

Couldn't read tree root
Could not open root, trying backup super
warning, device 10 is missing
warning, device 6 is missing
warning, device 4 is missing
warning, device 1 is missing
warning, device 3 is missing
bad tree block 11194332086272, bytenr mismatch, want=11194332086272, have=0
ERROR: cannot read chunk root
Could not open root, trying backup super
warning, device 10 is missing
warning, device 6 is missing
warning, device 4 is missing
warning, device 1 is missing
warning, device 3 is missing
bad tree block 11194332086272, bytenr mismatch, want=11194332086272, have=0
ERROR: cannot read chunk root
Could not open root, trying backup super

Okay, at this point I don’t know what this means. I’m not sufficiently familiar with the internal organisation of btrfs to make sense of this, or to even know whether I actually did what you suggested. Is the regex important? From the manpage, I’d guess that restore should copy everything it can get unless it’s limited by the regex. So without it I should be getting a full listing of everything accessible on the filesystem, no? (My guess: no, otherwise there’d been something by now.)

So mount -o ro will prevent scrub, balance or other filesystem-internal processes?
I’ve read through a lot of more-or-less helpful stuff online today, and unfortunately cannot remember where I read that ro alone was not safe. I don’t think I saw it in the official documentation, so there’s a chance that I’m reproducing somebody else’s misconceptions – do you happen to have a reference?

Mod edit: Consecutive posts merged.

14

Yes, that is correct.

Just the man pages. :wink:

Addendum: I just thought of something that does need to be pointed out…

  1. If the filesystem has read-only mounted subvolumes and read/write-mounted subvolumes, then the scrub and balance operations do still work, and they will work on the whole filesystem, not just on the read/write-mounted subvolumes.

  2. If the filesystem has both read-only mounted subvolumes and read/write-mounted subvolumes, then once a read-only-mounted subvolume is remounted read/write, you cannot remount it as read-only anymore without a reboot. This is because as soon as the read-only subvolume gets remounted read/write, it becomes part of the writable pool along with the free space, and any attempt to remount it (or unmount it) will result in an error message that the filesystem is busy.

15

Goes to show that there’s always one more thing I did not happen to remember.

❯ sudo btrfs dev us /bulk
/dev/sdd2, ID: 1
   Device size:           576.17GiB
   Device slack:            3.00KiB
   Data,RAID10/8:         162.89GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          43.58GiB
   Data,RAID10/4:           4.47GiB
   Metadata,RAID1C4:        4.00GiB
   System,RAID1C4:        256.00KiB
   Unallocated:           198.15GiB

/dev/sdb2, ID: 2
   Device size:           576.17GiB
   Device slack:            3.00KiB
   Data,RAID10/8:         161.00GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          43.54GiB
   Data,RAID10/4:           3.92GiB
   Metadata,RAID1C4:        4.00GiB
   System,RAID1C4:        256.00KiB
   Unallocated:           200.62GiB

/dev/sdc2, ID: 3
   Device size:           576.17GiB
   Device slack:            3.00KiB
   Data,RAID10/8:         164.79GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          43.58GiB
   Data,RAID10/4:           5.00MiB
   Metadata,RAID1C4:        2.00GiB
   System,RAID1C4:          2.00MiB
   Unallocated:           202.72GiB

/dev/sde2, ID: 4
   Device size:           576.17GiB
   Device slack:            3.00KiB
   Data,RAID10/8:         161.91GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          43.58GiB
   Data,RAID10/4:           1.47GiB
   Metadata,RAID1C4:        2.00GiB
   System,RAID1C4:          2.50MiB
   Unallocated:           204.13GiB

/dev/sdi3, ID: 5
   Device size:           435.76GiB
   Device slack:              0.00B
   Data,RAID10/8:         131.74GiB
   Data,RAID10/10:        163.08GiB
   Unallocated:           140.94GiB

/dev/sdf3, ID: 6
   Device size:             1.20TiB
   Device slack:              0.00B
   Data,RAID10/8:         173.96GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          43.58GiB
   Data,RAID10/4:           4.94GiB
   Data,RAID10/2:         195.00GiB
   Metadata,RAID1C4:        3.00GiB
   System,RAID1C4:          2.50MiB
   Unallocated:           644.96GiB

/dev/sdg3, ID: 7
   Device size:           202.83GiB
   Device slack:              0.00B
   Data,RAID10/8:           5.00GiB
   Data,RAID10/10:        163.08GiB
   Unallocated:            34.75GiB

/dev/sdk3, ID: 8
   Device size:           435.76GiB
   Device slack:              0.00B
   Data,RAID10/8:         133.58GiB
   Data,RAID10/10:        163.08GiB
   Data,RAID10/6:          37.50MiB
   Unallocated:           139.06GiB

/dev/sdj3, ID: 9
   Device size:           435.76GiB
   Device slack:              0.00B
   Data,RAID10/8:         127.88GiB
   Data,RAID10/10:        163.08GiB
   Unallocated:           144.79GiB

/dev/sdh3, ID: 10
   Device size:             1.20TiB
   Device slack:              0.00B
   Data,RAID10/10:        163.08GiB
   Data,RAID10/8:         168.96GiB
   Data,RAID10/6:          43.58GiB
   Data,RAID10/4:           4.94GiB
   Data,RAID10/2:         195.00GiB
   Metadata,RAID1C4:        1.00GiB
   System,RAID1C4:          2.50MiB
   Unallocated:           651.96GiB

Looking at this myself, I notice:

  • the two largest partitions of 1.2 TiB have the lowest utilisation, which has been like this for a long time
  • They also seem to be the only drives with the Data,RAID10/2 profile, and that is interesting in the context of whether or not the space they provide can be used. So at some point, btrfs must have been using this, but if it had no problem allocating new space in this profile, it could be using the two big partitions completely, no problem.
  • probably more relevant for my vanished files: why have so many of the devices absolutely no slack? I’m not entirely certain when and how that’s used but intuition says that this can’t be good. Rigid systems fail badly.
16

Educated guess: because of the stripe size, pethaps?

17

Since it turns out that the man pages for btrfs are a little long reads, I’m linking that here, for posterity:

A filesystem mounted read-only will not affect the underlying block device in almost 100% (with highly unlikely exceptions). The exception is a tree-log that needs to be replayed during mount (and before the read-only mount takes place), working memory is needed for that and that can be affected by bit flips. There’s a theoretical case where bit flip changes the filesystem status from read-only to read-write.

…okay, that’s good enough for me.

The thing that had confused me previously was in the notes for the scrub command, but does not actually talk about read-only filesystems, just about the read-only option of the scrub command sometimes still writing (if the file system permits it)

18

Also remember:

sudo btrfs device stats /bulk

If there are errors, it will show you the device.

This is the most bizarre thing to me so far.

Every disk has a different type of RAID10 profile. Most of it is spread over 8 or 10 discs, but some 4 or 6?

This does not mean anything is corrupt though, and a simple balance will fix this part.

19

But do see my additional notes… :backhand_index_pointing_down:

21

That’s actually fairly clear to me:

Btrfs needs to allocate groups of at least 4 blocks on different devices. 6, 8 or 10 is also okay, of course, as this means more parallel stripes. 2 would be completely fine by me, though btrfs seems to have some reservations there.

Here’s the online calculator I used to find out whether I could use all the space provided by the disks, and the answer is yes, as the Btrfs calculator at carfax.org.uk tells me:

image
image855×728 49.8 KB

…however, that calculator has no problem making use of the RAID10/2 profile (that is: actually just RAID1), which my PC has been very reluctant to use. The remaining bits of RAID10/2 have not grown in a long time, and that’s what likely caused the issue I had in February, when part of the system claimed the filesystem was full, and btrfs reported a few hundred GiB of unallocated space but could not be persuaded to allocate that space.
I suppose that the use of RAID10/2 could be avoided while still using all available space, but that would require reducing the amount of space allocated to the 10 and 8 profiles, and that would require a more involved allocation algorithm than what btrfs uses – so I shan’t complain.

There is probably also some historical “imbalance” (if you want to call it that) because two hard drives failed over the last few years and were replaced with drives of different sizes.

Now, about those drive stats:

❯ sudo btrfs dev stats /bulk
[sudo] Passwort für sascha: 
[/dev/sdd2].write_io_errs    0
[/dev/sdd2].read_io_errs     0
[/dev/sdd2].flush_io_errs    0
[/dev/sdd2].corruption_errs  1
[/dev/sdd2].generation_errs  0
[/dev/sdb2].write_io_errs    0
[/dev/sdb2].read_io_errs     0
[/dev/sdb2].flush_io_errs    0
[/dev/sdb2].corruption_errs  0
[/dev/sdb2].generation_errs  0
[/dev/sdc2].write_io_errs    5464344
[/dev/sdc2].read_io_errs     5219316
[/dev/sdc2].flush_io_errs    0
[/dev/sdc2].corruption_errs  13840
[/dev/sdc2].generation_errs  0
[/dev/sde2].write_io_errs    0
[/dev/sde2].read_io_errs     0
[/dev/sde2].flush_io_errs    0
[/dev/sde2].corruption_errs  50
[/dev/sde2].generation_errs  0
[/dev/sdi3].write_io_errs    0
[/dev/sdi3].read_io_errs     0
[/dev/sdi3].flush_io_errs    0
[/dev/sdi3].corruption_errs  0
[/dev/sdi3].generation_errs  0
[/dev/sdf3].write_io_errs    0
[/dev/sdf3].read_io_errs     0
[/dev/sdf3].flush_io_errs    0
[/dev/sdf3].corruption_errs  14
[/dev/sdf3].generation_errs  0
[/dev/sdg3].write_io_errs    0
[/dev/sdg3].read_io_errs     0
[/dev/sdg3].flush_io_errs    0
[/dev/sdg3].corruption_errs  0
[/dev/sdg3].generation_errs  0
[/dev/sdk3].write_io_errs    0
[/dev/sdk3].read_io_errs     0
[/dev/sdk3].flush_io_errs    0
[/dev/sdk3].corruption_errs  0
[/dev/sdk3].generation_errs  0
[/dev/sdj3].write_io_errs    0
[/dev/sdj3].read_io_errs     0
[/dev/sdj3].flush_io_errs    0
[/dev/sdj3].corruption_errs  0
[/dev/sdj3].generation_errs  0
[/dev/sdh3].write_io_errs    0
[/dev/sdh3].read_io_errs     0
[/dev/sdh3].flush_io_errs    0
[/dev/sdh3].corruption_errs  0
[/dev/sdh3].generation_errs  0

That /dev/sdc2 was having issues is not new to me, and although it’s still limping along (SMART data is still showing as okay), I was planning to replace it soon.

What’s new to me is that /dev/sde2 /dev/sdf3 are showing some errors, too. Although those are corruption_errs, so I wonder if they are really from the power failure and not indicators that the drives are damaged.

Yes, I had seen them, and I’m aware of the trap. I usually do have a subvolume mounted in my home dir at boot-up. Took the opportunity to comment all relevant entries in /etc/fstab so nothing gets mounted during next boot.

Mod edit: Consecutive posts merged.

22

The power failure may indeed have caused filesystem corruption, and neither RAID 1 nor RAID 10 can correct those. All they will protect you from is system outage due to a disk failure.

That’s why one needs to maintain a good backup strategy.

23

Even disconnecting is probably better, it’s not like you are short on drives. :smile:

The SMART data is actually okay?

Either way…

Reset the stats and keep an eye on them.

sudo btrfs device stats --reset
24

Oh I have a really good strategy, I just haven’t implemented it … :face_with_diagonal_mouth:[1]

What I’m asking myself, though: Is it plausible than a few corruption errors, on three drives out of ten, could remove around 1 TiB worth of files when every piece of data exists at least twice? That includes a few fairly large ones but also a large part of my photo collection, which has tens of thousands of files. The directories that used to contain those files are all still intact. There are some fairly deeply-nested directory structures with not a single file in them.

  1. but at least I eat healthy, exercise, brush my teeth, use a local password manager and an e-mail-forwarding service to create independent user credentials for all services, I very rarely drink, I drive carefully (though usually avoid it), I (usually) sleep enough, always validate my CFD simulations, don’t fly if I can take a train instead and don’t accept candy from strangers. I even shave more than once per week! That’s got to count for something, no? Do I also have to clean my room, repair my bike, water the flowers and set up my backup? Actually, I’m feeling really sorry for my bike. I meant to fix it this weekend, and now it has to wait. ↩︎