It depends on whether the missing data was striped, and where the two halves of the stripes were stored.
RAID 10 is a combination of mirroring and striping, so while you do have two copies of everything, each copy is also split over two drives, and if you’re unlucky, then the corruption is on the drives holding the same halves, which invalidates all copies of the other halves.
Unfortunately it’s one of the larger drives, so it would cost me some amount of storage space. However, I need a new drive to restore the lost files to anyway, so I can replace this one afterwards.
I just checked again: raw read error rate and sector reallocation count are in the “pre-failure” range, but the overall SMART metrics still pass, also the short self test.
Thanks for the hint. I just did that – let’s see what happens
Yes, I understand that. For every file lost, both corresponding copies of at least one of its chunks have had to be lost. The striping distributes components of larger files potentially evenly across all drives, so that any two drives failing could affect a lot of files, even if they’re small drives.
Of course, none of my drives failed completely – I just had one producing a lot of errors, and two others a much smaller number.
Assuming that the one drive really lost chunks from most of the files on the system, then I’d still only lose anything of a copy of a lost chunks is affected on one of the other drives. Assuming even distribution of data, only 1/9 of the chunks damaged on the first drive are affected. So it would still seem really unlikely that a third of the files (thousands, probably tens of thousands, including a lot of small sidecar files in the photo collection) would lose both copies of one of their chunks this way, unless a huge proportion of my files would have been in an unsafe state at exactly the moment when the power went down.
That doesn’t mean it could not have happened. It just means I think that there’s likely something else that went wrong.
Mod edit: Consecutive posts merged.
Well, you’ve got filesystem damage due to a power failure, and you’ve got drives in their pre-fail stage. So, if you want to calculate the odds, be my guest, but from where I’m sitting, it’s no mystery. 
I have very little experience with btrfs - and the little I have - was so discouraging that I avoid btrfs.
But you asked the question
I have been using RAID constructs of various kinds over the years - and there is only one that has worked stable for me.
The linux kernel implementation of raid 10.
Using RAID is usually because you want to ensure your data survives in the case of a single disk failure - very few raid constructs can survive the simultanous failure of multiple disks.
Rule number one - always mix disks different vendors, and different batches, different sizes are OK - just keep in mind the smallest disk decides how much space is used on a single member.
If you run a RAID using 5 disks from the same vendor and batch - they are likely to fail at the same time or close to each other.
Do not - ever - use all disks for data - depending on the size of your array - keep 2 or 3 devices as spares - always add a single device as hot-spare.
You can use btrfs as your file system on the RAID no problem - or you can use xfs which is tailored for use with RAID systems.
The md/raid 10,far2 construct uses an even number of disks and a 2 disk setup provides the same redundancy as a standard RAID 10 with 4/8/16 disks and access time on par with RAID0.
In such a case, there is only one sensible decision:
The affected drives must be removed from the file system. Anything else is playing with fire.
Combining several unreliable drives into a RAID does not result in a reliable RAID. The idea behind RAID is that no damage occurs if a drive fails “unexpectedly.” (And even then, there have been situations where a RAID could fail because one drive fails, and while it’s being replaced, a second one fails (due to the stress).)
If you must know, I’ve already ordered a replacement.
The main issue right now, though: How do I get back whatever data can be recovered from those drives?
Okay, we’re going properly off-topic now, but let’s just go there: Why do I have a Frankensteinian 10-disk array?
The reason I’m using a btrfs RAID and not mdraid [1] is precisely because it allows me to make reasonable use of heterogeneous bunches of drives. If one of them dies, I either replace it with a bigger one or I don’t, but I’m never in a situation where I have to replace the whole bunch or some majority. Most of those disks are leftovers from other computers and would have gone to the trash if I had not found this place for them.
Having that many small, old drives in a computer is no fun because I’ll never remember what I stored on which drive. It’s also really asking for trouble because sooner or later one will fail. That risk is much lower with the RAID 10 setup. For the last 4 years, that has run stably, with two drives having failed and been replaced since I set the thing up. Definitely safer than a single non-Raid drive.
… of course, until that power failure. I’m still very confused about how that could have removed 1/3 of all the data. I don’t think that all those files just happened to have both copies of one of their chunks open for writing/balancing and in limbo at exactly that moment.
Mod edit: Consecutive posts merged.
If there are no intervening replies, please edit your previous post instead to add further information. Cheers! 
as I have for some time before – I bought those 4 640GiB disks as a set, in 2010 ↩︎
I would suggest trying the wrapper:
It will generate the regular expression for you. You just need to write the path. Then it will search each root tree one level deeper.
curl "https://raw.githubusercontent.com/danthem/undelete-btrfs/refs/heads/master/undelete.sh" > btrfs-undelete.sh
sudo bash btrfs-undelete.sh /dev/sdX /external/restore/path/
It is of the utmost importance that you do not alter the current state of your array.
Every time you try to fix it - you loose data.
Change the fstab to mount the array readonly.
Wait until you get new drives - create a new setup and salvage what can be salvaged.
20 years ago I had a 4 disk NAS running a controller based RAID5 with one hotspare.
Because a thunderstorm was coming in - and at that time my power protection would cut power at the slightest spike - I chose to power the NAS down.
Next morning when I powered the NAS - file system was completely gone - system unbootable.
I had everything there, my code, my documents… everything was gone.
I eventually had to cough up somewhere around DKK65000 for a professional data recovery.
I have also tried variations - one disk failed, and while I was replacing the disk, another failed and the whole raid was dead - it happens - especially for old disks.
And that is possibly why you experience a multiple disk failure.
I fully understand the flexibility of btrfs in that regard, but I also see the dangers if data is valuable.
The danger is because you apply the redundancy on a file system level instead of using disk redundancy - as I see it - btrfs RAID is a bunch of disks where you rely on the file system making sure copies exist somewhere on somedisk - but you actually don’t know what is where and why.
While btrfs has gained a lot of traction the past 5 years - the file system requires a lot of supervision and maintenance.
If you under stand the basics Copy on Write filesystems, you should understand how crucial metadata is, and how it works to piece your data together.
The data doesn’t necessarily physically disappear from the disks. But btrfs may no longer have a reliable way to determine which extents belong to which files, which versions of data are current, or even where the relevant filesystem structures are located.
With CoW, a file may be made up of extents scattered all over the filesystem, with multiple generations of those extents potentially still existing. The metadata is what ties all of that together.
Lose enough of the map to your data, and recovering the data becomes a forensic exercise (as already mentioned).
But all I see are guesses. (With bad hardware in the mix.) It still doesn’t add up to me.
I saw you go through some of the btrfs-progs ways of attempting of recovery.
I know there are pieces at least not mentioned here…
Which may not even be viable routes, as you couldn’t even btrfs-find-root some before.
I’ve only brushed up against bad metadata a handful of times myself. But I would explore all the btrfs-tools methods before resorting to third party.
Can you do this on all the drives?
sudo btrfs inspect-internal dump-super /dev/sdX
Not even sure if it will help. But I would be confirming things like generations, IDs, sizes, and who knows… A possible next step.
This is correct. One can even have a RAID 1 or RAID 10 with btrfs on a single disk, provided it has enough partitions.
This strategy — which is similar to how zfs does it on Solaris — is very convenient because it does not require the administrator to worry too much about the physical organization of the array, but at the same time, it loses the reliability and fault tolerance of a real RAID 1 or RAID 10.
It’s also mostly being used for one thing only, i.e. the snapshotting functionality. Very few people use it for anything else.
And snapshots are not backups. Most people seem to believe that they are. 
Correct as well.
This is a long post, and it touches on a lot of topics – but my understanding is that this is preferred to breaking it up by topic, so I’m keeping it in one piece:
I’m pretty sure I mentioned in one of the first posts that I removed the fstab entry. There were multiple posts about how to mount the filesystem or not. So … yes, duh.
So, this is one of the stories I don’t get, at all. Because it appears to imply that RAID arrays were somehow more vulnerable (to what? being turned off?) than regular old single hard drives. I actually had that experience with a single drive, too, but that’s beside the point. Stuff can go wrong and if you operate enough RAID arrays, it will. At my previous employer, I was in charge of setting up and managing around 5 different NAS devices, for about 5 years. When something broke, I usually knew what went wrong. Of course, in that context, I made sure that everything existed twice. The worst that happened was a plain old drive failure followed by a replacement. Nonetheless, I’m sure that something would have broken in an unrecoverable way sooner or later. It always does. I’m not arguing that nothing was broken on my end now. I’m trying to figure out how it happened and whether I can get some of the data back.
I haven’t! If multiple drives had failed, the scrub would have mentioned something about it, no?
It actually ensures that the four (or six, or eight…) chunks that belong together are each on a different disk. How would it help me if I knew which disk each chunk was on? How could that knowledge have prevented anything?
Now, there is one difference which is clear: data is distributed in “whichever” fashion, then taking out two disks from a btrfs RAID10 array means that for some number of chunks, both copies will be unrecoverable, while in an mdraid device, it means that either everything is fine or everything is gone. For restoring data, I’m not sure whether an mdraid system would have fared better: If one specific disk was broken, I could take it out of the equation, in either scenario. If I two were broken, and I knew their relation to each other (are they mirrors of each other or not?), that could tell me quickly whether or not I can recover anything without them – but I don’t see how the recovery itself would be more or less likely as a consequence. It’s kind of a Heisenbergian: If an mdraid would have had a 1/x chance of being a lost case, then a btrfs raid could expect to lose 1/xth of the data. …except:
Do you understand the basics of Metadata,RAID1C4?
If you do, then you know that every lost bit of metadata would have required that bit of metadata to be damaged in all four copies, which are on four different disks. How likely is that if sudo btrfs dev stat shows that only two of the disks have had any errors at all? You do the math.
Also, I have still lost no disk – I only have a lot of errors on one of them, which I still find something of a mystery.
… that seems like a really bad idea. If the drive dies, the data is dead. The only thing which single-disk RAID 1 would insure against are read/write failures from/to a bad sector because the same data is mirrored elsewhere on the same drive. okay, not horrible, but sounds like a waste of hard drive space.
… but anyway. I have no idea what you are all trying to convince me of, from “Losing data with btrfs is an art form” to “If you’re not using mdraid with hot spares you’re asking for trouble” (own paraphrasing…) so let’s just try and get back to the topic, shall we?
I managed to get two hard drives for the recovery, a 3TiB one, and a 2TiB one.
I then ran the script recommended by @megavolt, which tries going through all file system roots it can find and run btrfs restore from there. This took “a while” (around 24h), and afterwards the file dates seemed off, so I modified it to include all possible file attributes, symlinks etc. in the recovery.
Outcome: The recovered data volume is pretty much equal to what is already readable on the volume. Not very happy news.
And all files with zero size are missing. That is: zero-size files which are present in the read-only copy! Lock files, Python __init__.py files and more similar things. So there’s something seems to be wrong with btrfs restore. Or at least I could not find any argument to it that would cause it to skip/inlude such files. Or maybe it’s something in the undelete script? My bash-fu is limited so I haven’t spent the time needed to have any confidence about the reason.
In the meantime, I’ve also taken stock of what exactly is missing, and what parts of it I have backup data for. Turns out that I do have backups of most of it, just scattered in a somewhat unstructured manner[1] – but that’s not something I need advice with.
Soo, looking through the file system and comparing with some of the partial backups I have lying around, I can tell that more than fifty thousand files are missing but all directories are there. So the directory tree is fully intact, but the inside of some directories looks like the intro to a zombie movie. Weird.
Many of those files are photos which take up several megabytes apiece. So what is the chance that both copies of all of those files were momentarily in limbo when the power went off? I can’t imagine just how fast and loose BTRFS would have to play to create that kind of state. This is exactly what CoW is supposed to protect us from! The impossibility of implementing protection against exactly this is why RAID 5 is permanently labelled unsafe in BTRFS (I can confirm: When I did try RAID 5, every crash caused some files to be lost – orders of magnitude fewer than I lost this time, though).
Oh hey, since I paid attention to have all file dates preserved during recovery, I might just have a look, why don’t I?
…and I noticed something: The modification date of all the emptied directories is not the 16th of July (when the power went down) but about a month earlier.
Doing a search for all entries in the file system with that date shows that there must have been a process which modified those directories, for four seconds, and then stopped. Sorting them by time, the process does seems to have traversed the tree almost but not quite in an “ordered” fashion. So in some places, modification on some (sub-)-tree is interrupted by another subtree, some sub-trees are processed in alphanumeric order, but some parts are not. After about four seconds, it stops, leaving some (sub-) directories unchanged. There are some file changes directly beforehand which I do recognize, but none after.
So, this means that the power failure is very likely not at all responsible for the issue, although the scrub-and-balance afterwards very likely is the reason why recovery won’t find much of value. It’s somewhat unexpected but definitely possible that this kind of data loss went unnoticed for three weeks, as I was not using the PC much in those weeks, and mainly for half-hour long gaming sessions before bedtime. I only did spot it when I finally had some “quality” time again.
Okay, some knowledge gained… even if I don’t like it because this makes it even harder to figure out what even went on there.
If I had wanted to deliberately achieve the observed effect of clearing out a lot of directories, I’d have had to create some regex to match all files but not the containing directories, and hand that to rm. I’m pretty sure that I did nothing of that sorts. I’m not a fan of regular expressions in general, so it’s definitely not the kind of thing which I would do, especially not by accident.
… so I just did a search on the user home directory, for the same date, and looked at which files were changed around the same time.
It turns out I was trying (unsuccessfuly!) to set up my e-mail accounts in kmail. It offered to import my mail from claws and thunderbird. However, the import dialogue was … really confusing, at least for my case where one account has multiple aliases. Could it be that me entering a “folder name” misinterpreted the prompt and gave it a heading for the structure shown within kmail, while the importer interpreted it as the name of a directory on disk which it should empty out and use to store my e-mail? Now, I can already hear all of you screaming “yes!” behind your screens, but before you actually respond by telling me how I messed everything up, I’d really prefer if you could consider whether kmail is actually capable of doing that sort of thing.
Kmail is arguably not linked to btrfs but this thread is (also arguably…) about btrfs:
/dev/sdc2. After having some healthy amount of sleep and another look, it turns out that the SMART data is actually completely fine. The fact that the number of read/write errors has not changed during the rescue attempts contains unfortunately no information because it was mounted read-only, and changing the number requires it to be written. Nonetheless, the accumulation of errors likely has a different reason than the hard drive being faulty. I can think of two:the usual quick safety-copy-before-doing-something-that-could-be-dangerous which Yes I know should be an automated versioned off-site backup I know how Borg works so get off my back already Seriously, some of you mentioned it twice although I already said it in my initial post. … breathe .. okay. ↩︎
I’d like to think I understand a bit. I knew nothing about btrfs 5 years ago, so I’m always learning.
Like most modern filesystems, btrfs uses transactional commits. The point of a transaction is that, after recovery, the filesystem should be able to use either the old committed state or the new committed state, rather than some half-committed state.
I think we’re on the same page up to here.
But the actual physical writes involved in a commit are not magically atomic. It is not like a transactional database.
The data across four copies is not necessarily written simultaneously. More copies should reduce the chances, but that chance is not gone. More copies are generally good, but the vast range of different devices in the pool also introduces more variation in how writes are cached, acknowledged, and ultimately made persistent. That even increases the ever-so-slight chance of inconsistent physical states.
The drives all have independent internal cache, and consumer-grade drives can have very different behaviours. So an acknowledged write is not necessarily the same thing as data that has actually been written to the physical storage medium.
With enterprise storage this inconsistency is drastically reduced. One of the many reasons is that many enterprise SSDs come with power loss protection (PLP). They basically come with capacitors to hold enough power to flush out it’s volatile cache.
And this is not unique to btrfs. Any filesystem/RAID combination has to deal with the fundamental problem of coordinating updates across multiple persistent storage devices during an unexpected power loss.
You can easily search this whole topic in any direction you like.
That is a fair point, and it even made me say things don’t add up.
Btrfs adds a lot of reliability into this equation, but corruption can still happen when I/O errors aren’t logged. Especially on a power loss.
Checksums, journaling, copy-on-write, and redundancy can detect and recover from many forms of corruption, but they cannot guarantee recovery from every possible failure.
These scenarios are rare, but they do happen. Maybe the checksum mechanism works as intended by rejecting broken blocks, but if you are just left with orphaned generations, btrfs simply has no way to self-repair.
The lesson here for me is a refresher of the lessons: #1 Raid is not backup. IF you have a RAID you still need a backup plan and procedure, and you CANNOT say it succeeds until you do test restore and see that is really does succeed. #2 All hardware fails. Drives can fail (Perhaps not the case here, but this case reminded me of that), all drive CONTROLLERS fail, and all FILE SYSTEMS can fail, and often without warning. #3 a fire or flood in the room with the equipment trumps ALL protections other than offsite backups and redundant locations (redundant locations for a home user may be seriously overkill, but a DR plan and offsite copies are NEVER out of scope if the data has value). #4 Murphy’s law. Things WILL fail, at the WORST time, when you are LEAST prepared. And now I have to go back up my primary laptop, because it has been a full month and I feel stupid for not doing it weekly.
I noticed that you are using several Western Digital drives and I remember some articles that mention issues of the firmware, where it can get corrupted.
Unfortunately, I don’t remember the details and don’t have time right now to check whether your drives could be affected or look up the details.
Anyways, maybe the issue in this article could be the cause?
Let me write #5 myself: All users make mistakes. A significant portion of data losses I know about are honest mistakes. I know somebody who put a new hard drive in a PC, and then set up a new partition table … on the wrong drive. This deleted the only copy of some data for their PhD project, which they had just worked half a year to produce. They found a company which was able to restore it before anyone important noticed 
If you want to make your life easier with backups: use borg. Or if you’re like me and don’t want to memorize or look up all the requisite shell commands: use Vorta (which is a GUI front-end to Borg). It works best with Borg-specific backup servers, but is still enormously convenient with network drives, USB disks … whatever. It’s automated, versioned, incremental, de-duplicating and compressed, automatically prunes older backups, and it can encrypt backups in a very secure way, so you can even store them on remote servers whose owners you don’t trust to keep their noses out of your data. It even verifies backups regularly, i.e. tests if it could restore them if needed. No more forgetting backups, no more guessing which old backups you can overwrite or anything like that.
As mentioned before: I am actually paying for a Borg server. I’ve got it nicely configured and working for all userspace data on one of my machines. On the one which has the current data loss, I had simply not taken the time to make up my mind about some details and actually set it up. (see #5 above) Turns out that humans also have limited bandwidth, and I’m trying to use my computers more than I spend time dealing with maintenance.
In general, I think the trick to not having to spend too much time fixing/maintaining stuff is to have it all set up nicely so what can be automated is automated, doing the safe thing is also the most straightforward one, and important issues are brought to my attention. With btrfs RAID on Manjaro, It’d be nice to have some more convenience in that regard, though I won’t claim that it was “easy” to configure it on the distro side so that it works smoothly out of the box for everyone (knowing that “everyone” includes a huge range of different scenarios). On the other hand, I had to set up automated scrubbing and balancing by creating custom cron jobs, and only noticed that I should after having read a fair amount of stuff about it. It’d be nice if some default scrubbing interval was put in place for every btrfs RAID, and the user given a pointer that (and how) they can change/remove it if they like.
I’m now giving btrfs-desktop-notification (from AUR) a try, which is supposed to notify me when any volume has switched to read-only due to failures, or if btrfs-related warnings appear in dmesg – that seems like a decent step forward. There’s also btrfs-assistant in the Manjaro repo, which could make it quicker to set up regular snapshots and scheduling scrubs, balances and defragmentation without setting it up manually, though I’m not sure how useful it is to me since I did set those things up already, and I find the GUI somewhat ambiguous about what a button is actually going to do or not. Have only looked at it for three minutes so far, though, so my view needs an update eventually. I’m way too busy with other stuff right now, though. Like puzzling whatever data I can back together from a bunch of disparate partial back-ups. And setting up that Borg backup.
Good catch. I don’t have SMR drives, though. Most of mine are too old or small to be SMR drives , and for the less-old ones, I deliberately avoided it. SMR drives always have to re-write a full sector if they write anything, due to the overlapping magnetic regions. That’s is fine if you dump large amounts of data sequentially but bad for small random writes – I’m not sure if that issue is still as relevant for later generations of SMR drives, but I still don’t consider them suitable for desktop PCs.
Summary:
The undelete-script recommended by @megavolt helped me get a few things back. Not many, though.
I also had to run it twice because it ignores files with size of zero. As the author explained on github, that is intentional because some btrfs errors will remove the content of a file, and in that case, it’s smart to restore an earlier version instead of taking the zero-size one. My solution was to run the script twice, the second time after commenting the respective lines out. The second run went much faster, and filled in all the files that are only available with size of zero. Afterwards, meld confirms that the only difference between the read-only-mounted file system and the restored data are files that were lost and restored.
I’ve stopped trying to figure out what on earth caused the data loss because I have no time to play any more Sherlock Holmes now. Suffice it to say that I don’t think it was the power loss after all. I have not ben able to reset the error count on that /dev/sdc yet because it’s only been mounted read-only, so that is definitely one thing I’ll be watching. Since I still want to add another drive to the array, I think I’ll also experiment with connecting it to a different controller port in case there are further errors.
Order of business now:
Do a full balance of the file system while I have an exact copy on another disk, and some significant unused space – I think btrfs changed the way it interprets the RAID10 profile since I set it up, and that may be a reason why there is some data in RAID10/2 profile but the system doesn’t seem to want to use any more of it.
Configure that Vorta backup already
Do another full check-up of the file system and the physical drives. If any piece of hardware is broken, I’d like to know.
add another drive, or swap one if needed
Recover whatever is possible from whatever scattered backups I do have. This will take a while
… I think that’s about it for this thread, although there may be something interesting in a few weeks. We’ll see.