I have suffered a bis data loss from hitting by a flash and had to pay a lot for recovery by Ontrack. Now I want to setup my new computer ready to backup the system and the home / data volume.
When setting up the new 2 TB NVME, I followed the best practice recommendations for the structure - already discussed in this forum. Actually all is well functioning. I am the only user on this PC.
But there are the following questions open:
Is there a special function on btrfs, when I create new subvolumes and name them either with leading @ or without this @?
Question where to put the mounting point for data subvolume?
I have tried to create a subvolume @data. In this volume I want to have all configuration files for my programs and all the data, I am frequently using and creating with my programs. I will create symlinks for the configuration files, linking them into the corresponding place in /home/user. The subvolume “data” should act like a second /home/user subvolume so that I can easily backup this subvolume and when restoring, all my programs will run as before, because of the configuration files. I will call this second /home subvolume “the data-volume” in the following text.
For my backups, I have decided to use restic with backrest UI and restic-browser UI.
I have experimented a while and strange things seem to happen:
I have found out, that if I delete something of what is stored in the data subvolume it will be deletetd immediately without using the trash bin! This seems to happen independent from mount point and independent
from the first type being an @ or a regular typo. Of course this behaviour is very unwanted.
What can I do, that any deleted file or directory will safely go to the
trash bin in the /home/user subvolume at first?
And the last question for my backup planning is:
I want to backup the btrfs snapshots on a external disk drive, These snapshots are created by Timeshift and all are owned by “root”. I have found some info, that the backup of these snapshots can be done by Snapper plus some scripting. – In case of a disc failure, I can hopefully restore the entire system from such a snapshot.
But first, I have not really understood, how this backup by snapper and scripting would work. And second, I know about btrfs snapshots, that
they do not contain the real data, but they contain links and metadata to the files and their predecessors. When backing up such snapshots, of course the real files and folders should be contained in the backup and not only some symlinks or files which contain metadata.
How can I achieve this?
Well, I’m not sure what you mean by “best practice”, because there are many differing opinions on what that might be. There is of course the default layout as installed by calamares, but whether that is “best” is debatable.
Personally, I think the term “best” would be highly subjective, and relative to the level of expertise of the user who installs the system and acts as its administrator. And in my opinionated, um, opinion, the way calamares sets up things by default would be fine for the inexperienced user because it reduces the level of complexity, but at the same time, it also causes almost everyone here on the forum to think that the ability to make snapshots would be the only reason why btrfs exists.
My own btrfs setup for instance doesn’t even take snapshots into account — as the matter of fact, one cannot make any meaningful use of “recovery” snapshots with the way I’ve set things up — and I simply make real backups on a physically separate storage medium.
No, the naming of subvolumes with an “@” as the first character is actually just a convention. It’s quite useful for creating subvolumes right under the top-level subvolume of the filesystem that is to act as the one the operating system itself will be installed on. But for nested subvolumes, it doesn’t really make any sense. You are better off just giving them regular directory names.
For instance, on my system here, I have a separate but nested subvolume for /etc. When it’s not mounted, it’s still readable when the subvolume containing / — i.e. the /@ subvolume under the top-level domain — is mounted, which is necessary at boot, because you need to be able to read /etc/fstab. But my /@ subvolume is mounted read-only, and then — through /etc/fstab — /etc gets explicitly mounted as read/write afterwards.
If it’s a single-user system — meaning that all of the user data belongs to one user and must not be shareable between user accounts — then I would simply mount it to a directory inside your home directory itself. That way, the mountpoint is already owned by you and you have the correct permissions already.
Configuration files are almost always put under ~/.config and ~/.local/share, or at least, for XDG-compliant software. There are however applications which insist on creating their own “dot directory” straight under your home directory. Some of them even have their configuration data in more than one place.
So in that regard, it’s a little like herding cats. My advice therefore would be to not move the configuration data around, but to instead just separate your own documents and files from the rest. And this is what you have the XDG directories for, i.e. ~/Documents, ~/Desktop, ~/Pictures, and so on.
So what you could do is make those into individual subvolumes — remember, you don’t need to put an “@” at the front of their name — or make them into bind-mounts to similarly named directories under a common mountpoint for a Data subvolume. They don’t even need to be bind-mounts, because you can also use symbolic links instead, of course. Both approaches are valid.
I’m afraid I’m not familiar with that software.
That is probably some setting in your graphical user interface, because that’s not how it’s supposed to be.
But then again, there are two ways of deleting something — or at least, when working from a graphical environment.
The first method is the default, corresponding to pressing the Del key on the keyboard, and that’s the method which moves the pertinent file(s) to the bin.
The second method is by using ShiftDel or the corresponding “Delete” entry in a menu, which will effectively delete the file(s).
Again, this depends on the GUI you are using, and how things are set up in there. All I know is that in KDE Plasma, the default “delete” operation moves things to the bin, and you have to explicitly delete them using the other option if you don’t want that. Also, the explicit delete always asks for confirmation (unless one actively disables that).
That’s not entirely correct. A snapshot simply freezes the data in place at a particular point, after which the “regular” files may get updated, moved, deleted, and so on. So at the moment the snapshot is created, it consumes no additional data, but as soon as the “regular” data begins to change, its older blocks are still being kept in use by the snapshot. It’s not just metadata, and they are not symlinks.
Therefore, when copying a snapshot, you are effectively copying over the real data, but only the data in the snapshot, not the varying data in your active subvolume.
My advice therefore would be as follows…
Make snapshots only as rollback points — which is pretty much what they were intended for — so that you can immediately roll back your system to a previous point in time if a calamity were to occur.
Make physical backups of your data — not of your snapshots — on a separate storage medium. This separate storage medium should be another drive — whether internal or externally connected — or a NAS.
Ohhhh thank you very much for your detailed answer!!! It is so much more, than I did expect from a forum here. I have an appointment now and will read and hopefully understand your answer til tomorrow.
Only one thing to answer:
Concerning my btrsf layout I have set it up this way:
space filesystem mountpoint Label
8 MB unformatted ---
500 MB FAT32 boot /boot/efi
32 GB LinuxSwap linuxswap swap (1 x RAM)
all GB BTRFS / root
8 MB unformatted ---
Ah, but there is a lot more to the forum than meets the eye.
Part of the development and Q&A testing by the community is handled under the Manjaro Development section of the forum.
Whenever there are bundled updates, a dedicated announcement thread will be posted about it under the Stable Updates or Testing Updates categories. The first post of these threads contains the most important changes (with a link to the full Changelog), and the second post contains the known problems and how to work around them. It pays to subscribe to notifications for these announcements.
We have an extensive member-submitted Tutorials category.
The unformatted 8-MiB partitions are not necessary if your system boots in native UEFI mode. You only need one — and it shouldn’t be larger than 3 MiB — if your system boots in legacy BIOS mode in combination with a GPT partition layout. (It is not required for legacy BIOS boot in combination with an MBR partition layout.)
The size of the swap partition depends on how much RAM you have, in combination with whether you plan on hibernating the system — i.e. suspend-to-disk. In that case, you need it to be the capacity of your RAM plus about 50% more for the data that’s already on the swap device when you hibernate the system.
If you do not plan on hibernating the system and you have 32 GiB of RAM, then you can easily go without a swap device at all.
I have 16 GiB of RAM in my system and no swap device, and I’ve been running my system just fine like this for well over five years now. And I do have quite a lot of stuff loaded, by the way.
But of course, I don’t hibernate my system. It’s a desktop computer, and UNIX systems carry out a lot of maintenance tasks behind the scenes when the system is idle — which would otherwise all need to be done when bringing the system out of hibernation.
Ah yes, it’s a sred ritten by @andreas85, und he is a native German spekker. Zet is vy he posted it zere.
This sounds like this may go off in many directions. Forgetting about stuff like undeleting files from your desktop environment’s trash. That might want to be another post.
What you are doing seems to veer from from best practice, but there’s not enough to go on to be fair. I come from a career of doing things like this, mainly servers. But in many cases desktops, and they can be drastically different, based on crazy use cases.
Layout
The most common and basic layouts for a desktop PC is to either have it all on root/@, or with two volumes: root and home/@home.
When dealing with more complex applications, there can be many reasons to have different volume(s) for it’s data. An application that uses it’s own packaged database, would be a quintessential example for this. How does this application(s) work?
But with the two volume root/home layout. With it’s simplicity, it gives you this.. While you do your system upgrades and package updating, for the most part, is all on it’s own volume with it’s own separate snapshots. It makes sense to snapshot that volume around when you are making updates. If needed, rollback is easy, and you don’t have to touch home.
With home, you most likely want a completely different schedule, retention, and who knows what else. This also uncouples user configuration to the rest of the system.
Backup
I think you’re just misunderstanding how CoW file systems work. It’s not even unique to btrfs. I’ve been doing this with ZFS for almost 25 years.
To try to simplify how these snapshots work. All new data is written to a new place, while the old data is perfectly intact. Creating are using snapshots are near instant, but deleting them is more I/O intensive. Generally (Timeshift excluded), these are read only snapshots, so you can’t modify them. This does not mean they are impervious to failures, hardware or otherwise. Since what is modified, happens at the block level, these snapshots do depend on each other. Snapshots are rock solid, but only if hardware does not mess up, or software bug out.
What is unique compared to most file systems is that btrfs will know if it has an error with the data integrity at all. Though you cannot repair it without another copy of the data.
But you can build in the redundancy into the right into btrfs filesystem with another device (mirroring/RAID 1), and it will usually just repair itself.
Fun fact: You can even take your root btrfs file system, convert it to RAID 1 or another RAID level, LIVE (/aka while booting and using it)!!
Snapper, Timeshift, and just like btrfs snapshots on the command line; all are not backup utilities. They can be rollback points, but the snapshots themselves are what you backup. This is how you can backup a live root filesystem or database.
This is probably too much, too fast.
(Timeshift is an exception to this. There is a way so it will backup, using rsync, to another drive essentially. It’s very easy, but there’s little flexibility, and you lose many of advantages of using btrfs in the first place.)
There’s so many directions I could go, I’ll stop and get some input first.
The 8MB parted leaves still makes me use freaking fdisk to this day!
Just kidding, it’s perfect. Order can be whatever, but EFI first is always safe.
If more than one device (2, 3, …) is present and integrated into btrfs, data won’t be lost if one of the devices fails completely (or partially). As long as there are “only” write errors on one of the devices, btrfs will automatically repair them.
It’s recommended to use devices from different manufacturers to avoid what happened years ago with IBM hard drives. (They all failed sooner or later. RAID was useless because all drives had the same problem.)
I can present to you how I am doing things here at my end, but I am in no way suggesting that you — or for that matter, anyone else — would do the same. Also, bear in mind that…
I don’t have a swap partition, and therefore I do not hibernate my system — or for that matter, suspend-to-RAM.
I don’t use the snapshotting functionality, because that would not yield any bootable rollback system due to the way I’ve set things up.
I have a separate partition for /boot. This is because I initially had a setup with many more separate partitions — I think I had 11 of them — and because /boot used to be formatted as ext4 due to the fact that the grub boot loader does not fully support btrfs — it can read from it (under most circumstances) but it cannot write to it.
The following filesystems/subvolumes are all mounted read-only during normal system operation.
/boot/efi
/boot
/
/usr
/usr/local
/opt
/srv
/srv/ftp (if required at some point in the future, I can permanently change this to r/w)
/srv/http(if required at some point in the future, I can permanently change this to r/w)
/srv/mmedia/music
/srv/mmedia/video
I make full rsync backups — with the exception of /var/cache, /var/log, /var/spool and /var/tmp — by way of timeshift. This includes /home, but I also manually make a separate backup copy of that at the end of every day.
All backups are stored on /dev/sdb1, which is not mounted during normal system operation, and which is on a rotating SATA2-connected 750 GB HDD — /dev/sda in turn is an SATA3-connected 1TB SSD.
rsync backups are browsable directories. This means that I can restore individual files and/or directories from the backups by hand without needing to restore the whole backup.
Of course, this is not for the faint of heart, because it requires getting your hands dirty and using the command line. But I have already needed to do this in the past — when I still had the setup with the multiple distinct partitions — when a stupid typo at the command prompt caused the permissions on /etc to be all wrong, and I’ve also restored my home directory that way a couple of times during the Plasma 5.25 fiasco.
In your case however, given that you are seemingly new to Manjaro, I would recommend going with the recommended defaults of…
@ for /;
@cache for /var/cache;
@log for /var/log; and
@home for /home
This way, you can make bootable snapshots of your root subvolume and restore them while preserving your home directory and your system logs — which may help you diagnose the problem that caused you to require booting into a snapshot anyway.
You can do this by way of snapper or timeshift, but I would recommend using snapper for this — @andreas85 should be able to help you with that — and to instead set up timeshift so that it creates full rsync backups on a separate storage device, so that you will always have physical backups of your personal data in case your main drive fails.
By default, timeshift does not come configured for backing up /home, although it’s trivial to tell it to back that up as well. There are however yet other backup solutions which are specifically tailored for backing up only the home directories, such as deja-dup, which is a gtk front-end to duplicity.
It deserves to be noted that this is mostly pertinent to spinning HDDs and less so to SSDs, but yes, it is good advice nevertheless.
And do note that it wasn’t just IBM drives — specifically, the DeskStar model, which henceforth became known as the Death Star — that failed. It is indeed always best to pick drives from different manufacturers and/or a different model, because mass production does indeed increase the failure rate of devices manufactured as part of the same batch.
Oh dear! That’s quite a lot and I hope, I have fully understood all the precious responses. Thank you so much for this.
My experiences with Manjaro-Linux on brfs and Timeshift snapshots let me see the big value which is based on the btrfs-timeshift snapshots. I know, they are different from those, Timeshift would generate when using rsync option. With rsync option I would loose many features in contrast to btrfs timeshift snapshots. I have read this, have understood the details and then I have forgotten the details and kept the conclusion. That’s my way, to get around with the complexity of life today!
There were several occasions, when I could get back to a smoothly running system, using these btrfs timeshift snapshots and I don’t want to miss them.
Actually a friend of mine came with a laptop, which could not boot from any of these btrfs timeshift snapshots and I could not find the cause. This is a typical scenario, where I would be happy to have a complete and up to date system backup from an external disk available.
As far as I have understood the recommendations here, Timeshift could do such a backup, if it would do it in the rsync mode. – But then I would have to give up btrfs timeshift snapshots, as Timeshift only can generally be set either in btrfs mode or in rsync mode.
Other backup programs like borg or restic are difficult to setup, so that they would be able to access root data. (I had tried this and run into this problem.)
My problems with the trash bin seem to arise from some other circumstances and are not related to my questions about subvolumes, their names and their mount point. In order to avoid time consuming and complex actions, I decided to put all my data in a top level directory, because my backup software Restic can be configured to create backups fro a directory and all what is in it. This decision circumvents the trash bin theme.
By continuing my web search, I just found this discussion in the Restic forum (Sorry that I cannot post the URL to this thread. But a quick search for “Restic Lack of Documentation on how to do a full system back up” will do.)
It seems, that I have to run Restic with sudo, so that it has access to all files and directories of the entire system. Further I will have to exclude some directories from being backuped. And then I can backup to any external storage device. For restoring the system, Restic again must run with sudo privileges.
Perhaps this info about backing up the entire system could be of interest for many others here.
In the discussion above, there was a concern, that doing an system backup while the system is running, could lead to inconsistencies, as some files could change while they are accessed at the same time by the backup system.
Yes, I am aware of this problem and I would run the backup only, when all programs are closed – at least those which are using a database, like webbrowsers, email-clients, address books etc. If you want to avoid inconsistencies, the backup should be taken as an image from a system, which is not in operating mode like Clonezilla or similar solutions.
But if inconsistencies would be such a big problem, why are there millions of mostly professional users, using solutions like Veeam (and others), which do backups from a running system?
Yes, but you can switch it back and forth in the settings. Of course, this is not ideal, but there are other solutions you could try.
For instance, snapper can periodically create btrfs snapshots, and I believe it can even be triggered to do that before or after a system update. And you could then use timeshift or something like backintime to create rsync backups.
This way, you can have your cake and eat it.
Note: You will probably want to install grub-btrfs instead of the regular grub, so as to be able to boot into your older snapshots.
It depends on the filesystem type, and whether it allows for freezing the I/O at the moment a snapshot or backup is made. btrfs will do this whenever a snapshot is made.
xfs can do it too, but I don’t know whether this was implemented in the Linux port — the IRIX version certainly can, but not all of its functionality was ported to Linux.
Yes, that is possible. I use snap-pac for this. There are some notes on useful packages for this type of thing listed here: Snapper - ArchWiki
Additionally, for those who really want a GUI, there is btrfs-assistant - it can be useful, although it can take a little time to understand its approach and limitations.
It’s great to see that you see how inferior that solution is. But it does have simplicity going for it, since it is basically simple checkbox.
There was a Timeshift bug with booting them via grub-btrfs and restoring them. I have not been able to recreate the issue.
I think Timeshift is a great app to start using btrfs and snapshots. This can be used along side with whatever backup you provide.
Yes that’s always a concern. So you create the snapshot, then back that up.
It’s a point in time snapshot. When you restore a snapshot from a backup, then boot it. It’s like the computer was just reset (at the moment the snapshot was taken.) So your system usually comes up fine, exactly like an unclean shutdown.
The snapshot gets around this. For added reliability, many services like databases lets you run a quiesce command. (To mainly flush out mid-transactions and pause new ones.)
Veeam and just about everything like that for Windows uses Volume Shadow Copy. Let’s just say snapshots for Windows, with a big asterisks.
I am having a hard time which direction you want to go, or are set on going. But we’re also troubleshooting?
I believe you are just missing a few things to bridge this gap. If you are wanting an advanced backup, you will probably still want to use that. Timeshift, Snapper, etc. have nothing to do with that.
On the computer with Timeshift, and your snapshots, or is there one now? Have you poked around the top level volume? Not root, top level.
sudo mount /dev/sda3 -o subvolid=5 /mnt
(Since Timeshift makes RW snapshots, take care on what you do. But here is where you would be backing up the snapshots,)
… whereby “x” is the correct alphabetic character and “N” is the correct number.
If it’s an NVMe drive, then the designation will of course be different. I can’t give a concrete example right now because my SSD is SATA3-connected.
I guess my answer will be a bit off topic but as I had in the past also some catastrophic SSD failure, I adopted a different but simple backup approach.
I keep user data (documents, images, 3D files) on a NAS with RAID Mirroring.
After an update of Manjaro, I create a backup of the complete PC with Clonezilla on one of two external disks (rotating).
This is an easy solution not requiring much knowledge but only works as long as I keep doing it regularly. It also requires some additional hardware (NAS, external HDDs).
I also backup the NAS sometimes on to the HDDs.
Keeping the HDDs in a little carrying case in a safe place makes it easy to grab them an do a quick backup. I had one occasion where I had to restore the system and that worked very simple with Clonezilla.
It’s a different approach but maybe it is useful also for others.
As much as Clonezilla has matured over the decades, it’s still just fancy imaging software. Sometimes it is the most efficient way to go. I still use imaging immensely for my ARM computers, appliances, and what not. But for backup purposes, it’s more like a portable factory reset. It’s usually awkward and inefficient to do regularly for small changes.
Having NAS and/or external drives are invaluable for cost effective in house backup solutions. (And external drives can quickly become off site backups.)
You’ve already said you do not do this, but generally speaking and for everyone else, people use often use their external drives and NAS appliances as just a place to store files. Maybe his or her N:\ drive in the Microsoft universe. For data loss, this only accounts for most hardware failures. Software can potentially have severe bugs causing data loss, malicious software is out there (even in our AURs! ), of course accidental user error can never happen , or literally anything else! So if the data is important, there needs to be another layer, to consider it just a basic backup.
The sometimes is making it a sometimes backup. But that can be perfectly fine, it’s what ever you need.
I like to have them on a timer. For example, it’s fairly simple to have systemd keep trying to run a weekly script, until it succeeds. This could include: until you plug your external drive in; or you can get fancy, and verify your NAS appliance is running in non-degraded mode before you attempt to automatically backup to it.
Just to go on selling ButterFS! (For backup purposes. This includes using Timeshift, Snapper, or anything else including you just typing: sudo btrfs subvolume snapshot ..)
It seems snapshots and backups often get confused. In the event of disaster recovery, restoring snapshots (or specific files from snapshots), will account for most software issues. Whether it’s your root volume, or even my consumer grade Synology box supports snapshots (via btrfs!)
If you can snapshot, there is minimal overhead in creating them and using them. Snapper even has an hourly option, it has been invaluable to me for work. This is basically what I do with my root and home file system, basically.
To take btrfs even further… This may be considered more advanced, but I love the fact that you can send only the changes between snapshots on two separate btrfs file systems.
No need for convoluted backup software to figure out what’s changed. If a tiny part of a large file changed, it only sends that small part. Most alternatives, especially in the open source world, would send the whole file. And things like rsync have to read every single file’s metadata (and/or all of it’s contents) just to even detect a change.
And it doesn’t even have to be that complicated (though I would argue that it’s not complicated). Even plain ol’ cp or tar can get the job done on a live root file system (but only if you are using it on the snapshot).
Oh noes, I have new pacnews! Guess we imagin’ again!
That’s my way, to get around with the complexity of life today!
, or literally anything else! So if the data is important, there needs to be another layer, to consider it just a basic backup.