BuildIso vs. BTRFS System Backup/Restore

[limotux]

I am settled and enjoying my Manjaro.
I was just thinking and reading about system backup and restore. It is clear the best is BTRFS, Timeshift or Snapper.

But as I experienced before BTRFS (especially for /home) has problems (Baloo keeps reindexing all files thus makes the system unusable) plus I read some issues with BTRFS. I am on KDE and Baloo is essential for me)

So, I am considering one of two options in case system breaks:
1- Reinstall with BTRFS System partition and EXT4 /home or user data
2- Create ISO, (software, configuration, /home… etc) to install from in case needed.

For sure the first option is more convenient but I am a bit worried as I saw some issues with BTRFS/Grub, Restore… (Better Snapper or Timeshift?)

The second option appears to be easier, no BTRFS hassles as I am fully EXT4.

What would be the best buildiso command/options to create an ISO to install from that even reinstall pCloud with my user name and password included, my /home, Chromium browser with even cookies and saved username and passwords for websites…

Or you recommend I better go with BTRFS as I described above.

Sorry, I feel I can not make a final decision. So I prefer to hear from experts and users and their experiences to make a final decision.

[linux-aarhus]

@limotux

It is clear the best is BTRFS, Timeshift or Snapper.

Well - that is a matter of opinion. btrfs certainly has interesting features but when it pukes - you may be lost.

I even noted update announcement comment on known issues …

I have tried btrfs - an no amount of wild horses are going to drag me down into that pool again.

@limotux

What would be the best buildiso command/options to create an ISO

There is no such command

• You define a set of packages
• You copy your preferred configuration into the desktop-overlay folder
• You build the ISO

Depending on your choices the ISO can get quite big

[Tickles]

So you use EXT4? I liked BTRFS and it did seem to be problematic at times.

[Zesko]

If you are sure for using full EXT4, you should test your hardware before: CPU cache, wrong binary calculation in CPU, RAM and Disks with different test tools for example: Memtest86+, mprime and linpack etc…

If you feel sure that hardwares are all ok, then you can use EXT4 completely.

Some people have problem that they got a lot of corrupt data without checksum ability when hardware have some issue without easily noticing

For example:

Recover corrupt data from LUKS partition?

After a data loss, I’m trying to restore files from a backup I made a few days ago on a LUKS-encrypted external drive. I now found that some of the files on the backup are corrupt - text files for example contain random binary data. I’m not sure how this happened in the first place. (I created the backup with rsync). The physical drive is /dev/sdd1. The LUKS device is /dev/mapper/luks-{hex string}. Any ideas for recovering this data? I was thinking fsck, but I don’t know how to apply that to a…

BTRFS error occurred while copying the zip file

Hi everyone, I got the BTRFS error while copying the zip file in my disk using BTRFS. The Copy process is canceled immediately. I tried cp ❯ cp ~/MEGA1.zip ~/Downloads cp: error reading '/home/zesko/MEGA1.zip': Input/output error if cp does not work, I tried scp: ❯ scp ~/MEGA1.zip ~/Downloads cp: error reading '/home/zesko/MEGA1.zip': Input/output error if scp does not work, I tried rsync: ❯ rsync -c ~/MEGA1.zip ~/Downloads rsync: [sender] read errors mapping "/home/zesko/MEGA1.zip":…

The test tool Memtest86+ can not detect the error of RAM, but BTRFS did.

[limotux]

@linux-aarhus

I have tried btrfs - an no amount of wild horses are going to drag me down into that pool again.

@linux-aarhus
So, this confirms my point of view, BTRFS is a no-go, as I don’t care about RAID or whatever other features. I would better stick with EXT4.

@linux-aarhus

There is no such command

How is that and we have Build Manjaro ISOs with buildiso - Manjaro that shows:

Build your ISO with the following command:

buildiso -p xfce

You need to specify the name of your ISO profile after the -p argument. In this case, it is xfce.

If the build process fails with an error, start it again.

and

You can use arguments with the buildiso command for more build options:

buildiso -f -p xfce -b stable

• -f let’s you specify if you want the full ISO. If omitted a minimal ISO will be build.
• -b let’s you specify the branch. You can also set this in your manjaro-tools.conf file.

You can find other examples of builds using different arguments for buildiso here.

So, I should perhaps

buildiso -f -p MyManjaro -b stable

But I am not sure what options to add to build an ISO of EVERYTHING on my laptop (including even cookies, user names and passwords in Chromiuom and pCloud… or whatever is installed, configured, logged in)

[limotux]

@Zesko

If you feel sure that hardwares are all ok, then you can use EXT4 completely.

Yes @Zesko
Everything is currently running perfectly since I stopped distrohopping since 11 March. (and installing only EXT4, no BTRFS)
I have been here before (some time ago but had lots of issues I guess was because of BTRFS and Baloo indexing all the time, I had some issues as well with Grub, I couldn’t reboot again. You may check my earlier posts last August/September 2021 e.g. System Gets Unresponsive (KDE) - #9 by limotux)

[limotux]

@Tickles

So you use EXT4? I liked BTRFS and it did seem to be problematic at times.

Yes, to summarize my experience with BTRFS:
1- It is good to have snapshots you can boot to just in case, but following problems were really bad.
2- I’m on KDE, and Baloo is very important to me (a book worm here + many other documents like medical reports, blood tests… (a bit old man here 60+, contracts, Legal… etc)
With BTRFS for /home, Baloo kept indexing ALL documents with every snapshot as if they were copies, so searching for a unique file XYZ.txt with unique text, it returns in Dolphin 3, 4, 5, … 10 results ALL pointing to the same file)

So, imagine I did only 2 blood tests, and BTRFS made with time 10 snapshots, if I want retrieve the 2 to compare I would get 20 results (10 and 10). And try to find out which are the 2 tests!
3- I had issues with Grub, sometimes it couldn’t boot the system to ANY snapshot… couldn’t boot at all. So, I had to continue distrohopping, tried EndeavourOS, and had same issues.

Then after lot of distrohopping I am settled with Manjaro BUT without BTRFS.
And in my Linux life since 2000 I was never happier.
A great distro, a great community, everything is more than perfect.

[Zesko]

@limotux

I guess was because of BTRFS and Baloo indexing all the time,

Yes, but the bug is Baloo, not BTRFS and not ZFS.

[limotux]

@Zesko

Yes, but the bug is Baloo, not BTRFS and not ZFS.

Sure, I understand. I saw a bug filed there. and to my surprise it was given a normal priority not a top priority I wonder how/why.

Still remains some issues with BTRFS as my previous post.

I am not saying by any means that BTRFS is bad, absolutely not. It is a wonderful file system and has lots of really needed features. (for me, snapshots, RAID perhaps, plus many more.)

Maybe if I don’t care about Baloo that much it may be perfect with me.
Maybe other issues I had with Grub was that I was trying some “bad” things like installing LXQT or XFCE, Dolphin, Baloo, Snapper+Timeshift… I know I messed up. But that time was my first attempt to try ARCH or Arch Distro.

[limotux]

@limotux

So, I should perhaps

buildiso -f -p MyManjaro -b stable

But I am not sure what options to add to build an ISO of EVERYTHING on my laptop (including even cookies, user names and passwords in Chromiuom and pCloud… or whatever is installed, configured, logged in)

So, I think I should go for buildiso.
Hopefully I can “cron” it daily or weekly, automagicaly copy it to pCloud folder (over write previous ISO), so I can install from it and get instantly my system as it was. Just in case.

[Locutus]

@limotux

(Baloo keeps reindexing all files thus makes the system unusable)

Not true. Baloo every few updates does run amok but it simply eats tons of memory slowing the system down when it does, but it has never actually harmed the file system.

[limotux]

@Locutus

it simply eats tons of memory slowing the system down when it does, but it has never actually harmed the file system.

My bad! Maybe I didn’t put it clear. I never said harming the file system I mean by

@limotux

makes the system unusable

Making ‘my system’ i.e. my machine, my laptop. not the file system.
Yes as you said, I agree with you, with what I quoted.

[linux-aarhus]

@limotux

How is that and we have Build Manjaro ISOs with buildiso - Manjaro that shows:

look at the process as an onion - with the tears

At the core you have the profile.conf - where can define as much as you like - in everything related including theming of the installer should you feel inclined t - including kernel, branch etc.

addgroups="lp,network,power,wheel"
autologin="true"
branch="unstable"
disable_systemd=('pacman-init')
disable_systemd_live=('tlp' 'tlp-sleep')
displaymanager="sddm"
efi_boot_loader="grub"
enable_systemd=('systemd-timesyncd' 'systemd-resolved' 'fstrim.timer' 'NetworkManager' 'firewalld')
extra="true"
geoip="true"
hostname="lxqt-kwin"
kernel="linux515"
login_shell="/bin/bash"
mhwd_used="true"
multilib="false"
nonfree_mhwd="false"
office_installer="false"
password="manjaro"
user_shell="/bin/bash"
username="manjaro"

Configuration are applied in order

1./etc/manjaro-tools/manjaro-tools.conf
2.~/.config/manjaro-tools/manjaro-tools.conf
3.~/iso-profiles/custom/my-xfce/profile.conf
4. command line override

If you run buildiso on its own it assume a build list - default is defined in manjaro-tools.conf - and stored in ~/.config/manjaro-tools/iso.list.d

#buildlist=default

I don’t think the option to build a list of profiles has been in much use - but it made sense when the tools was developed - before my time with Manjaro - which goes back to 2016.

To override that build list - you supply a profile using -p <profilename>.

In fact everything added to the commandline will override what is defined in configuration files.

E.g. the branch - if your profile says ‘stable’ you can override by specifying -b unstable and vice-versa.

So if your profile.conf defines everything you need, you can modify the default.list and specify your profile name and then all you have to do is to run buildiso with no arguments.

[limotux]

@linux-aarhus

look at the process as an onion - with the tears

Oooooopsss

I will see and try!
Maybe

buildiso -f -p manjaro -b stable -v -q 

[megavolt]

@limotux

With BTRFS for /home, Baloo kept indexing ALL documents with every snapshot as if they were copies, so searching for a unique file XYZ.txt with unique text, it returns in Dolphin 3, 4, 5, … 10 results ALL pointing to the same file)

Looks like baloo indexes your snapshots? That is a really bad behavior, but look, according to the wiki, you can exclude folders: Baloo/Configuration - KDE Community Wiki

Please don’t say you index the whole root filesystem with baloo!?

[linux-aarhus]

@limotux

I will see and try!

If you update your kernel without rebooting - then run buildiso - then you will get weird errors.

If you plan to run buildiso just after an update - reboot first … - just a heads-up

[Zesko]

@megavolt

you can exclude folders: Baloo/Configuration - KDE Community Wiki

I’ve already tried to configure Baloo filter that Baloo only indexes the ~/Documents directory, that works but:
Nevertheless, Baloo reindexes this same directory without changing after creating a new read-only snapshot and rebooting. That looks like a duplicates index of Baloo.

[limotux]

@megavolt

Looks like baloo indexes your snapshots?

Yes, it seems so. I did not select or configure anything. I just followed defaults.

@megavolt

Please don’t say you index the whole root filesystem with baloo!?

Honestly I don’t know. Followed defaults. I would like Baloo to index only /home (but there is a bug already filed about this)

Hmmm… sounds there might be a way to avoid this continuous reindexing?

The best thing I can think of is BTRFS for system and EXT4 for /home

[limotux]

@linux-aarhus

If you plan to run buildiso just after an update - reboot first … - just a heads-up

OK.
Just to report back.
After a few tries

sudo buildiso -f -p kde -b stable 

Seemed to wotk BUT I noticed it is like downloading and installing stuff not really creating an ISO of the installed system!
Did I do something wrong?

I couldn’t understand this. So I quit and recovered to the latest snapshot I took with TimeShift.

[linux-aarhus]

@limotux

sudo buildiso

don’t use sudo - you will mess with the permissions of the output dir

@limotux

I noticed it is like downloading and installing stuff not really creating an ISO of the installed system!

buildiso cannot create an image of your running system

@limotux

Did I do something wrong?

You used buildiso as a backup tool - which it is not - it is a deployment tool - but it can be used to create a user defined ISO - it does not backup your system