Failed system update with REISUB afterwards lead to boot problem

Hello,
First I want to say that I’ve been using Manjaro for my PAW workstation for ~4 years and I’m really happy about the stability and up-to-dateness.

Today I was apparently less lucky. After a not very successful system update via the xfce packetmanager I had to shutdown the system via REISUB. Then when booting it did not find a certain kernel in /boot. Unfortunately I have not yet found the exact error message via journalctl. I was able to work around the boot error by taking a BTRFS snapshot from the grub boot menu. (Thanks for that!!)

Updating again via the package manager, which looked successful, on reboot not only did not fix the error, but the snapshot now showed the same kernel/boot error. Each time I had to go back one snapshot further. The default boot entry is still not working. I would like to fix this if possible.
I would be happy to provide any log files needed here.

English is not my native language, I use Deepl for translations. I would be very happy about any help.

Are you using Timeshift? That creates writable snapshots by default, not read-only snapshot.

The issue is:

After booting into your selected writable snapshot, if you ran the system update using update-grub to overwrite the good snapshot, then you have to boot into the same snapshot after every reboot.

For example:

1 Like

Your problems “may be” the result of not resolving some pacnew-files

1st Rollback

After you did this successfully, create a snapshot.

2nd resolve your pacnew-files !

After you did this successfully, create a a snapshot.

3rd Retry the update

But don´t reboot yet !

4th resolve your pacnew-files !

If there where any, retry the update, then reboot.

Again take a snapshot

:footprints:

1 Like
1 Like

Hello Zesko!
Thanks for replying to my post!

You asked me:

Are you using Timeshift? That creates writable snapshots by default, not read-only snapshot.

Yes, it is a default installation and yes it seems timeshift is already installed beside btrfs/snapshots which are created automatically.

After booting into your selected writable snapshot, if you ran the system update using update-grub to overwrite the good snapshot, then you have to boot into the same snapshot after every reboot.

I will try this with my current snapshot! After updating my system via packetmanager successfully I will use the update-grub command afterwards and will report the result here. I hope this fixes my “default” (first) boot option. Thanks again for your efforts!

Hello @Zesko!
Was there, done that!

I successfully updated my system and ran the update-grub command. This did not result in a standard boot procedure via default (per countdown), but at least I can now reuse the last snapshot and do not have to go back further in the snapshot history!

I have attached two screenshots with error messages for the default boot and one snapshot that no longer works (#1):

#1

Translation:

“Error: File >>/@/boot/vmlinuz-6.1-x86_64<< not found”
“Error: You have to load the kernel first.”
“Press any key to continue”

#2

Translation:

"Loading Snapshot […]
"Loading Kernel: vmlinuz-6.1-x86_64 […]
“Error: File >>timeshift/snapshot […] not found.”
“Error: You have to load the kernel first.”
“Press any key to continue”

I would really like to avoid booting a specific (the current) snapshot and would like to go back to default if possible.

N00b yes, but a very patient one! ^^

I just saw your message under @Zesko’s and of course will do it right away. Thanks, for everything - not just this solution here. Manjaro is getting better and better. It takes a long time for a new standard to catch on, unfortunately. I wish you and all of us the time, strength and patience to make it happen!

Get back to you in a minute, been using Ventoy for some time here at the company. Very cool project with a lot of added value. (Thanks for this tip too!).


Moderator edit: Fixed topic quote

@Aragorn N00b’ling here again! :smiley:

I got the following error from manjaro-chroot -a in the terminal after sudo su -:

I bootet the Manjaro .iso in normal mode from my Ventoy USB stick.

There is no “normal”-mode :man_shrugging:

You need to select (when selecting the start-entry) the same mode as you used when installing your manjaro

So if you tried with BIOS/legacy-mode this time, you may need to try UEFI-mode instead

1 Like

Ventoy provides two modes. “Normal” boot mode and “Grub2” mode.

I tried both meanwhile. But no success with manjaro-chroot -a because of "Error: No Linux partitions detected"

I tried both as written above, as well as UEFI/Legacy from the BIOS - with no success. Maybe you should take a look at this:

Patriot M.2 contains the installed and currently broken Manjaro installation.

This is the “default” Manjaro (edit) boot option of the Patriot M.2. It is BTRFS. Maybe this is the reason why manjaro-chroot -a does not work immediately(?). Maybe ext4 is assumed?


Moderator edit: Fixed topic quoting…again

Yes, manjaro-chroot does not support btrfs, and I wrote that HowTo for people who use ext4.

With btrfs, you have to manually mount all the pertinent filesystems and subvolumes before chrooting.

So, assuming that your system boots up in native UEFI mode, that your EFI partition is /dev/sda1 and that your root filesystem is at @ on /dev/sda2 — adjust this to the situation on your machine! — you would do something like this…
:arrow_down:

sudo su -
mount -t btrfs -o subvol=@ /dev/sda2 /mnt
mount -t btrfs -o subvol=@home /dev/sda2 /mnt/home
mount -t btrfs -o subvol=@log /dev/sda2 /mnt/var/log
mount -t btrfs -o subvol=@cache /dev/sda2 /mnt/var/cache
mount -t vfat  /dev/sda1 /mnt/boot/efi
mount --bind /dev /mnt/dev
mount -t proc proc /mnt/proc
mount -t sysfs sysfs /mnt/sys
mount -t efivarfs efivarfs /mnt/sys/firmware/efi/efivars
chroot /mnt /bin/bash
[ -f /var/lib/pacman/db.lck ] && rm -f /var/lib/pacman/db.lck
pacman-mirrors -f 5 && pacman -Syyu
exit

Let the whole thing finish, no matter how long it takes. Once it’s done and you get a new command prompt, then you can safely reboot the machine.

1 Like

Thank you @Aragorn . You have already done a lot to support my case!

I have tried to adopt it to my specific case and checked the mounted and working snapshot I use atm:

Based on the findmnt and mnt information (below), it is atm over my head how I should adopt it. Could maybe someone else help me out?

Please don’t use screenshots for posting terminal output. Copy the output with your mouse and paste it in between two lines of three backticks. That way it’s properly formatted and readable.

So you are apparently in a working environment now. Now update your system… :arrow_down:

sudo pacman-mirrors -f 5 && sudo pacman -Syyu

Alternatively, you can also use pamac — without sudo! — although it’s still better to use pacman for Manjaro packages. You can then use pamac afterwards for updating anything from the AUR, Snaps, et al. :arrow_down:

pamac update --force-refresh

Then, as @andreas85 said, merge your .pacnew files, reboot, and if everything works, create a new snapshot of your working system.

1 Like

mount

sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=32801756k,nr_inodes=8200439,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/nvme0n1p2 on / type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=697,subvol=/timeshift-btrfs/snapshots/2023-08-14_11-00-01/@)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1551)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-vconsole-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/nvme0n1p2 on /swap type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@swap)
/dev/nvme0n1p2 on /home type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=447,subvol=/@home)
/dev/nvme0n1p2 on /var/cache type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=258,subvol=/@cache)
/dev/nvme0n1p2 on /var/log type btrfs (rw,relatime,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@log)
tmpfs on /tmp type tmpfs (rw,noatime,inode64)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=6565704k,nr_inodes=1641426,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/sda1 on /run/media/hpweecks/ADATA_SU650_SSD type btrfs (rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/,uhelper=udisks2)
/dev/sdb1 on /run/media/hpweecks/Samsung_850_SSD type btrfs (rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/,uhelper=udisks2)

findmnt

/                                                       /dev/nvme0n1p2[/timeshift-btrfs/snapshots/2023-08-14_11-00-01/@] btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=697,subvol=/timeshift-btrfs/snapshots/2023-08-
├─/proc                                                 proc                                                             proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                            systemd-1                                                        autofs          rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1551
│   └─/proc/sys/fs/binfmt_misc                          binfmt_misc                                                      binfmt_misc     rw,nosuid,nodev,noexec,relatime
├─/sys                                                  sys                                                              sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                           efivarfs                                                         efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                                securityfs                                                       securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                      cgroup2                                                          cgroup2         rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore                                      pstore                                                           pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                         bpf                                                              bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                   debugfs                                                          debugfs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                 tracefs                                                          tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections                            fusectl                                                          fusectl         rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config                                  configfs                                                         configfs        rw,nosuid,nodev,noexec,relatime
├─/dev                                                  dev                                                              devtmpfs        rw,nosuid,relatime,size=32801756k,nr_inodes=8200439,mode=755,inode64
│ ├─/dev/pts                                            devpts                                                           devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                                            tmpfs                                                            tmpfs           rw,nosuid,nodev,inode64
│ ├─/dev/mqueue                                         mqueue                                                           mqueue          rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages                                      hugetlbfs                                                        hugetlbfs       rw,relatime,pagesize=2M
├─/run                                                  run                                                              tmpfs           rw,nosuid,nodev,relatime,mode=755,inode64
│ ├─/run/credentials/systemd-sysctl.service             ramfs                                                            ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/media/hpweecks/ADATA_SU650_SSD                 /dev/sda1                                                        btrfs           rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
│ ├─/run/credentials/systemd-tmpfiles-setup-dev.service ramfs                                                            ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-vconsole-setup.service     ramfs                                                            ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/media/hpweecks/Samsung_850_SSD                 /dev/sdb1                                                        btrfs           rw,nosuid,nodev,relatime,ssd,space_cache=v2,subvolid=5,subvol=/
│ ├─/run/credentials/systemd-tmpfiles-setup.service     ramfs                                                            ramfs           ro,nosuid,nodev,noexec,relatime,mode=700
│ └─/run/user/1000                                      tmpfs                                                            tmpfs           rw,nosuid,nodev,relatime,size=6565704k,nr_inodes=1641426,mode=700,uid=1000,gid=1000,inode64
│   ├─/run/user/1000/gvfs                               gvfsd-fuse                                                       fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│   └─/run/user/1000/doc                                portal                                                           fuse.portal     rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
├─/swap                                                 /dev/nvme0n1p2[/@swap]                                           btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@swap
├─/home                                                 /dev/nvme0n1p2[/@home]                                           btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=447,subvol=/@home
├─/var/cache                                            /dev/nvme0n1p2[/@cache]                                          btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=258,subvol=/@cache
├─/var/log                                              /dev/nvme0n1p2[/@log]                                            btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=259,subvol=/@log
├─/tmp                                                  tmpfs                                                            tmpfs           rw,noatime,inode64
└─/boot/efi                                             /dev/nvme0n1p1                                                   vfat            rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-r

I don’t know what you are trying to show me with this. You clearly have a working system again now, so now you have to properly update it. :man_shrugging:

Done. I replied it as an seperate answer to my own post.

Done.

Output

::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
::INFO => Mirror status: https://repo.manjaro.org/status.json
::INFO Using default mirror file
::INFO Querying mirrors - This may take some time
  0.540 Belarus        : http://mirror.datacenter.by/pub/mirrors/manjaro/
  1.023 United_States  : https://codingflyboy.mm.fcix.net/manjaro/
  ..... South_Africa   : http://mirror.is.co.za/mirrors/manjaro.org/
  1.199 Belgium        : http://ftp.belnet.be/mirrors/manjaro/repos/
  2.125 New_Zealand    : https://mirror.2degrees.nz/manjaro/
::INFO Writing mirror list
::Belarus         : http://mirror.datacenter.by/pub/mirrors/manjaro/stable
::United_States   : https://codingflyboy.mm.fcix.net/manjaro/stable
::Belgium         : http://ftp.belnet.be/mirrors/manjaro/repos/stable
::New_Zealand     : https://mirror.2degrees.nz/manjaro/stable
::South_Africa    : http://mirror.is.co.za/mirrors/manjaro.org/stable
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
:: Paketdatenbanken werden synchronisiert …
 core                                                                                                              145,7 KiB   280 KiB/s 00:01 [#######################################################################################] 100%
 extra                                                                                                               8,6 MiB   647 KiB/s 00:14 [#######################################################################################] 100%
 community                                                                                                          29,0   B   580   B/s 00:00 [#######################################################################################] 100%
 multilib                                                                                                          146,8 KiB   302 KiB/s 00:00 [#######################################################################################] 100%
:: Vollständige Systemaktualisierung wird gestartet …
 Es gibt nichts zu tun

Done.

Output:

Synchronisiere Paketdatenbanken...
Aktualisierung von core.db...                                                                                                                                                                                                                
Aktualisierung von extra.db...                                                                                                                                                                                                               
Aktualisierung von community.db...                                                                                                                                                                                                           
Aktualisierung von multilib.db...                                                                                                                                                                                                            
Aktualisierung von core.files...                                                                                                                                                                                                             
Aktualisierung von extra.files...                                                                                                                                                                                                            
Aktualisierung von community.files...                                                                                                                                                                                                        
Aktualisierung von multilib.files...                                                                                                                                                                                                         
Aktualisierung von AUR...                                                                                                                                                                                                                    
Es gibt nichts zu tun.                                                                                                                                                                                                                       
Vorgang erfolgreich abgeschlossen.

I will figure out what .pacnew files are and will report back after finishing the merge.

This is not the Manjaro “default” boot option but a working BTRFS snapshot. As I wrote above I would like to have a fixed “default” boot option for the future instead booting manually into a working snapshot.

Link to my problem (please extend the view):

Source : System Maintenance - Manjaro Wiki

One of the .pacnew files deals with an important change to /etc/pacman.conf, i.e. the fact that the community repository was merged into the extra repository, and that as such, the community repository is now empty and should be removed from /etc/pacman.conf.

1 Like

Seems I am ready. What can I do now to restore the Manjaro default boot option? I suppose I need to run manajro-chroot -a and mount the appropriate disk/partitions.

Well, there are two options…

  1. Delete your main btrfs root volume and rename the current working snapshot to @; or…
  2. Copy this working snapshot onto your existing @.

#2 will probably be easiest, because #1 would require changing your boot loader configuration and your /etc/fstab to point to the correct subvolume, given that this working snapshot has a different subvolid. So #2 is the option I am going to discuss below, and we’re going to be doing this the old-school way, by literally copying the whole directory structure and all of the files.

First, you will need to mount the parent subvolume of the btrfs filesystem, because you cannot do that from within the running subvolume. And that means that you’ll have to boot up from the live USB/CD/DVD. Then, open up a terminal window in the live USB/CD/DVD and become root… :arrow_down:

sudo -i

Mount the parent volume to /mnt:arrow_down:

mount -t btrfs /dev/nvme0n1p2 /mnt

Check the contents of the root volume… :arrow_down:

ls -1 /mnt/

If all is well, you should see the following…

@
@var
@home
@log
timeshift-btrfs

Now we will copy over the contents of the working snapshot to the @ subvolume… :arrow_down:

cp -RPpv /mnt/timeshift-btrfs/snapshots/2023-08-14_11-00-01/@/* /mnt/@/ && sync

Let the command finish, and when you’re done, unmount the volume again… :arrow_down:

umount /mnt

… and reboot the machine while crossing your fingers (and removing the USB/CD/DVD medium).

:crossed_fingers: