"emergency mode" after Timeshift snapshot recovery

No thats not the case… If I have no idea at this moment, then I don’t answer. And it is bad if only one person answers. More heads produce better results… (something like that). So @andreas85 explained it in an excellent way.

So on /boot there are initramfs. These initramfs are not rolled back and therefore it crash. I guess an easy fix could be:

  1. chroot like explained above
  2. recreate initramfs:
mkinitcpio -P
  1. recreate the grub menu:
update-grub
1 Like

You do all right, in only following one advice at a time! megavolt is very competent like a lot of members in this forum. I must be honest. you wrote the most of this in your first post. But i did not recognize some of it

So what do we know until now:

  • your /boot is an extra partition
  • you did a rollback with timeshift
  • the /boot is not corrupted

Lets assume that “Failed to start Load Kernel Modules” is because the Kernel-Modules on /lib/modules don’t match the kernel in /boot

There seem to be several ways to rescue your install:

  • try to make the corresponding kernel modules in btrfs available again
  • try to install new kernel, modules and initramfs (without updating the system)
  • try to undo the rollback (this will only work, if Timeshift did not delete the newest snapshot)
  • try to backup the data you need and to do a fresh install (this may be very time-comsuming)
  • maybe i can come up with others (this may take some time)
    As i said, i got burned this way already.

But i was able to rescue my installation.

After this i did remove the /boot and included it into /

1 Like

This is indeed the first(best) way to go. It may be additionally necessary to replace the kernel with mkwd-kernel so that kernel and /lib/modules/* have the same version.

1 Like

I don’t think so… Let’s assume he have installed an update. It updates the kernel, creates a new initramfs image and updates grub.

Now he rolled it back. But since /boot is not on the same partition (and therefore not included in the snapshot), it is not rolled back and stays the same. Now the new images after update stay and everything else is rolled back… So if he install a new kernel with mhwd, then the initramfs image will be recreated anyway, but the other stay.

Oh man… that is getting complicated…

On BTRFS /boot should be included and not a separated partition. Less headache, when using snapshots. Fully agree here with @andreas85 :slight_smile:

I think in /lib/modules on the disk there may be an old version of the modules from the rollback. When mkinitcpio runs it will try to use these. but the kernel is an newer version. At the moment the kernel and the initramfs may be the same version.

But You may be right. And it does not harm to try this. There will be no damage

1 Like
[manjaro /]# mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux419.preset: 'default'
  -> -k /boot/vmlinuz-4.19-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.19-x86_64.img
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-4.19-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux419.preset: 'fallback'
  -> -k /boot/vmlinuz-4.19-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.19-x86_64-fallback.img -S autodetect
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-4.19-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'default'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64.img
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-5.10-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'fallback'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64-fallback.img -S autodetect
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-5.10-x86_64'

I made / boot a separate partition because grub doesn’t work with btrfs. So I had to do / boot ext4. This is how they explained it to me.

Since /boot is a seprated partition, you need to mount /boot also and then chroot :wink:

sudo mount -U 901e9437-31d8-4a4f-ac41-d7e585b7cc79 /mnt/boot
1 Like

A few years ago this has been true :wink:

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_Btrfs.3F

  • I do use grub
  • i do use btrfs for /
  • i do use zstd=9 !!!
  • I don’t have /boot (but i did configure /boot so that btrfs does not use compression there :wink: )
  • i do use zstd with initramfs (to save space even in /boot because kernels actually do decompress even zstd)
    The only thing btrfs can’t do at this time is to store the last boot-entry on btrfs (GRUB_DEFAULT=saved does not work with btrfs)
[manjaro /]# update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

I don’t understand what the mistakes are

Ok I try to explain it more simple:

  1. When creating a snapshot on btrfs with timeshift, it only saves files on btrfs and therefore skipped the boot partition.
  2. When creating a snapshot with rsync on timeshift, then it does not use the btrfs snapshot feature and therefore should also backup /boot (which is ext4).

The mistake is that the files on /boot are not on your btrfs partition: /@/boot and therefore timeshift skip it when using the btrfs snapshot feature on timeshift. That results in updated boot files, but rolled back modules, when rolling back.

So i would suggest to:

  • boot a live cd/usb
  • mount /boot also !
  • chroot into your installation

Now you should see /boot with the kernel and initramfs!

then please:
install kernel 5.10 with mhwd-kernel
https://wiki.manjaro.org/index.php/Manjaro_Kernels
look wehter the kernel in /boot/… changed
look wehter the modules in /lib/modules/kernel-version are present
look wehter the generation of the initramfs did work (timestamp of /boot/
This is the first step. When this succeeded we will work from there on

I tried to install the kernel

# mhwd-kernel -i linux510
:: Synchronizing package databases ...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to update multilib (unable to lock database)
error: failed to synchronize all databases

And I couldn’t do it.
Then I tried to update the system, but the same thing came out

# pacman -Syyyu
:: Synchronizing package databases ...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to update multilib (unable to lock database)
error: failed to synchronize all databases

then i found a solution to this problem
rm -f /var/lib/pacman/db.lck
and now I can update.
But the update will make my system the freshest, and I would like to revert to my old snapshot. Is it possible?

was the rigth thing to do :+1:

Don’t update the complete system (pacman). In this special case you only have to install the kernel (mhwd-kernel …)
You will have the old system, but the new kernel. this will help us to boot again. :wink:

# mhwd-kernel -i linux510
:: Synchronizing package databases...
 core                                                              169.6 KiB   427 KiB/s 00:00 [#######################################################] 100%
 extra                                                            1893.0 KiB   431 KiB/s 00:04 [#######################################################] 100%
 community                                                           6.6 MiB   283 KiB/s 00:24 [#######################################################] 100%
 multilib                                                          174.6 KiB   262 KiB/s 00:01 [#######################################################] 100%
The following packages are out of date, please update your system first: 
........
Do you want to continue anyway? [y/N] y
error: no targets specified (use -h for help)

I tried to solve the problem myself but wasted too much time. Say that mhwd-kernel does not work in a chroot environment. How do I install the kernel via pacman?

[manjaro /]# pacman -Sy linux510
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
resolving dependencies...
looking for conflicting packages...

Packages (1) linux510-5.10.61-1

Total Download Size:    73.84 MiB
Total Installed Size:   78.23 MiB
Net Upgrade Size:      -71.11 MiB

:: Proceed with installation? [Y/n] y
:: Retrieving packages...
 linux510-5.10.61-1-x86_64                                          73.8 MiB   133 KiB/s 09:27 [#######################################################] 100%
(1/1) checking keys in keyring                                                                 [#######################################################] 100%
(1/1) checking package integrity                                                               [#######################################################] 100%
(1/1) loading package files                                                                    [#######################################################] 100%
(1/1) checking for file conflicts                                                              [#######################################################] 100%
(1/1) checking available disk space                                                            [#######################################################] 100%
:: Running pre-transaction hooks...
(1/2) Removing linux initcpios...
(2/2) Save Linux kernel modules
:: Processing package changes...
(1/1) upgrading linux510                                                                       [#######################################################] 100%
:: Running post-transaction hooks...
(1/5) Arming ConditionNeedsUpdate...
(2/5) Updating module dependencies...
(3/5) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'default'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64.img
==> Starting build: 5.10.61-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-5.10-x86_64.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'fallback'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64-fallback.img -S autodetect
==> Starting build: 5.10.61-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-5.10-x86_64-fallback.img
==> Image generation successful
(4/5) Updating Grub-Bootmenu
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done
(5/5) Restore Linux kernel modules
ls: cannot access 'backup': No such file or directory

==> Warning:
         -> Kernel has been updated. Modules of the current kernel
         -> have been backed up so you can continue to use your
         -> computer. However, the new kernel will only work 
         -> at next boot.


[manjaro /]# grub-update
bash: $'\320\277\320grub-update': command not found
[manjaro /]# update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done
1 Like

Thank you for the tip. I didn’t know mhwd-kernel does not work in chroot. I never used it in chroot myself @megavolt

So does it boot ? IF yes, you are save for now.

But we have to do something so that this never will happen again:

  • unmount /boot ! so that the /boot partition is invisible, and mount it at /mnt
    now you should have an empty dir /boot

  • update grub to 2.04-1 or newer

  • modify /boot (inside /) so that btrfs will never compress files inside /boot
    Can_I_disable_compression_on_a_file
    Do this with no (for no compression) for /boot
    Read the warning there
    FS#63235 : [grub] 2.04 missing support for zstd on btrfs
    This is only for safety. In case your GRUB is old this is needed ! If your GRUB is new this is not necessary

  • move all data from /mnt (your outside /boot partition) with rsync to /boot (inside /)

  • change the fstab, so that the /boot partition will not be mounted again
    comment the line with /boot in it with an starting #

  • create a new grub.cfg, so that grub does not try to get the kernel from the /boot (external partition) but from btrfs /boot

After that you are really save (and can rollback any time)

no, now I am greeted with a new conclusion.

there may be one additional problem. Did you use compession in btrfs ?

what version is your grub ? 2.xx or 2.04-1 ?
only for safety:

  • you did mount the right snapshot (!!!) from your btrfs at ??? (maybe /mnt/)
    so you could see there /mnt/home/, /mnt/usr … and an empty /mnt/boot (!!!), and /mnt/boot was empty (!!!)
  • you did mount /boot inside your ???/boot (maybe /mnt/boot/)
    so you could see /mnt/boot/grub with all files included and a kernel and initramdisk at /mnt/boot/…
  • you did chroot after that
    so you could see now /boot/grub with all files included and the kernel at /boot/…
  • you did update your kernel
  • you did recreate initramdisk with mkinitcpio
  • you did recreate grub configfile
    (some commands may have included others already)

:point_down: So reinstalling grub seems to be a good idea
(sorry, i myself did use GPT even with BIOS)

I guess you forgot to mention that grub needs be installed again and since it is a pure old school BIOS:

So @gadzhi I believe, since it is a BIOS, the partition table must be also MS-DOS (and not GPT), therefore you need to reinstall grub (overwrite the MBR for the new boot location):

:warning: Do this in chroot!

grub-install --force --target=i386-pc --recheck --boot-directory=/boot /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg

This should fix it…

1 Like