"emergency mode" after Timeshift snapshot recovery

megavolt · 11 September 2021 17:41

I don’t think so… Let’s assume he have installed an update. It updates the kernel, creates a new initramfs image and updates grub.

Now he rolled it back. But since /boot is not on the same partition (and therefore not included in the snapshot), it is not rolled back and stays the same. Now the new images after update stay and everything else is rolled back… So if he install a new kernel with mhwd, then the initramfs image will be recreated anyway, but the other stay.

Oh man… that is getting complicated…

On BTRFS /boot should be included and not a separated partition. Less headache, when using snapshots. Fully agree here with @andreas85

andreas85 · 11 September 2021 17:46

I think in /lib/modules on the disk there may be an old version of the modules from the rollback. When mkinitcpio runs it will try to use these. but the kernel is an newer version. At the moment the kernel and the initramfs may be the same version.

But You may be right. And it does not harm to try this. There will be no damage

gadzhi · 11 September 2021 18:16

[manjaro /]# mkinitcpio -P
==> Building image from preset: /etc/mkinitcpio.d/linux419.preset: 'default'
  -> -k /boot/vmlinuz-4.19-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.19-x86_64.img
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-4.19-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux419.preset: 'fallback'
  -> -k /boot/vmlinuz-4.19-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-4.19-x86_64-fallback.img -S autodetect
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-4.19-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'default'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64.img
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-5.10-x86_64'
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'fallback'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64-fallback.img -S autodetect
==> ERROR: specified kernel image does not exist: `/boot/vmlinuz-5.10-x86_64'

gadzhi · 11 September 2021 18:16

I made / boot a separate partition because grub doesn’t work with btrfs. So I had to do / boot ext4. This is how they explained it to me.

megavolt · 11 September 2021 18:19

Since /boot is a seprated partition, you need to mount /boot also and then chroot

sudo mount -U 901e9437-31d8-4a4f-ac41-d7e585b7cc79 /mnt/boot

andreas85 · 11 September 2021 18:33

A few years ago this has been true

https://btrfs.wiki.kernel.org/index.php/FAQ#Does_grub_support_Btrfs.3F

I do use grub
i do use btrfs for /
i do use zstd=9 !!!
I don’t have /boot (but i did configure /boot so that btrfs does not use compression there )
i do use zstd with initramfs (to save space even in /boot because kernels actually do decompress even zstd)
The only thing btrfs can’t do at this time is to store the last boot-entry on btrfs (GRUB_DEFAULT=saved does not work with btrfs)

gadzhi · 11 September 2021 18:53

[manjaro /]# update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

I don’t understand what the mistakes are

megavolt · 11 September 2021 19:05

Ok I try to explain it more simple:

When creating a snapshot on btrfs with timeshift, it only saves files on btrfs and therefore skipped the boot partition.
When creating a snapshot with rsync on timeshift, then it does not use the btrfs snapshot feature and therefore should also backup /boot (which is ext4).

The mistake is that the files on /boot are not on your btrfs partition: /@/boot and therefore timeshift skip it when using the btrfs snapshot feature on timeshift. That results in updated boot files, but rolled back modules, when rolling back.

andreas85 · 12 September 2021 06:17

So i would suggest to:

boot a live cd/usb
mount /boot also !
chroot into your installation

Now you should see /boot with the kernel and initramfs!

then please:
install kernel 5.10 with mhwd-kernel
https://wiki.manjaro.org/index.php/Manjaro_Kernels
look wehter the kernel in /boot/… changed
look wehter the modules in /lib/modules/kernel-version are present
look wehter the generation of the initramfs did work (timestamp of /boot/
This is the first step. When this succeeded we will work from there on

gadzhi · 12 September 2021 16:42

I tried to install the kernel

# mhwd-kernel -i linux510
:: Synchronizing package databases ...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to update multilib (unable to lock database)
error: failed to synchronize all databases

And I couldn’t do it.
Then I tried to update the system, but the same thing came out

# pacman -Syyyu
:: Synchronizing package databases ...
error: failed to update core (unable to lock database)
error: failed to update extra (unable to lock database)
error: failed to update community (unable to lock database)
error: failed to update multilib (unable to lock database)
error: failed to synchronize all databases

then i found a solution to this problem
rm -f /var/lib/pacman/db.lck
and now I can update.
But the update will make my system the freshest, and I would like to revert to my old snapshot. Is it possible?

andreas85 · 13 September 2021 07:12

was the rigth thing to do

Don’t update the complete system (pacman). In this special case you only have to install the kernel (mhwd-kernel …)
You will have the old system, but the new kernel. this will help us to boot again.

gadzhi · 13 September 2021 10:02

# mhwd-kernel -i linux510
:: Synchronizing package databases...
 core                                                              169.6 KiB   427 KiB/s 00:00 [#######################################################] 100%
 extra                                                            1893.0 KiB   431 KiB/s 00:04 [#######################################################] 100%
 community                                                           6.6 MiB   283 KiB/s 00:24 [#######################################################] 100%
 multilib                                                          174.6 KiB   262 KiB/s 00:01 [#######################################################] 100%
The following packages are out of date, please update your system first: 
........
Do you want to continue anyway? [y/N] y
error: no targets specified (use -h for help)

I tried to solve the problem myself but wasted too much time. Say that mhwd-kernel does not work in a chroot environment. How do I install the kernel via pacman?

gadzhi · 13 September 2021 11:36

[manjaro /]# pacman -Sy linux510
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
resolving dependencies...
looking for conflicting packages...

Packages (1) linux510-5.10.61-1

Total Download Size:    73.84 MiB
Total Installed Size:   78.23 MiB
Net Upgrade Size:      -71.11 MiB

:: Proceed with installation? [Y/n] y
:: Retrieving packages...
 linux510-5.10.61-1-x86_64                                          73.8 MiB   133 KiB/s 09:27 [#######################################################] 100%
(1/1) checking keys in keyring                                                                 [#######################################################] 100%
(1/1) checking package integrity                                                               [#######################################################] 100%
(1/1) loading package files                                                                    [#######################################################] 100%
(1/1) checking for file conflicts                                                              [#######################################################] 100%
(1/1) checking available disk space                                                            [#######################################################] 100%
:: Running pre-transaction hooks...
(1/2) Removing linux initcpios...
(2/2) Save Linux kernel modules
:: Processing package changes...
(1/1) upgrading linux510                                                                       [#######################################################] 100%
:: Running post-transaction hooks...
(1/5) Arming ConditionNeedsUpdate...
(2/5) Updating module dependencies...
(3/5) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'default'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64.img
==> Starting build: 5.10.61-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-5.10-x86_64.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'fallback'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64-fallback.img -S autodetect
==> Starting build: 5.10.61-1-MANJARO
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-5.10-x86_64-fallback.img
==> Image generation successful
(4/5) Updating Grub-Bootmenu
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done
(5/5) Restore Linux kernel modules
ls: cannot access 'backup': No such file or directory

==> Warning:
         -> Kernel has been updated. Modules of the current kernel
         -> have been backed up so you can continue to use your
         -> computer. However, the new kernel will only work 
         -> at next boot.


[manjaro /]# grub-update
bash: $'\320\277\320grub-update': command not found
[manjaro /]# update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

andreas85 · 13 September 2021 11:43

Thank you for the tip. I didn’t know mhwd-kernel does not work in chroot. I never used it in chroot myself @megavolt

So does it boot ? IF yes, you are save for now.

But we have to do something so that this never will happen again:

unmount /boot ! so that the /boot partition is invisible, and mount it at /mnt
now you should have an empty dir /boot
update grub to 2.04-1 or newer
modify /boot (inside /) so that btrfs will never compress files inside /boot
Can_I_disable_compression_on_a_file
Do this with no (for no compression) for /boot
Read the warning there
FS#63235 : [grub] 2.04 missing support for zstd on btrfs
This is only for safety. In case your GRUB is old this is needed ! If your GRUB is new this is not necessary
move all data from /mnt (your outside /boot partition) with rsync to /boot (inside /)
change the fstab, so that the /boot partition will not be mounted again
comment the line with /boot in it with an starting #
create a new grub.cfg, so that grub does not try to get the kernel from the /boot (external partition) but from btrfs /boot

After that you are really save (and can rollback any time)

gadzhi · 13 September 2021 12:49

no, now I am greeted with a new conclusion.

andreas85 · 13 September 2021 13:13

~~there may be one additional problem. Did you use compession in btrfs ?~~

what version is your grub ? 2.xx or 2.04-1 ?
only for safety:

you did mount the right snapshot (!!!) from your btrfs at ??? (maybe /mnt/)
so you could see there /mnt/home/, /mnt/usr … and an empty /mnt/boot (!!!), and /mnt/boot was empty (!!!)
you did mount /boot inside your ???/boot (maybe /mnt/boot/)
so you could see /mnt/boot/grub with all files included and a kernel and initramdisk at /mnt/boot/…
you did chroot after that
so you could see now /boot/grub with all files included and the kernel at /boot/…
you did update your kernel
you did recreate initramdisk with mkinitcpio
you did recreate grub configfile
(some commands may have included others already)

So reinstalling grub seems to be a good idea
(sorry, i myself did use GPT even with BIOS)

megavolt · 13 September 2021 13:43

I guess you forgot to mention that grub needs be installed again and since it is a pure old school BIOS:

So @gadzhi I believe, since it is a BIOS, the partition table must be also MS-DOS (and not GPT), therefore you need to reinstall grub (overwrite the MBR for the new boot location):

Do this in chroot!

grub-install --force --target=i386-pc --recheck --boot-directory=/boot /dev/sda

grub-mkconfig -o /boot/grub/grub.cfg

This should fix it…

gadzhi · 13 September 2021 14:18

# grub-install --version
grub-install (GRUB) 2.06~1~manjaro

i mounted like this:

[manjaro@manjaro ~]$ sudo mount -o subvol=@ -U abd07649-4af0-49fa-906b-c8cc116b98e6 /mnt
[manjaro@manjaro ~]$ sudo mount -U 901e9437-31d8-4a4f-ac41-d7e585b7cc79 /mnt/boot
[manjaro@manjaro ~]$ manjaro-chroot /mnt /bin/bash

as far as I understand, yes

I don’t know what it is, but I entered these commands a few days ago.

[manjaro /]# grub-install --force --target=i386-pc --recheck --boot-directory=/boot /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.

[manjaro /]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-5.10-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-5.10-x86_64.img
Found initrd fallback image: /boot/initramfs-5.10-x86_64-fallback.img
Found linux image: /boot/vmlinuz-4.19-x86_64
Found initrd image: /boot/intel-ucode.img /boot/initramfs-4.19-x86_64.img
Found initrd fallback image: /boot/initramfs-4.19-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
grub-probe: error: cannot find a GRUB drive for /dev/sdb4.  Check your device.map.
Found Windows 10 on /dev/sda1
Found Windows 10 on /dev/sda2
Found Manjaro Linux on /dev/sda8
Found Windows 10 on /dev/sdb1
Found memtest86+ image: /boot/memtest86+/memtest.bin
done

sorry

andreas85 · 13 September 2021 15:18

Since all 3 of us don’t know what there really happens. Then i would suggest:

Stop grub when grub-menu shows

Let it show (edit) the boot - entry it is gona use.
There examine

which device grub uses
where grub will look for the kernel
where grub will look for the initramfs
which startline will grub give to the kernel
does kernel mount @ or does it mount default ?

take this to a note on paper or take a photo

Then check these places

there must be such device with this uid or devicename for grub
the kernel (vmlinuz…) there must have an date from like in your notes and a reasonable size
the initramfs must have a date from yesterday (you may look into it with mc) and a reasonable size
look for the exact kernel version. Are there the needed files in /lib/modules/ with this exact directory. are the modules realy there ? Test with du
```
du --summarize --human-readable 5.10.59-1-MANJARO   
```
88M 5.10.59-1-MANJARO

Don´t compare my sizes for initramfs because it may contain other modules, and is compressed with zstd

 ls -lA
insgesamt 141812
-rw-r--r-- 1 root root    40960 19. Jul 21:16 amd-ucode.img
drwxr-xr-x 3 root root     4096  1. Jan 1970  efi
drwxr-xr-x 1 root root        6  1. Jan 1970  efi_backup
drwxr-xr-x 1 root root      122 12. Feb 2020  grml
drwxr-xr-x 1 root root      182 17. Aug 20:03 grub
-rw-r--r-- 1 root root 28835270 28. Aug 16:38 initramfs-4.19-x86_64-fallback.img
-rw-r--r-- 1 root root  9844849 28. Aug 16:38 initramfs-4.19-x86_64.img
-rw-r--r-- 1 root root 29406909 28. Aug 16:39 initramfs-5.10-x86_64-fallback.img
-rw-r--r-- 1 root root  9381096 28. Aug 16:39 initramfs-5.10-x86_64.img
-rw-r--r-- 1 root root 33090801 28. Aug 16:39 initramfs-5.13-x86_64-fallback.img
-rw-r--r-- 1 root root  9528972 28. Aug 16:39 initramfs-5.13-x86_64.img
-rw-r--r-- 1 root root       23 15. Aug 13:51 linux419-x86_64.kver
-rw-r--r-- 1 root root       22 15. Aug 15:10 linux510-x86_64.kver
-rw-r--r-- 1 root root       22 15. Aug 15:14 linux513-x86_64.kver
drwxr-xr-x 1 root root       22 13. Jul 17:00 memtest86+
-rw-r--r-- 1 root root  5986112 17. Aug 20:02 vmlinuz-4.19-x86_64
-rw-r--r-- 1 root root  9387104 17. Aug 20:02 vmlinuz-5.10-x86_64
-rw-r--r-- 1 root root  9681216 17. Aug 20:02 vmlinuz-5.13-x86_64

something to compare while in grub editor:

menuentry 'Manjaro Linux' --class manjaro --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-3487ba3d-1cba-4cdc-a043-c420ebca2aca' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod btrfs
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  3487ba3d-1cba-4cdc-a043-c420ebca2aca
else
  search --no-floppy --fs-uuid --set=root 3487ba3d-1cba-4cdc-a043-c420ebca2aca
fi
linux	/@/boot/vmlinuz-5.13-x86_64 root=UUID=3487ba3d-1cba-4cdc-a043-c420ebca2aca rw rootflags=subvol=@  bootsplash.bootfile=bootsplash-themes/manjaro/bootsplash modprobe.blacklist=ccp udev.log_priority=3 audit=0 sysrq_always_enabled=1
initrd	/@/boot/amd-ucode.img /@/boot/initramfs-5.13-x86_64.img

}
be aware, my /boot is already on btrfs