Cannot boot after last update hanged

Zeosleus · 27 April 2023 15:31

Hello everyone. I am writing this from a Manjaro Live USB as I currently cannot boot into my installation.
This morning I started an update (typical -Syu) and while I was using my computer, it froze for about 5 minutes. Although I know from personal experience that update should not be interrupted, I actually hard-restarted my PC using the corresponding button from the case.

I now have the typical “vmlinuz” not found error, as the update was disrupted before the kernel generation stage.
This is something that I have dealt with in the past and I know the procedure of chrooting using a live USB into the actual installation and performing the update.
So this is what I actually did :

chroot-ed into the installation
pacman -Syu to finish the update (had to force it because pacman db was locked)
tried to run grub-update and reboot but it was not successful.

I know that this is a relatively common issue, and have tried everything I have found in older posts of people having the same issue… However nothing seems to work, and I have started to freak out a bit since I am on a tight schedule with some work that needs to be done.

The output of the basic commands for this type of issue, can be seen below.

mhwd-kernels -li :

Currently running: 6.1.25-1-MANJARO (linux61)
The following kernels are installed in your system:
   * linux510
   * linux515
   * linux54

/boot contains (no vmlinuz) :

drwxr-xr-x  5 root root  4096 Apr 27 14:12 .
drwxr-xr-x 17 root root  4096 Apr 27 13:32 ..
-rw-r--r--  1 root root 51200 Apr  1 06:39 amd-ucode.img
drwxr-xr-x  3 root root  4096 Jan  1  1970 efi
drwxr-xr-x  6 root root  4096 Apr 27 15:50 grub
-rw-r--r--  1 root root    23 Apr 20 14:40 linux510-x86_64.kver
-rw-r--r--  1 root root    23 Apr 20 14:40 linux515-x86_64.kver
-rw-r--r--  1 root root    22 Apr 20 14:44 linux54-x86_64.kver
drwxr-xr-x  2 root root  4096 Apr 27 14:13 memtest86+

mkinitcpio -P :

==> ERROR: No presets found in /etc/mkinitcpio.d

and when i try to (say) mhwd-kernel -i linux63 i get the “error: no targets specified (use -h for help)” error.

So the main issue seems to be that my installation “thinks” that the 3 kernels listed above are actually installed, while they are not. So should i use mhwd-kernels to remove those kernels and then try to install them again ? However, if something like that would fix the issue, why doesn’t the mhwd-kernel -i linux63 succeed?

Any help will be greatly appreciated !

stephane · 27 April 2023 15:50

create with nano 3 kernels.preset
under chroot

nano /etc/mkinitcpio.d/linux515.preset
# mkinitcpio preset file for the 'linux515' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-5.15-x86_64"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-5.15-x86_64.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-5.15-x86_64-fallback.img"
fallback_options="-S autodetect"

then redo

mkinicpio -p

Zeosleus · 27 April 2023 16:09

Thanks, but i ended up solving it another way…

While inside chroot, I ended up doing “pacman -S linux515” , which obviously didn’t cross my mind… So i got to boot up into my installation.

However, if I let my computer boot normally, it goes into grub rescue CLI.
The only way to boot into my system, is by going into the boot options and selecting the drive where my installation is…
So it’s like grub cannot see the correct bootable drive.
When I booted into my system, i ran “update-grub” again, but nothing changed for the above issue…

Any ideas?

I ran “mkinitcpio -P” again and for kernel 5.10 it returns an error.

==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'default'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64.img
==> ERROR: '/lib/modules/5.10.178-1-MANJARO' is not a valid kernel module directory
==> Building image from preset: /etc/mkinitcpio.d/linux510.preset: 'fallback'
  -> -k /boot/vmlinuz-5.10-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.10-x86_64-fallback.img -S autodetect
==> ERROR: '/lib/modules/5.10.178-1-MANJARO' is not a valid kernel module directory
==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'default'
  -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64.img
==> Starting build: '5.15.108-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-5.15-x86_64.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux515.preset: 'fallback'
  -> -k /boot/vmlinuz-5.15-x86_64 -c /etc/mkinitcpio.conf -g /boot/initramfs-5.15-x86_64-fallback.img -S autodetect
==> Starting build: '5.15.108-1-MANJARO'
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
  -> Running build hook: [keyboard]
  -> Running build hook: [keymap]
  -> Running build hook: [resume]
  -> Running build hook: [filesystems]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: '/boot/initramfs-5.15-x86_64-fallback.img'
==> Image generation successful

stephane · 27 April 2023 17:17

can you return

cat /etc/mkinitcpio.d/linux510.preset

for grub restore , check before if luks is required

cat /etc/fstab

and then re install grub and do update-grub

varikonniemi · 27 April 2023 17:21

booting directly uses the entry in your firmware memory. Booting from menu, probably the “fallback/portable/removable” version of GRUB is used.

No other reason shouls exist why on uefi system direct boot fails, while specifying boot device works.

After you successfully ran update-grub i don’t know why it still would not work correctly, maybe dare i say it, you need the big hammer and do grub-install again?

This is why even though i have a modern system i use it in “compatibility” mode and things just work as the bootloader resides in MBR and that’s it.

Zeosleus · 27 April 2023 18:27

# mkinitcpio preset file for the 'linux510' package

ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-5.10-x86_64"

PRESETS=('default' 'fallback')

#default_config="/etc/mkinitcpio.conf"
default_image="/boot/initramfs-5.10-x86_64.img"
#default_options=""

#fallback_config="/etc/mkinitcpio.conf"
fallback_image="/boot/initramfs-5.10-x86_64-fallback.img"
fallback_options="-S autodetect"

my /etc/fstab looks fine :

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=B779-D001                            /boot/efi      vfat    umask=0077 0 2
UUID=7af99ff8-6ce6-4359-8f27-3601b0570142 /              ext4    defaults,noatime 0 1
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0
/var/cache/swap/swapfile                none            swap    sw      0       0

Zeosleus · 27 April 2023 18:28

If i let it boot normally without going into boot options, grub goes into rescue mode, and prints " grub_debug_malloc not found" …

I have tried reinstalling grub while being inside my installation (not by a chroot from live usb) and nothing changed…

stephane · 27 April 2023 18:34

you have to go in chroot for that :
/boot/efi have to be from disk , not from USB live Key ( its also be blocked for that reason )

varikonniemi · 27 April 2023 18:43

i think you can do it from normal boot or chroot, not directly from live usb boot.

varikonniemi · 27 April 2023 18:44

how did you try reinstall grub?

Zeosleus · 27 April 2023 18:55

I tried to reinstall grub while inside my system, using the commands from here https://wiki.manjaro.org/index.php/GRUB/Restore_the_GRUB_Bootloader#Reinstall_GRUB
but no luck…

I even booted from the live usb and tried :

manjaro-chroot -a
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=manjaro --recheck (completed wirhout errors)
update-grub

and then when i rebooted, it still goes into grub rescue, if i don’t manually select the boot device…
wtf is going on…

varikonniemi · 27 April 2023 19:34

i have no idea. One thing that came to mind is to try grub-install with additional --removable flag. If this works, then it would seem some file is corrupt that grub cannot fix.

Which would be weird, as currently when you select from boot menu you arrive at same point as if you just installed --removable, AFAIK

stephane · 27 April 2023 19:39

can you , from USB live iso manjaro ,
provide from terminal

inxi -Fza
sudo parted -l
sudo lsblk -fs

Zeosleus · 28 April 2023 11:10

inxi -Fza :

System:
  Kernel: 6.1.25-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 12.2.1
    parameters: BOOT_IMAGE=/boot/vmlinuz-x86_64 lang=en_US keytable=us tz=UTC
    misobasedir=manjaro misolabel=MANJARO_KDEM_2210 quiet
    systemd.show_status=1 splash driver=nonfree nouveau.modeset=0
    i915.modeset=1 radeon.modeset=1
  Desktop: KDE Plasma v: 5.27.4 tk: Qt v: 5.15.8 wm: kwin_x11 vt: 1 dm: SDDM
    Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: Micro-Star model: B450 TOMAHAWK MAX (MS-7C02) v: 1.0
    serial: <superuser required> UEFI: American Megatrends v: 3.30
    date: 09/17/2019
CPU:
  Info: model: AMD Ryzen 5 3600 bits: 64 type: MT MCP arch: Zen 2 gen: 3
    level: v3 note: check built: 2020-22 process: TSMC n7 (7nm)
    family: 0x17 (23) model-id: 0x71 (113) stepping: 0 microcode: 0x8701013
  Topology: cpus: 1x cores: 6 tpc: 2 threads: 12 smt: enabled cache:
    L1: 384 KiB desc: d-6x32 KiB; i-6x32 KiB L2: 3 MiB desc: 6x512 KiB
    L3: 32 MiB desc: 2x16 MiB
  Speed (MHz): avg: 3615 high: 3889 min/max: N/A cores: 1: 3600 2: 3600
    3: 3491 4: 3600 5: 3600 6: 3600 7: 3889 8: 3600 9: 3600 10: 3600 11: 3600
    12: 3600 bogomips: 86441
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: retbleed mitigation: untrained return thunk; SMT enabled with STIBP
    protection
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Retpolines, IBPB: conditional, STIBP:
    always-on, RSB filling, PBRSB-eIBRS: Not affected
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: NVIDIA GK208B [GeForce GT 720] vendor: Micro-Star MSI
    driver: nvidia v: 470.182.03 alternate: nouveau,nvidia_drm non-free:
    series: 470.xx+ status: legacy-active (EOL~2023/24) arch: Kepler
    code: GKxxx process: TSMC 28nm built: 2012-18 pcie: gen: 2 speed: 5 GT/s
    lanes: 4 link-max: lanes: 8 bus-ID: 25:00.0 chip-ID: 10de:1288
    class-ID: 0300
  Device-2: Logitech Webcam C310 type: USB driver: snd-usb-audio,uvcvideo
    bus-ID: 1-6:3 chip-ID: 046d:081b class-ID: 0102 serial: <filter>
  Display: x11 server: X.Org v: 21.1.8 compositor: kwin_x11 driver: X:
    loaded: nvidia gpu: nvidia display-ID: :0 screens: 1
  Screen-1: 0 s-res: 3200x1080 s-dpi: 101 s-size: 805x272mm (31.69x10.71")
    s-diag: 850mm (33.45")
  Monitor-1: DVI-D-0 pos: primary,left res: 1920x1080 hz: 60 dpi: 102
    size: 480x270mm (18.9x10.63") diag: 551mm (21.68") modes: N/A
  Monitor-2: VGA-0 pos: right res: 1280x1024 hz: 60 dpi: 96
    size: 338x270mm (13.31x10.63") diag: 433mm (17.03") modes: N/A
  API: OpenGL v: 4.6.0 NVIDIA 470.182.03 renderer: NVIDIA GeForce GT
    720/PCIe/SSE2 direct-render: Yes
Audio:
  Device-1: NVIDIA GK208 HDMI/DP Audio vendor: Micro-Star MSI
    driver: snd_hda_intel v: kernel bus-ID: 1-6:3 chip-ID: 046d:081b pcie:
    gen: 2 class-ID: 0102 serial: <filter> speed: 5 GT/s lanes: 4 link-max:
    lanes: 8 bus-ID: 25:00.1 chip-ID: 10de:0e0f class-ID: 0403
  Device-2: AMD Starship/Matisse HD Audio vendor: Micro-Star MSI
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 27:00.4 chip-ID: 1022:1487 class-ID: 0403
  Device-3: Logitech Webcam C310 type: USB driver: snd-usb-audio,uvcvideo
  API: ALSA v: k6.1.25-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsamixer,amixer
  Server-1: JACK v: 1.9.22 status: off tools: N/A
  Server-2: PipeWire v: 0.3.70 status: off with: wireplumber status: off
    tools: pw-cli,wpctl
  Server-3: PulseAudio v: 16.1 status: active tools: pacat,pactl
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: Micro-Star MSI driver: r8169 v: kernel pcie: gen: 1 speed: 2.5 GT/s
    lanes: 1 port: f000 bus-ID: 22:00.0 chip-ID: 10ec:8168 class-ID: 0200
  IF: enp34s0 state: up speed: 100 Mbps duplex: full mac: <filter>
Drives:
  Local Storage: total: 1.03 TiB used: 0 KiB (0.0%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/sda maj-min: 8:0 vendor: Western Digital
    model: WDS120G2G0A-00JH30 size: 111.8 GiB block-size: physical: 512 B
    logical: 512 B speed: 6.0 Gb/s type: SSD serial: <filter> rev: 0000
    scheme: GPT
  ID-2: /dev/sdb maj-min: 8:16 vendor: Seagate model: ST1000DM010-2EP102
    size: 931.51 GiB block-size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s
    type: HDD rpm: 7200 serial: <filter> rev: CC43 scheme: MBR
  ID-3: /dev/sdc maj-min: 8:32 type: USB vendor: Generic model: Flash Disk
    size: 7.27 GiB block-size: physical: 512 B logical: 512 B type: SSD
    serial: <filter> rev: 8.07 scheme: MBR
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
Partition:
  Message: No partition data found.
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 36.4 C mobo: N/A gpu: nvidia temp: 39 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 35%
Info:
  Processes: 244 Uptime: 19m wakeups: 0 Memory: 15.57 GiB
  used: 3.23 GiB (20.7%) Init: systemd v: 252 default: graphical
  tool: systemctl Compilers: gcc: 12.2.1 clang: 15.0.7 Packages: pm: pacman
  pkgs: 1140 libs: 317 tools: pamac Shell: Zsh v: 5.9 default: Bash v: 5.1.16
  running-in: yakuake inxi: 3.3.26

sudo parted -l

Model: ATA WDC WDS120G2G0A- (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      2097kB  317MB  315MB  fat32              boot, esp
 2      317MB   120GB  120GB  ext4


Model: ATA ST1000DM010-2EP1 (scsi)
Disk /dev/sdb: 1000GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size   Type     File system  Flags
 1      1049kB  218GB   218GB  primary  ntfs
 2      218GB   1000GB  782GB  primary  ext4


Model: Generic Flash Disk (scsi)
Disk /dev/sdc: 7801MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 2      3503MB  3507MB  4194kB  primary               esp

sudo lsblk -fs :

NAME  FSTYPE   FSVER            LABEL             UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0                                                                           0   100% /run/miso/sfs/livefs
loop1 squashfs 4.0                                                                           0   100% /run/miso/sfs/mhwdfs
loop2 squashfs 4.0                                                                           0   100% /run/miso/sfs/desktopfs
loop3 squashfs 4.0                                                                           0   100% /run/miso/sfs/rootfs
sda1  vfat     FAT32                              B779-D001                                           
└─sda                                                                                                 
sda2  ext4     1.0                                7af99ff8-6ce6-4359-8f27-3601b0570142                
└─sda                                                                                                 
sdb1  ntfs                      test_200          6203F6F92C52018A                                    
└─sdb                                                                                                 
sdb2  ext4     1.0                                187cf711-432c-4993-8928-6a3cc10eadd8                
└─sdb                                                                                                 
sdc1  iso9660  Joliet Extension MANJARO_KDEM_2210 2023-04-21-10-01-10-00                              
└─sdc iso9660  Joliet Extension MANJARO_KDEM_2210 2023-04-21-10-01-10-00                     0   100% /run/miso/bootmnt
sdc2  vfat     FAT12            MISO_EFI          A606-BD92                                           
└─sdc iso9660  Joliet Extension MANJARO_KDEM_2210 2023-04-21-10-01-10-00                     0   100% /run/miso/bootmnt

@stephane can’t see anything really wrong…

stephane · 28 April 2023 11:41

update your UEFI motherboard

Zeosleus · 28 April 2023 11:45

I thought about that, however how could everything be OK before yesterday’s update and now all of a sudden this is required?

Also, which section of the command output you highlighted, indicates that the MOBO needs an update ?

dmt · 28 April 2023 13:24

Have you tried re-installing linux510?

It sounds like you have more than one grub, or perhaps it’s trying to boot 5.10. How are you selecting this device?

Let’s see what boot entries you have.

efibootmgr

Zeosleus · 28 April 2023 13:29

After some fixes, I seem to have 5.10 installed (pacman -Q linux510 returns the package) and the corresponding vmlinuz file is inside /boot.

I don’t know about grub… I might have messed something up yesterday when I was attempting to fix it…
I also have previously check the efibootmgr output which is :

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0002,0000,0001
Boot0000* manjaro       HD(1,GPT,91e9646f-77db-5e45-a9d2-b26c61b5195b,0x1000,0x96000)/File(\EFI\MANJARO\GRUBX64.EFI)
Boot0001* Hard Drive    BBS(HD,,0x0)/VenHw(5ce8128b-2cec-40f0-8372-80640e3dc858,0200)0000474f00004e4fcb000000010000008b005700440043002000570044005300310032003000470032004700300041002d00300030004a004800330030000000050109000200000000010416008b12e85cec2cf040837280640e3dc85802007fff040002010c00d041030a0000000001010600030101010600010003120a000000ffff00007fff040001043e00ef47642dc93ba041ac194d51d01b4ce63800310038003300340038003000380035003600390035002000200020002000200020002000200000007fff04000000424f00004e4fc3000000010000008b0053005400310030003000300044004d003000310030002d003200450050003100300032000000050109000200000000010416008b12e85cec2cf040837280640e3dc85802007fff040002010c00d041030a0000000001010600030101010600010003120a000100ffff00007fff040001043e00ef47642dc93ba041ac194d51d01b4ce62000200020002000200020002000200020002000200020004e005a00380031004e0054004300530000007fff04000000424f
Boot0002* UEFI OS       HD(1,GPT,91e9646f-77db-5e45-a9d2-b26c61b5195b,0x1000,0x96000)/File(\EFI\BOOT\BOOTX64.EFI)0000424f

I manually changed the BootOrder yesterday from 0000, 0002, 0001 to the one listed above to see if that would fix the issue…
The drive where my installation is located has the manjaro label…

What puzzles me is what is the UEFI OS entry? I guess that it should be the bootloader section…

dmt · 28 April 2023 13:38

Was it the initial issue you were dealing with at the time? Your current boot is 0000, so I’d try changing it back.

I think UEFI OS is a generic version, but I must admit I don’t know much about UEFI etc at this level.

Zeosleus · 28 April 2023 13:45

No, I changed the boot order, to see if I could boot up without having to select a boot device by F11 upon power-up…

I guess I will try to change the boot order to put 0000 first…