Install-grub: a new way to keep your EFI/MBR in-sync with grub package

[cscs]

Having 2 ESPs … and both occupied by manjaro boot files probably resulted in install-grub only running on the first one it found. I dont think its coded to ‘keep looking’ for more.

As to multiple ESPs, different disks, etc.

None of that really matters.

In general I suggest and use the one ESP for all systems on the same machine … because that just makes sense, and it works. (and works well for all forms of management … firmware updates, etc)

But if you want to have 2 … on the same disk … or different ones … you absolutely can.
I dont really see much benefit, aside from some edge scenarios like wishing to full-encrypt root/boot/all of linux and grub but not apply it to the win stuff, but you can.

[Lycan]

I got

$ sudo install-grub
WARNING: EFI directory not found! Grub couldn't be installed.

I have

$ sudo ls /boot/efi -Rl
[sudo] password for alex: 
/boot/efi:
total 4
drwx------ 4 root root 4096 jul 24  2020 EFI

/boot/efi/EFI:
total 8
drwx------ 2 root root 4096 jul 24  2020 boot
drwx------ 2 root root 4096 jul 24  2020 Manjaro

/boot/efi/EFI/boot:
total 132
-rwx------ 1 root root 135168 jul 29  2020 bootx64.efi

/boot/efi/EFI/Manjaro:
total 148
-rwx------ 1 root root 151552 dic 17  2022 grubx64.efi

and

$ sudo update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-6.6-x86_64
Found initrd image: /boot/amd-ucode.img /boot/initramfs-6.6-x86_64.img
Found initrd fallback image: /boot/initramfs-6.6-x86_64-fallback.img
Found linux image: /boot/vmlinuz-6.1-x86_64
Found initrd image: /boot/amd-ucode.img /boot/initramfs-6.1-x86_64.img
Found initrd fallback image: /boot/initramfs-6.1-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Adding boot menu entry for UEFI Firmware Settings ...
Found memtest86+ image: /boot/memtest86+/memtest.bin
/usr/bin/grub-probe: warning: unknown device type nvme0n1.
done

What is wrong with my system?

EDIT: I found

$ lsblk -rno parttypename,mountpoint
 /var/lib/snapd/snap/core/16091
 /var/lib/snapd/snap/core/16202
 /var/lib/snapd/snap/core20/2015
 /var/lib/snapd/snap/core20/2105
 
Linux\x20swap [SWAP]
Linux\x20filesystem /Data500
 
Linux\x20filesystem /home
 
BIOS\x20boot /boot/efi
Linux\x20filesystem /
 
Linux\x20filesystem /BackUp
Linux\x20filesystem /Data

So my system doesn’t match to the command

lsblk -rno parttypename,mountpoint | \
    awk -e '/EFI\\x20\(FAT-12\/16\/32\)|EFI\\x20System/ {print $2}' | tr -s '\n'

Changing the line to

lsblk -rno parttypename,mountpoint | \
    awk -e '/EFI\\x20\(FAT-12\/16\/32\)|EFI\\x20System|BIOS\\x20boot/ {print $2}' | tr -s '\n'

Solves my issues

$ sudo install-grub
[sudo] password for alex: 
Grub will be installed on: EFI
Installing for x86_64-efi platform.
Installation finished. No error reported.
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/manjaro/theme.txt
Found linux image: /boot/vmlinuz-6.6-x86_64
Found initrd image: /boot/amd-ucode.img /boot/initramfs-6.6-x86_64.img
Found initrd fallback image: /boot/initramfs-6.6-x86_64-fallback.img
Found linux image: /boot/vmlinuz-6.1-x86_64
Found initrd image: /boot/amd-ucode.img /boot/initramfs-6.1-x86_64.img
Found initrd fallback image: /boot/initramfs-6.1-x86_64-fallback.img
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Adding boot menu entry for UEFI Firmware Settings ...
Found memtest86+ image: /boot/memtest86+/memtest.bin
/usr/bin/grub-probe: warning: unknown device type nvme0n1.
done

[bedna]

@Lycan

So my system doesn’t match to the command

lsblk -rno parttypename,mountpoint | \
    awk -e '/EFI\\x20\(FAT-12\/16\/32\)|EFI\\x20System/ {print $2}' | tr -s '\n'

Changing the line to

lsblk -rno parttypename,mountpoint | \
    awk -e '/EFI\\x20\(FAT-12\/16\/32\)|EFI\\x20System|BIOS\\x20boot/ {print $2}' | tr -s '\n'

Solves my issues

I came to a similar conclusion yesterday when looking at the script. For efi, would it not be better just check fstab first instead?
IIRC there was a for loop that would not be great either, and the name of the ESP was designated with some program I do not even have on my computer.
Edit. nvm, the “program” is a function, I just checked it, I did not spend that much time on it yesterday. lol

config_item()
{
  for x in /etc/default/grub /etc/default/grub.d/*.cfg; do
    if [ -e "$x" ]; then
	# Lose any output here so we don't confuse our
	# caller. The xen packages echo stuff here, Aargh!
      . "$x" > /dev/null
    fi
  done
  if [ "$(eval echo "\${$1+set}")" = set ]; then
    eval echo "\$$1"
  else
    return
  fi
}

But again, our usecases might be considered “strange” or something, i dunno.

About using windows partition instead of 2 partitions. I mentioned I first used to share the boot partition, first update in windows and the boot menu (that can be brought up with F8 as @soundofthunder reminded me about) was removed. I did not want to deal with that over and over (I booted into windows daily back then). And after reinstallation with a separate partitions windows now only changes the bios order (sometimes) and gigabyte removes the ESP completely every firmware update. A grub-install fixes it, install-grub does not.

[Teo]

@cscs

Having 2 ESPs … and both occupied by manjaro boot files probably resulted in install-grub only running on the first one it found. I dont think its coded to ‘keep looking’ for more.

As i said, i am currently booting from the second ESP, and it works. As far as i understand, the script looks and takes the variables and paths from the currently booted system, which is the purpose. Have not checked if it does anything on the first ESP and i do not care, since i do not use its grub anymore, i keep the first ESP only for windows.

[soundofthunder]

@cscs

As to multiple ESPs, different disks, etc.
None of that really matters… But if you want to have 2 … on the same disk … or different ones … you absolutely can.
I dont really see much benefit, aside from

Aside from an element of redundancy, and ease of maintenance, if for example one opts to replace one OS with another, or wishes to remove an OS from service quickly/temporarily.

Having each OS on a separate disk (with its own ESP) is particularly beneficial in my case, as well; I use removable caddies, which allows me to swap several OS in a multiboot configuration, as needed – quite useful, when available disks (whether hosting an OS or not) outnumber available SATA ports.

If you don’t multiboot, then sure, I can understand how you wouldn’t see much benefit. Recent years have shown an increase in multiboot adoption (no stats, only observation), and I maintain the one-OS-per-disk principle as being the most convenient, as has been generally recommended for many years.

Before UEFI, multibooting was not as convenient, although still possible. It was common to adopt the many-OSs-on-one-disk approach; and, let’s face it, storage was never as cheap as is popularly claimed, so economically, that made a lot of sense.

Some people still prefer that approach with UEFI, and there’s nothing inherently wrong with that. In fact, the original UEFI standard actually supported @cscs one-ESP-per system; and this makes sense on typical installations.

Whether manjaro-install-grub should ultimately anticipate a multiboot scenario with multiple ESP’s, I can’t say. However, I can positively suggest it should be given due consideration.

(I don't think it absolutely *must* anticipate multiple ESPs; Idealy, the disk should only *need* one, despite many being possible, and Grub would detect the first ESP in the expected location.)

[Mirdarthos]

Just did a little test on this. It worked before this, so I was of the opinion it should again. I had an error:

Partition Table Entries is not in disk order

…or something. So, I followed these instructions:

It is very easy to fix partition name ordering with that most feared and deadly disk management tool called fdisk. Here is the summary. For details, see for e.g., here.

sudo fdisk /dev/sda

• enter x to enter expert mode
• enter f to fix partition order
• enter r to return to main menu
• enter w to write the partition table changes to disk

And rebooted…

_{The warning about grub wasn’t there, I improved and added it.}

…right into a grub rescue CLI.

So I booted with my Live environment, and entered a chroot environment and ran:

install-grub

And was confronted by:

WARNING: EFI directory not found! Grub couldn't be installed.

So I continued and reinstalled GRUB according to this page on the wiki, rebooted, and I could use my PC again.

Curios about the error, I tried again:

$ sudo install-grub                                                                                                                                                                                                                                   WARNING: EFI directory not found! Grub couldn't be installed.

This made absolutely no sense. So, I checked lsblk:

$ NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0   3.6T  0 disk
└─sda1        8:1    0   3.6T  0 part
sdb           8:16   0   4.5T  0 disk
└─sdb1        8:17   0   4.5T  0 part /home/mirdarthos/virtualbox
/home/mirdarthos/Video
/home/mirdarthos/Pictures
/home/mirdarthos/Music
/home/mirdarthos/KeePass
/home/mirdarthos/Documents
/mnt/5TB
nvme0n1     259:0    0 232.9G  0 disk
├─nvme0n1p1 259:1    0   500M  0 part /boot/efi
├─nvme0n1p2 259:2    0   7.8G  0 part [SWAP]
└─nvme0n1p3 259:3    0 224.6G  0 part /

…which looked fine. So I checked mount:

$ mount | grep efi
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,noatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

…and everything looked OK to me.

And so I checked:

$ sudo fdisk -l
[...]
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 960 EVO 250GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5053CB27-A627-1B45-9E5E-131BA81C39E6

Device            Start       End   Sectors   Size Type
/dev/nvme0n1p1     2048   1026047   1024000   500M Microsoft basic data
/dev/nvme0n1p2  1026048  17410047  16384000   7.8G EFI System
/dev/nvme0n1p3 17410048 488392031 470981984 224.6G Linux filesystem

…and the EFI partition seems to be another filesystem. (Or I’m still missing something.)

So I’m thinking in my attempt to fix the partition order warning, The partittion type changed. Or something.

But I’m booted into a working system at the moment and the warning is gone. So it would seem all worked out in the end.

And yes, I do realize that this isn’t the intended use of it, but I thought mentioning it anyway would be interesting. FWIW, this is my installed version:

$ pamac search -i install-grub                                                                                                                                                                                                                                install-grub  2.12-3.10                                                                                                                                                                                                                                  core
GNU Grub (2) Install Script on Updates

[soundofthunder]

@Mirdarthos

The partittion type changed. Or something.

It might be interesting to see how another tool (such as gdisk) reports that information: – sudo pacman -S gptfdisk

sudo gdisk -l /dev/nvme0n1

[Mirdarthos]

@soundofthunder

sudo gdisk -l /dev/nvme0n1

Here ya go:

$ sudo gdisk -l /dev/nvme0n1

GPT fdisk (gdisk) version 1.0.9.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/nvme0n1: 488397168 sectors, 232.9 GiB
Model: Samsung SSD 960 EVO 250GB               
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 5053CB27-A627-1B45-9E5E-131BA81C39E6
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 2048, last usable sector is 488397134
Partitions will be aligned on 2048-sector boundaries
Total free space is 5103 sectors (2.5 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1026047   500.0 MiB   0700  
   2         1026048        17410047   7.8 GiB     EF00  
   3        17410048       488392031   224.6 GiB   8300  

[soundofthunder]

Well, that’s distressing:

1. 0700 Microsoft basic data  (500.0 MiB) 
   --> Should be:
   --> EF00 (the ESP) EFI system partition
2. EF00 EFI system partition    (7.8 GiB) 
   --> Should be:
   --> 8200 (if swap) Linux swap
3. 8300 Linux filesystem      (224.6 GiB) 
   --> Is correct:
   --> 8300 (assumes) Linux filesystem

If these partition types were not assigned like this via Calamares, then it looks like fsck has seriously fornicated with your partition types.

Check partitions 1 and/or 2 for any sign of additional boot files.

The Microsoft basic data might indicate the partition was either created as, or detected as, a normal Fat32 filesystem, without the EF00 to specify otherwise.

I have seen a very recent post where someone claimed “Calamares didn’t create the ESP properly”, but it was largely dismissed as user-error by others (including moderators).

Perhaps this needs further investingation.

[Mirdarthos]

@soundofthunder

fornicated with your partition types

VERY WELL FILTERED. I’LL REMEMBER THAT.

@soundofthunder

If these partition types were not assigned like this via Calamares

They were like that after installation, IIRC. So it wasn’t calamares.

/dev/nvme0n1p2

…can’t be mounted, since it’s my (very rarely used) swap:

$ sudo mount -t auto /dev/nvme0n1p2 /tmp/tmp
mount: /tmp/tmp: unknown filesystem type 'swap'.
dmesg(1) may have more information after failed mount system call.

[soundofthunder]

@Mirdarthos

VERY WELL FILTERED. I’LL REMEMBER THAT.

Well, perhaps it was, before that.

[Mirdarthos]

@soundofthunder

The Microsoft basic data might indicate the partition was either created as, or detected as, a normal Fat32 filesystem, without the EF00 to specify otherwise.

Looks that way:

$ cat /etc/fstab                                                                                                                                                                                                                                      1 ↵
# <file system> <mount point> <type> <options> <dump> <pass>

UUID=D563-DAB7  /boot/efi   vfat    noatime,umask=0077  0   2
UUID=1b3f894f-5481-4241-ace5-c129a0cdb412   swap    swap    noatime 0   2
UUID=9a26c8d0-43f4-44ad-a7d3-861d6f6cdbfa   /   ext4    noatime,data=ordered    0   1

Edit:

You reckon I should change it? Like this:

https://unix.stackexchange.com/a/508894

[soundofthunder]

@Mirdarthos

You reckon I should change it? Like this:

I’ve never tried to change the partition type post-install, so I can’t be sure of the results. Gdisk might be able to change the type codes, no doubt there. The complication is that after that, you need to save those changes to the partition layout. I don’t know how Gdisk handles that on an already active disk. Fdisk, I’d be even less confident with.

@Mirdarthos

Boot to the Installer, use GParted to re-create and format the ESP, and Swap (for the sake of re-establishing the proper type), reboot with the Live Installer again, manjaro-chroot -a and reinstall Grub.

This seems the logical workaround.
Don’t forget to reassign the mount points.

[Mirdarthos]

In that case, and considering this is my only PC and it seems to be working ass it is at the moment, I’ll leave it. At least for now, until someone an give me more information.

Edit:

That’s what I figured as well. But it’ll have to wait a bit, seeing as my day is almost at an end, it works at the moment, and I’ll only suspend my PC, not shut it down, so in that regard, there is no problem.

In the mean time, if anybody else has a better plan, or idea, I’m all eyes.
_{Nope. Trust me, not ears.}

[andreas85]

1. Have a recent backup
2. Use a live manjaro
3. Use gparted !
4. Save all files from the efi-partition to usb (mount / save / unmount)
5. Recreate swap partition
6. Recreate efi partition (be aware of the flags)
7. Restore all the files to efi partition
8. Go into your uefi and select the right entry to boot

[soundofthunder]

@Mirdarthos

Nope. Trust me, not ears.

Your avatar suggests otherwise.

And, yes, if the boot files are ok, just zip the entire /EFI folder, save it somewhere, and restore it after reconditioning the ESP (and swap).

[Mirdarthos]

Just rebooted and everything seems still fine.

What I don’t get how or why is:

$ fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 960 EVO 250GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5053CB27-A627-1B45-9E5E-131BA81C39E6

Device            Start       End   Sectors   Size Type
/dev/nvme0n1p1     2048   1026047   1024000   500M Microsoft basic data
/dev/nvme0n1p2  1026048  17410047  16384000   7.8G EFI System
/dev/nvme0n1p3 17410048 488392031 470981984 224.6G Linux filesystem

$ swapon
NAME           TYPE      SIZE USED PRIO
/dev/nvme0n1p2 partition 7.8G   0B   -2
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0   3.6T  0 disk
└─sda1        8:1    0   3.6T  0 part /mnt/4tb_backup
sdb           8:16   0   4.5T  0 disk
└─sdb1        8:17   0   4.5T  0 part /home/mirdarthos/virtualbox
/home/mirdarthos/Video
/home/mirdarthos/Pictures
/home/mirdarthos/Music
/home/mirdarthos/KeePass
/home/mirdarthos/Documents
/mnt/5TB
nvme0n1     259:0    0 232.9G  0 disk
├─nvme0n1p1 259:1    0   500M  0 part /boot/efi
├─nvme0n1p2 259:2    0   7.8G  0 part [SWAP]
└─nvme0n1p3 259:3    0 224.6G  0 part /

With lsblk the type is swap but with fdisk not? And it seems neither with gdisk. I don’t get it. AT ALL!

And I always thought of fdisk better/easier than others…

If someone can explain it, THAT’D BE GREAT…

[andreas85]

@Mirdarthos

With lsblk the type is swap but with fdisk not? And it seems neither with gdisk. I don’t get it. AT ALL!

I only trust gparted !

[Mirdarthos]

@andreas85

I only trust gparted !

AHA!

That led me to the solution, thanks!

The /dev/nvme0n1p1 had the msfdata flag set, and the /dev/nvme0n1p2 had the boot and esp flags set!

Moved those to /dev/nvme0n1p1 and set the flag for /dev/nvme0n1p2 to swap, and all is well again:

$ fdisk -l /dev/nvme0n1                                                                                                                                                                                                                             Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 960 EVO 250GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5053CB27-A627-1B45-9E5E-131BA81C39E6

Device            Start       End   Sectors   Size Type
/dev/nvme0n1p1     2048   1026047   1024000   500M EFI System
/dev/nvme0n1p2  1026048  17410047  16384000   7.8G Linux swap
/dev/nvme0n1p3 17410048 488392031 470981984 224.6G Linux filesystem

[soundofthunder]

@andreas85

I only trust gparted !

@Mirdarthos

That led me to the solution, thanks!

You’re very welcome!

@soundofthunder

Boot to the Installer, use GParted to re-create and format the ESP, and Swap (for the sake of re-establishing the proper type)

@philm

Consideration:

(Added: 2024-01-13)

In a recent thread, it was revealed that Tuxedo are shipping their pre-installed Linux laptops with a non-standard (UEFI) boot configuration:

The ESP is mounted via a /boot mount point rather than /boot/efi, the more common convention. I’ve seen vague references to this an “upstream” preference, for many years, but rarely see it used in the wild.

Additionally, --bootloader-id=grub seems to be universally applied irrespective of the installed distribution, which results in a $ESP/EFI/grub/grubx64.efi bootloader path.

Grub should handle these variants without much fuss, and I presume the new script does, also. However, does it?

• Unable to boot with any kernels following update: No working init found