Ideas to unify ARM boot sequences and kernel namings

Hi,

As most guys have already known, currently Manjaro ARM support is quite fragmented, and for super-upstream guys like me, it’s not a super fancy experience.

Thus there are some ideas to try to unify the boot sequence and make multiple kernels to co-exist as peacefully as possible:

  • Unified (non-UEFI) boot sequence

Although there are tons of boards supported, but considering what we really care are Aarch64, the boot sequence is already unified to some extent.

So the most basic and common boot sequence would be:

ATF/Hardware firmware -> Uboot (hopefully upstream) -> Kernel (either upstream or vendor specific)

Boards like RK3399 will belongs to this, the most straight-forward one.
With upstream ATF, upstream Uboot, upstream kernel support.

Then boards like RPI can go this direction too, but Manjaro skips Uboot and boot into kernel directly.
This is a little problematic, as it doesn’t allow multiple kernel to co-exist, nor the ability to choose the kernel to boot at boot time.

This is partly due to lack of resource poured into packaging uboot-raspberry.

Another bad example is VIM3, which has its ATF closed source, and we even chain loaded another vendor specific Uboot.
At least we have a partially working Uboot, but still not ideal.

  • Kernel naming

Currently all vendor kernels conflict with upstream kernel.
For some boards, like RK3399 with upstream Uboot support, this is definitely not necessary.
As with Uboot, we can choose which kernel to boot at boot time, as long as you know how to load kernel/initramfs/fdt manually.

Thus my purpose here is, to name upstream kernel and initramfs as Image and initramfs-linux.img.
Then vendor specific kernel/initramfs to something like Image-rpi4 and initramfs-rpi4.img.

Of course, upstream and vendor kernels can’t use the same kernel suffix, or their modules path may conflict.
Thus vendor kernel needs to add something like -rpi to their kernel string.

But as long as we have Uboot (which is normally the first thing to be upstreamed), we could have multile kernels to co-exist without problem.

Even for the bad example, RPI4, I have purposed a newer version of uboot-raspberry to handle CM4/400 and is able to boot into upstream kernel without any problem.

  • Future SBSA (UEFI) boot sequence

UEFI firmware will be the future, and is already landing for RPI4 and some other server grade boards.

For those boards, it can’t be more straight-forward:

UEFI -> GRUB2/Systemd-boot/etc -> kernel

This means, we can skip the Uboot completely, and directly into UEFI bootloader.

But for certain boards, like RPI4, we can support both boot sequence by fully utilizing Uboot:

Regular device tree booting:
Firmware bootloader -> Uboot -> kernel

UEFI booting:
Firmware bootloader -> Uboot -> EDK2 UEFI -> GRUB2/systemd-boot/etc -> kernel

For now, I think we can make the UEFI firmware optional for RPI4 (already crafting a PKGBUILD), but still make it relying on Uboot to load.

But for the future, we should try our best to enable UEFI firmware by default for supported boards.
As it should provide a more user friendly interface than Uboot itself.

I’m all ears for feedback.
As I only have a small amount of boards running Manjaro ARM, namingly RockPi 4B (RK3399 based),
Khadas VIM3 (Amlogic based), CM4 (RPI based) and Xavier AGX (experimental UEFI firmware, Tegra based).

Thus I may miss some special boards which needs extra effort to boot.

2 Likes

BTW, although it looks like an extra step to boot into UEFI firmware, then to GRUB2, it is also what openSUSE did for its ARM flavor.

The point here is to provide a unified way for both boards without official UEFI firmware and server grade boards.

Thus I believe that’s the correct way to go.

1 Like

Unfortunately, the RPI_EFI firmware from edk2 is an ARM stub, which means it can’t be easily chain loaded by Uboot.

Thus I guess for RPI4, we have to skip Uboot if we want to use EFI firmware.

I like the idea of changing the kernel names for vendor specific kernels (even though we do try to get rid of them).

The main reason they are all using the same filename (Image) right now, is that we don’t have a working Grub setup yet. So all the bootscripts have “hardcoded” file paths for booting stuff. So if we had different names, and we wanted to test a vendor kernel, we would have to also change the bootscript whenever we did such a test. And since we had no real “multi-kernel” workflow yet, it made the most sense just to hardcode the filename into the bootscripts and don’t allow multiple kernels to be installed at once.

But I do know some people are working on getting Grub up and running.

And what boards would that be?
Be aware, that we don’t build uboot for amlogic boards, since it requires an x86_64 binary to sign/encrypt uboot. So we use the vendor provided ones there.

The main reason they are all using the same filename (Image) right now, is that we don’t have a working Grub setup yet.

No need for Grub, Uboot can already handle it, although less user-friendly.

Currently, upstream kernels all use /boot/extlinux/extlinux.conf to inform Uboot how to load the kernel/initramfs/fdt and pass boot options.

And that extlinux.conf can support multi-kernels without problem. The only problem is, we have to explicitly specify which is the default kernel.

When kernel fails to boot, one has either to change the extlinux.conf to other kernels (only possible if the file is on removable storage), or use TTL and use Uboot commands to manually load specific kernel/initramfs/fdt and kernel options.

Which is far from ideal for regular users. But on the other hand, end user shouldn’t need to bother how to fallback to another kernel. They should only need to choose which kernel to boot.

It’s our awesome testers ensuring the kernel won’t crash regular end users.

And what boards would that be?

All server grade boards, like HoneyComb:

Or Xavier AGX which already has experimental UEFI firmware.

Be aware, that we don’t build uboot for amlogic boards, since it requires an x86_64 binary to sign/encrypt uboot. So we use the vendor provided ones there.

I tried to go chain boot, but it seems that chain loaded Uboot won’t detect eMMC, which can be a problem.

For now, I have deprecated my VIM3 board, as its closed source ATF is just a crap.
Those guys doing the test in Khadas/Amlogic doesn’t even bother testing 64K page size, and cause random crash for 64K page size kernel.

But at least, the binary chainloaded Uboot can still work with extlinux.conf afaik.

1 Like

That’s true. And when we supply an image for people it should “just work”.
We can’t expect our users to manually change the extlinux.conf file to boot a different kernel.

Yeah. Those are not consumer grade boards. Manjaro ARM is focused on consumer boards, but I agree it can’t hurt to think about EFI boards, as more boards are bound to get support for it, even consumer grade ones.

In my opinion. What we need to have, before we can go around and change kernel file names, is something (like grub) to auto detect what’s currently present on the system and present it to the user in a nice list. If the user has to do any manual work in the boot sequence to boot other kernels, we have failed.

So my idea is this boot order:
BOOTROM/SPL → Uboot, which directs to Grub binary → Grub, which probes for existing kernels and presents a list to user → Kernel, which the user selected.
This would allow for multi-kernel setups and even make it possible for us to provide linux510, linux512 linux513 packages, like x86_64 does right now.

I mention grub, because that’s the bootloader I know of, that has auto detection of stuff. If something could autogenerate extlinux.conf files, that would be sweet, but I don’t know of any solution that does that.

2 Likes

We can’t expect our users to manually change the extlinux.conf file to boot a different kernel.

What if changing kernel is as simple as choosing one entry in systemd-boot/GRUB?

I forgot that Uboot can directly boot EFI programs, thus it can chainload UEFI bootloader.

This in fact provides a better thing: UEFI bootloader don’t need to load board specific FDT.
Uboot will just pass it, and since UEFI bootloader won’t overwrite the FDT in memory, the UEFI bootloader config doesn’t need to include board specific FDT.

By this, we finally could get a board independent boot sequence:

Non UEFI firmware -> U-boot (load board specific FDT if needed)
                           \
                            -> UEFI bootloader (GRUB/systemd-boot) -> kernel
                           /
Sever grade UEFI firmware

How about this benefit introduced by UEFI?

Wouldn’t this require Uboot to be compiled with EFI support for the board?

Wouldn’t this require Uboot to be compiled with EFI support for the board?

Most board already has that ability to execute EFI format, since most kernel has EFI_STUB built in by default.

Although during my test, Uboot can’t emulate a full UEFI environment without file ubootefi.var.
But even without that, Uboot can still load systemd-boot without problem, and load kernel safely too.

Just tried that, and user can even choose which kernel to boot! As the uboot-rapi has HDMI enable!

I’ll try to take a photo of that choice screen.

And does systemd-boot probe for available kernels, or does it have to be defined somewhere?

And does systemd-boot probe for available kernels, or does it have to be defined somewhere?

Needs manual defining, but should be pretty simple, at least to me, much simpler than grub probing:

https://wiki.archlinux.org/title/systemd-boot

One of the advantage here is, we can add systemd-boot entry for each kernel, thus no need to probe at all, and allowing user to choose which kernel to boot.
Just put something like /boot/loader/entries/linux-rpi4-mainline.conf for linux-rpi4-mainline and /boot/loader/entries/linux.conf for upstream linux package.

Let me re-compile the linux-rpi4-mainline with my uboot-raspberry dependency and test systemd-boot to choose kernels.
As I just want to change the eeprom for my CM4, but upstream doesn’t support the rpi user space tool.

And IIRC for UEFI boot, the Arch install ISO uses systemd-boot by default.

Sure, but as I said already. We can’t expect regular users to make these kinds of changes, when they want to try other kernels. So it needs to be generated automatically.

But yeah. If the kernel packages themselves add a loader entry for itself, it should be doable.

Do you know which config options are needed from the kernel to do this and how to get uboot to start systemd-boot?

Do you know which config options are needed from the kernel to do this

It’s already in upstream kernel.
And quite some other vendor kernels also have them compiled in already.

The configs are

CONFIG_EFI=y
CONFIG_EFI_STUB=y

and how to get uboot to start systemd-boot?

We can go uboot script, but I want to make it the fallback method if we can’t find anything better.

I see the uboot has some command compiled in to load EFI from /boot/efi/boot/bootaa64.efi, which is exactly the same file installed by systemd-boot.

But I haven’t yet found a solid doc on the boot sequence, I’ll need to do more test to see what’s needed to make it to boot EFI executable by default.
(I hope removing extlinux directory would do the trick, but needs more testing)

Great. :slight_smile:

Looking forward to your solution.

BTW, currently the most time consuming part is recompiling both linux and linux-rpi4-mainline to get an environment with two kernels to select.

If you guys can use my diff to build test packages, it would save me quite some time.

You would only have to compile them both once, to get a test environment.
We also compile “on device”, so it takes us a bit of time as well. Plus, we can’t add them to the repo, since the package names will be the same as existing ones.

You would only have to compile them both once, to get a test environment.

That would only be true if the modified linux-rpi4-mainline kernel has enough change to avoid any file conflicts…
Or even with distcc, I still need to wait for one hour to build it.

At least the upstream version linux has been recompiled and booted.

Furthermore, it looks like at least the rpi uboot has the following boot sequence:

- scan_dev_for_extlinux
  The current one for "extlinux.confg".

- scan_dev_for_scripts
  The old uboot script, at least now we don't use that

- scan_dev_for_efi
  What I want.

Thus it just means, if I removed extlinux.conf, it would be enough to load EFI.

Just found this documentation on uboot.

It states that:

Since most of our devices are big.little endian, EFI can’t be built for most of our devices. :frowning:
Unless I misinterpret that statement.

And I don’t see

CONFIG_CMD_BOOTEFI=y
CONFIG_EFI_LOADER=y

in any of the uboot defconfigs for our supported boards. Not even the raspberry pi.

Since most of our devices are big.little endian

You got it completely wrong. Little endian is the default endian for x86, and most distros go little endian even for arches which support both big and little endian.

Thus it’s not a problem at all, because no one really use big endian at all.

I guess you get confused with BIG.little cores layout, which is completely unrelated to memory endian.

in any of the uboot defconfigs for our supported boards. Not even the raspberry pi.

Because those are Uboot default for almost all boards already.

Just compile one Uboot, and check the .config:

$ pwd
/home/adam/uboot-raspberrypi/src/u-boot-2021.04
$ grep EFI .config
...
CONFIG_EFI_PARTITION=y
CONFIG_EFI_PARTITION_ENTRIES_NUMBERS=128
CONFIG_EFI_PARTITION_ENTRIES_OFF=0
CONFIG_EFI_LOADER=y
CONFIG_CMD_BOOTEFI_BOOTMGR=y
CONFIG_EFI_VARIABLE_FILE_STORE=y
...

Trust me, EFI is more popular than you thought.

So what you are saying is, that EFI support should already be in the uboots we build ourselves?