Boot manjaro ISO from encrypted drive

[tial]

Before I get to the topic: I am new here, thanks for having me. I have been looking for help in the forum for some time, but always found my answers without creating a new topic. Thanks to everyone, who is doing an amazing job on this forum!!
That said, I am stuck with something, that is out of my understanding and ability (so far).
The situation: I was trying to follow this HowTo
https://forum.manjaro.org/t/howto-boot-manjaro-iso-directly-with-grub/15892/44
on how to boot manjaro ISO directly with GRUB. Why: I am in a very remote area, limeted access to working computers, internet and even electricity. So I really depend on my machine working properly. But s**t happens sometimes, and one might need a live system to repair things. With flash drives being rare and no other machine to flash a drive either, I thought: This guide is what I need!
The Problem: I have my system luks encrypted. That seems to make the whole thing alot more complicated. If I understand it correctly, my system has full disk encryption (even boot), because I have to enter my passphrase, before the GRUB menu shows up (do I understand that right?). I found another related topic about booting an ISO from an encrypted system, but I wasn’t able to resolve the problem so far with its help:
https://forum.manjaro.org/t/iso-image-on-luks-encrypted-partition/74426
I am not sure, if it makes a difference, if just /root or /root and /boot are encrypted.
What I tried so far: In the beginning I just followed the HowTo step by step, and everything worked, until I wanted to start the ISO from GRUB menu. A bunch of errors showed up. So I did some more research. I tried to follow the advice in the second linked topic to create a non encrypted partition just for this purpose. For that I had to resize my root partition (scary thing for me), and of course ruined my working system. . But I managed to resolve that, with the help of this forum. Now I can normally boot again, have a separate unencrypted partition, created a folder on it and copied the ISO to it. But so far I wasn’t able to get it running. I tried a few other things from the archwiki, ubuntuusers etc… In the end, I always end up with (different) error messages, when I select the entry in GRUB.
My guess: My GRUB configuration is still wrong, since I don’t really understand what I am doing and/or GRUB has somehow no access to the unencrypted partition, because it operates in an encrypted environment. When I list the partitions in the GRUB command line I see that it shows “no known filesystem detected” for all partitions except from boot(?).
At the moment my /boot/grub/custom.cfg looks like this (but I have tried a hundred different versions…):

menuentry "Manjaro  grub_iso"  {
    set isofile="/miso/manjaro.iso"
    set dri="free"
    set lang="en_US"
    set keytable="de"
    set timezone="Europe/Berlin"
    search --no-floppy -f --set=root $isofile
    probe -u $root --set=abc
    set pqr="/dev/disk/by-uuid/$abc"
    loopback loop(hd0,gpt4) $isofile
    linux  (loop)/boot/vmlinuz-x86_64  img_dev=$pqr img_loop=$isofile driver=$dri tz=$timezone lang=$lang keytable=$keytable
    initrd  (loop)/boot/intel_ucode.img (loop)/boot/initramfs-x86_64.img
}

output of lsblk is this:

NAME FSTYPE FSVER LABEL          UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda1 vfat   FAT32 NO_LABEL       60B5-8A6A                             295,2M     1% /boot/efi
└─sda
                                                                                     
sda4 ext4   1.0   unencryptediso e82df732-e38d-4a3a-bfb8-c78d2d2dea2f                
└─sda
                                                                                     
luks-ffc0a8e3-431f-4e00-a614-fb14f0ba1b0f
     ext4   1.0                  36fb0418-e109-41e5-b442-74fba7f9f99d   10,2G    85% /
└─sda2
     crypto 1                    ffc0a8e3-431f-4e00-a614-fb14f0ba1b0f                
  └─sda
                                                                                     
luks-3646510a-e376-4eb9-97a5-feec2aec0623
     swap   1     swap           7a294832-8fb9-4e39-a5fb-64252dface16                [SWAP]
└─sda3
     crypto 1                    3646510a-e376-4eb9-97a5-feec2aec0623                
  └─sda

The manjaro ISO is located on sda4 in the folder /miso/manjaro.iso
GRUB is saying “no such device: /miso/manjaro.iso”, “unknown filesystem”, “no server is specified” and other things, depending which version of my not working configutation I am using
I can’t really figure out, how I tell GRUB, were to find the ISO, because I don’t really get if I have to edit the $isofile, $root, $pqr or even a different variable. Any help or pointers in the right direction are appreciated. Apart from solving the problem, it would be great to understand, what I am actually doing…
Thanks in advance!
Tim

[linux-aarhus]

You cannot - perhaps more correct, should not - do that with an encrypted root.

What you want to achieve can only be done using un-encrypted boot with extra space for the iso.

This topic explains how to achieve this [root tip] [How To] Use Calamares to install encrypted root using unencrypted boot.

Please be aware that you need to increase the size of the boot partition to be big enough or use an extra partition to store the ISO file.

If you decide to use /boot - depending on the size of the ISO - must be 4GiB as minimum.

If the inside of your luks container is tomb-stoned for any reason you have a completely unusable system - if you have the ISO outside on a separate partition you have more flexibility.

[Nachlese]

As I understand it how booting a fully encrypted system with Grub works,
you are asked a passphrase and only after that you get to see the menu to choose which kernel and which system you want to boot.

In that thread from 2021 you “linked” to, I have also made some comments which seem pertinent here:

in particular, what I noticed so far, you need to adapt these two lines to your actual situation:

set isofile="/miso/manjaro.iso"
set pqr="/dev/disk/by-uuid/$abc"

The directory /miso/manjaro.iso should probably be at this:
/dev/disk/by-uuid/$abc
device - of course “abc” is not a valid UUID but just a placeholder

I have not tried this myself - I keep a USB drive with one or more iso’s that I can use in case I need to access the (encrypted) system.
Further: my system is not fully encrypted - I have a separate and unencrypted /boot partition - like @linux-aarhus said.
So this would likely be easy to set up for me, but I don’t know whether it is possible with a fully encrypted system like yours is.
See first sentence.

[bedna]

I use ventoy nowdays, but I have these notes about making custom menus that I used a while ago, maybe you can find some use for them.

Just examples of a few iso files.
You might want to change the inird to intel instead of amd depending on your system, or maybe it should be removed since microcode is now a hook in mkinitcpio.conf, I have not tried these since.
Also, make sure you either use free or nonfree with the manjaro iso depending if you have nvidia or not.
I use non free even though I have nvidia because it boots WAY faster, but if I were to install from the iso and not just use it as a live version of manjaro, I would start it with nonfree.
ISO files are located on a separate ext4 partition with the iso files inside a directory called /miso

Create /etc/grub.d/40_custom containing:

menuentry "Manjaro-live linux66 ISO" {
    set isofile="/miso/manjaro-kde-23.1.0-231215-linux66.iso"
    #set dri="nonfree"
    set dri="free"
    set lang="en_US"
    set keytable="se"
    set timezone="Europe/Stockholm"
    search --no-floppy -f --set=root $isofile
    probe -u $root --set=abc
    set pqr="/dev/disk/by-uuid/$abc"
    loopback loop $isofile
    linux (loop)/boot/vmlinuz-x86_64 img_dev=$pqr img_loop=$isofile driver=$dri tz=$timezone lang=$lang keytable=$keytable
    initrd (loop)/boot/amd_ucode.img (loop)/boot/initramfs-x86_64.img
}

menuentry "Debian-live ISO" {
    set isofile="/miso/debian-live-12.0.0-amd64-standard.iso"
    search --no-floppy -f --set=root $isofile
    loopback loop $isofile
    linux (loop)/live/vmlinuz boot=live components splash findiso=$isofile
    initrd (loop)/live/initrd.img
}

menuentry "CloneZilla-live ISO" {
   set isofile="/miso/clonezilla-live-3.1.1-18-amd64.iso"
   search --no-floppy -f --set=root $isofile
   loopback loop $isofile
   linux (loop)/live/vmlinuz boot=live components config findiso=$isofile ip=frommedia toram=filesystem.squashfs union=overlay copytoram
   initrd (loop)/live/initrd.img
}

Then run sudo update-grub

Then again, it might be completely different since you encrypt.

This might give you some ideas of how to do it, I used this and info on other sites to achieve above configs.

Side note, DO NOT EDIT /boot/grub/custom.cfg, edit /etc/default/grub and then run sudo update grub.

Good luck!

Edit
While you are at it, maybe you want to add these too:

/etc/grub.d/91_reboot

#!/bin/sh

echo "Adding reboot option." >&2

cat << EOF
menuentry 'Reboot' --class tool --class restart --id reboot {
        reboot
}
EOF

/etc/grub.d/92_poweroff

#!/bin/sh

echo "Adding poweroff option." >&2

cat << EOF
menuentry 'Poweroff' --class tool --class shutdown --id poweroff {
        halt
}
EOF

sudo chmod +x /etc/grub.d/{91_reboot,92_poweroff}
sudo update-grub

[tial]

Wow, you guys are quick!

@linux-aarhus

You cannot - perhaps more correct, should not - do that with an encrypted root.

Hm, ok. Thats sad, so all efforts in vain?
Could you elaborate for a newbie, why I should not? Possible dangerous effects? I guess you meant with an “encrypted boot” instead of “encrypted root”, right?

I quickly checked your linked HowTo, but as far as I can tell, these steps only work for a fresh install, right? That is no option for the moment…

@linux-aarhus

If you decide to use /boot - depending on the size of the ISO - must be 4GiB as minimum.

If the inside of your luks container is tomb-stoned for any reason you have a completely unusable system - if you have the ISO outside on a separate partition you have more flexibility.

Like I tried to explain, I have already successfully created an UNencrypted partition just for the ISO file (so outside the luks container?!). Does that change your sentence above, or is it still “impossible/undesirable”?

@Nachlese

As I understand it how booting a fully encrypted system with Grub works,
you are asked a passphrase and only after that you get to see the menu to choose which kernel and which system you want to boot.

That is my understanding too. I get to enter my password right after pressing the power button, then the slot is opened and I get to see the GRUB menu.
I was actually hoping you would answer here too, because I saw your answer in that thread, but the poster of it gave up so fast without success.
Maybe you can help me to understand, what each of these variables has to point to, so I understand how to modify them. Since in the original post everything is on one partition, and in every tutorial I found too, I am just stuck with that.
If I understand it correctly, $abc is not a placeholder but a variable defined in the line above it.

    probe -u $root --set=abc
    set pqr="/dev/disk/by-uuid/$abc"

[bedna]

@tial

Maybe you can help me to understand, what each of these variables has to point to, so I understand how to modify them. Since in the original post everything is on one partition, and in every tutorial I found too, I am just stuck with that.
If I understand it correctly, $abc is not a placeholder but a variable defined in the line above it.

    probe -u $root --set=abc
    set pqr="/dev/disk/by-uuid/$abc"

As you can see in the post I did above, all you have to point to is the file itself, grub scans with the line search --no-floppy -f --set=root $isofile and then it probes it with probe -u $root --set=abc and finally defines everything with set pqr="/dev/disk/by-uuid/$abc"

If I understand what others has said in this thread, and you have the iso file on a separate unencrypted partition inside a folder called /miso (and you edit the file name to the iso you have) you can just copy paste the whole manjaro menu section.

[tial]

Hi @bedna
sorry, when I replied earlier, I didn’t get the chance yet, to go trough your first answer.
So what I understand from your first post is the following:
Your example Manjaro entry is working just like that , even though the ISO file is located on a different partition that root, without the partition being specified in the entry?! You only specified the path and filename on the partition, not the partition itself. Do I see that correctly?
I have tried that first (because thats basically what the linked HowTo says), but that doesn’t work in my case, like I elaborated in the first post. That is why I tried to somehow tell Grub, at which partition to look at (like other examples suggest), but also without success. I am not sure though, where exactly I should specify the partition and in which style (e.g. GRUB won’t understand sda4, because that is just how it will be called by the OS, if I understand it right).

@bedna

Side note, DO NOT EDIT /boot/grub/custom.cfg, edit /etc/default/grub and then run sudo update grub.

Could you explain why? The linked HowTo says:

@Chrysostomus

Create the file /boot/grub/custom.cfg or /etc/grub.d/40_custom file with the following contents:

Thanks for the other suggestions, but for now I will deal with the problem at hand.
To your last post:

@bedna

If I understand what others has said in this thread, and you have the iso file on a separate unencrypted partition inside a folder called /miso (and you edit the file name to the iso you have) you can just copy paste the whole manjaro menu section.

Like written before, that is exactly what I tried in the first place, without any success.

Honestly I am not sure right now, if I am not just wasting my (and your) time, since @linux-aarhus wrote, it is not the way to go anyway (if even possible), even though I would still like to understand why!

But since I have been obsessed with this idea the last one week, I won’t give up so easily. I had another idea: Maybe I could also try to install another GRUB in the unencrypted partition, that has a menu entry to start the ISO in the (same) unencrypted partition or otherwise links to the bootloader for the encrypted system. So far, I have no idea, if that is possible, but I will try to find out…

[linux-aarhus]

The only reason I suggest the approach of having an un-encrypted boot with the space to hold the ISO or a separate partition holding the ISO is the availability when your encrypted luks parttion fails you.

If you can make it work - while holding the ISO inside the encrypted partition - go for it …

The dominant reason to use /boot/grub/custom.cfg is the fact that it will never break your boot experience - even if you make a typo.

The grub configuration $abc is a place holder for the result from the probing command - that means the result from the probe will always be correct - it is never a placeholder where you need to include an UUID

[bedna]

@tial

Your example Manjaro entry is working just like that , even though the ISO file is located on a different partition that root, without the partition being specified in the entry?! You only specified the path and filename on the partition, not the partition itself. Do I see that correctly?

Yes.
I do not know exactly how it works, but I suspect it is similar to how os-prober works, it tries to mount every single partition and checks for stuff.
That is why, you might have read that it is not recommended (and disabled as default in grub) to have os-prober active all the time because it might pose a security risk.

@tial

I have tried that first (because thats basically what the linked HowTo says), but that doesn’t work in my case

I know there are ways you can use the old method of defining hd1 hd2 etc, but that is outdated afaik.
I will update my menus and download the latest iso and test, maybe it no longer works.
My menus are still in my grub, i just never use them (and the iso file is very old as you can see), ventoy is just so much more convenient.

@tial

Could you explain why?

Because the next time you run sudo pacman -Syu and it contains stuff that is lets say kernel related, so pacman runs the hook for mkinitcpio -P. Then it will also do an update-grub. And update-grub will look in your /etc/default/grub and then overwrite the changes in /boot/grub/custom.cfg. The files in /etc/grub.d/ does not get overwritten, and those files gets sourced into the grub menu.

Update
So @tial
I just added the latest KDE image to my partition inside the /miso directory, edited my 40_custom, ran update-grub, rebooted, selected the iso in the grub menu, and it works.

I have actually never gotten to see the actual tty chatter before (I think a bios update made this happen), but this time I did and it did exactly what I thought, it starts from the top mounting every single partition looking for /miso/manjaro-kde-23.1.3-240113-linux66.iso and when it finally found it (it is on my second last partition out of a LOT), it booted from it into a live session.

So it might be your drives being encrypted that throws an error and therefore fails before even finding the mount. And from what I saw from the chatter on tty it DID use the hd1 hd2 etc when mounting, so maybe that is the way you should go after all?
I have one ntfs partition, and it did not complain about that, so idk.

Since I have extremely limited knowledge about encryption, this is all I can help with, sorry…

Have you tried filling in the actual uuid instead of using the variables?

Good luck, I’m sure you’ll figure it out!

[tial]

@linux-aarhus

The only reason I suggest the approach of having an un-encrypted boot with the space to hold the ISO or a separate partition holding the ISO is the availability when your encrypted luks parttion fails you.

Thanks for clarifying that! That is very understandable, but a fresh install is no option for the moment.

@linux-aarhus

If you can make it work - while holding the ISO inside the encrypted partition - go for it …

I feel like, we are misunderstanding each other here. I do NOT have the iso inside the encrypted partition, since that didn’t seem to work. But even being on the separate unencrypted partition, grub seems unable to “detect” it. My guess is still, that it is looking from the perspective of the encrypted boot partition and therefore somehow not able to see/read/access the unencrypted partition. But that is really nothing more than my guess. I am fare out of my league here, and I should probably stop messing with things I don’t understand

@linux-aarhus

The dominant reason to use /boot/grub/custom.cfg is the fact that it will never break your boot experience - even if you make a typo.

@bedna

And update-grub will look in your /etc/default/grub and then overwrite the changes in /boot/grub/custom.cfg.

Both makes sense. I guess, I will keep working with the custom.cfg until I have a working setup and then copy it to the other file.

@linux-aarhus

The grub configuration $abc is a place holder for the result from the probing command - that means the result from the probe will always be correct - it is never a placeholder where you need to include an UUID

That’s exactly what I guessed, meaning I don’t have to mess with that one.

@bedna

So it might be your drives being encrypted that throws an error and therefore fails before even finding the mount. And from what I saw from the chatter on tty it DID use the hd1 hd2 etc when mounting, so maybe that is the way you should go after all?
I have one ntfs partition, and it did not complain about that, so idk.

Since I have extremely limited knowledge about encryption, this is all I can help with, sorry…

Have you tried filling in the actual uuid instead of using the variables?

Thanks for taking the time to try it out!!
I guess, than it has to be either the encrytion (likely) or some other thing that is messed up with my machine (unlikely since everything else works flawlessly).
I have not yet tried the uuid, because I am not sure where I should put it…

[bedna]

@tial

Both makes sense. I guess, I will keep working with the custom.cfg until I have a working setup and then copy it to the other file.

That will never work, since every time you run update-grub it will get overwritten and the iso will never be shown in your grub menu.
Use the way I gave you in my initial post and put that inside /etc/grub.d/40_custom then sudo update-grub

@tial

I guess, than it has to be either the encrytion (likely) or some other thing that is messed up with my machine (unlikely since everything else works flawlessly).

No, it most likely have to do with you editing grub.cfg and then running update-grub after.

Anyway, om off, this is unusal for me, but you were such a nice person.
You’ll get it working!

[tial]

I am not running update-grup since the original HowTo discourages that!
Like I wrote before, adding the entry to the menu works. It just won’t start the ISO. Read my previous post again, and the answer from @linux-aarhus concerning the use of custom.cfg… Or am I missing something here?!

@bedna

Anyway, om off, this is unusal for me, but you were such a nice person.
You’ll get it working!

Thanks again for your help!

[dmt]

@bedna

That will never work, since every time you run update-grub it will get overwritten and the iso will never be shown in your grub menu.

You’re thinking of grub.cfg. Using custom.cfg is fine.

@tial

Like I wrote before, adding the entry to the menu works. It just won’t start the ISO.

What’s the content of the file?

[tial]

The content of the ISO file? its just a normal manjaro live system… It tried to explain it in my opening post, but maybe I wasn’t clear about that, sorry.

[dmt]

Sorry, I meant /boot/grub/custom.cfg or /etc/grub.d/40_custom, whichever you’re using.

[tial]

The current (not working) version of my custom.cfg is quoted in the very first post as well. I have tried also an exact copy of the HowTo (of course with edited filename) and some other versions, but so far no success.

[dmt]

Ok, I just expected it to have changed by now.

I’ve modified my working version with the details from yours, don’t change anything. Make sure the ISO is really called manjaro.iso, and is in the correct path.

menuentry "Manjaro grub_iso"  {
        set isofile="/miso/manjaro.iso"
        set dri="free"
        set lang="en_US"
        set keytable="de"
        set timezone="Europe/Berlin"
        search --no-floppy -f --set=root $isofile
        probe -u $root --set=abc
        set pqr="/dev/disk/by-uuid/$abc"
        loopback loop $isofile
        linux  (loop)/boot/vmlinuz-x86_64  img_dev=$pqr img_loop=$isofile driver=$dri tz=$timezone lang=$lang keytable=$keytable copytoram
        initrd  (loop)/boot/intel_ucode.img (loop)/boot/initramfs-x86_64.img
}

@bedna is right, it does search the partitions for the path, so you don’t need to tell it which partition to use (at least assuming only one has a /miso).

[Nachlese]

@dmt

linux (loop)/boot/vmlinuz-x86_64 img_dev=$pqr img_loop=$isofile driver=$dri tz=$timezone lang=$lang keytable=$keytable copytoram

esp the first item:

@dmt

linux (loop)/boot/vmlinuz-x86_64

When I boot my ISO, there is no file or symlink by that name.
Only a file with the actual name - for example, for the particular ISO I have here:
vmlinuz-6.6-x86_64

Is the xxx_ucode.img actually needed?

@tial

and removed cryptoram

it was “copytoram” - but no idea whether that can just be ommitted …

[tial]

I just changed to intel_ucode.img and removed cryptoram.
Get the error again…

[dmt]

@tial

Get the error again…

Perhaps I missed it…what’s the error message?

@Nachlese

@Nachlese

When I boot my ISO, there is no file or symlink by that name.
Only a file with the actual name - for example, for the particular ISO I have here:
vmlinuz-6.6-x86_64

Well I just followed the tutorial that was linked, it was some time ago though.

@Nachlese

Is the xxx_ucode.img actually needed?

I don’t think so.