[HowTo] Install full Manjaro on ZFS filesystem

Difficulty: ★★★★☆

Background

Below you can find a comprehensive tutorial on how to install Manjaro on ZFS collected from various sources into one. Although Manjaro-architect supports installing system on ZFS, such tool cannot find the existing zfs pool for system installation. The below instruction involves manual installation, i.e. installing in the Arch way.

0. Preparation

Download the latest Manjaro ISO (the edition of your choice) from the Manjaro website and burn it to a bootable Live Media. This tutorial was tested using the latest KDE ISO as it has ZFS modules installed in the Live environment.
Check Burn an ISO File - Manjaro for more information.

1. Boot into Live environment

Configure your BIOS/UEFI to allow your computer boot with your bootable Manjaro Live Media
Check Booting the Live environment section for more information.

2. Prepare your partitions

Using UEFI, partition the system disk into three parts:

There are many ways to create a GUID partition table (GPT).

  • Use [gdisk](GPT fdisk - ArchWiki for more information) to prepare the partitions . Make sure to make the ESP as type EF00 or EFI system partition, the swap partition 8200 or Linux swap, and the zfs partition BF00 or Solaris Root.
  • Or use your favourity GUI Partition Manager.

3. Create zfs pools and datasets

ZFS uses virtual devices so it can’t be used as raw partitions. This is the most important part. The creation options vary depending on which bootloader (GRUB or systemd-boot) you’re going to use.

First, ensure the zfs module is loaded:

sudo modprobe zfs

Find the disk ID of your ZFS (Solaris Root) partition

ls -lh /dev/disk/by-id

The output should look like this:

nvme-Force_MP600_21038229000128553384-part3 → …/…/nvme0n1p3

So in this case, the nvme-Force_MP600_21038229000128553384-part3 part will be the disk ID you’ll use later.

Create the zfs pool with sudo if you don’t use GRUB.

sudo zpool create -f -o ashift=12           \
                  -O acltype=posixacl       \
                  -O relatime=on            \
                  -O xattr=sa               \
                  -O dnodesize=legacy       \
                  -O normalization=formD    \
                  -O mountpoint=none        \
                  -O canmount=off           \
                  -O devices=off            \
                  -R /mnt                   \
                  -O compression=lz4        \
                  <zfs-pool-name> /dev/disk/by-id/<id-to-partition-partx>

If you use GRUB as your bootloader, you must create the zfs pool with GRUB compatible option., oherwise, grub-install will fail. The details of the zpool create option can also be found here.

If you have other disks that want to be used as ZFS, create the zfs pool with them similarly. It is noted that if other disks have only one partition, then you can specify the whole disk (i.e. nvme-Force_MP600_21038229000128553384 instead of nvme-Force_MP600_21038229000128553384-part3) to create ZFS pool. zpool will rearrange the partitions itself.

It is recommended to create datasets under your zfs pool to mount your system.

Create zfs dataset for root partition with sudo:

sudo zfs create -o mountpoint=none <zfs-pool-name>/manjaro
sudo zfs create -o mountpoint=/ -o canmount=noauto <zfs-pool-name>/manjaro/root

You can create the datasets hierarchy by yourself. This is only to demonstrate the potential of zfs datasets structure. Remember the option -o canmount=noauto is a must for dataset mounted at /, and the option -o canmount=off for the datasets that are mounted at /var and /var/lib.

See Install Arch Linux on ZFS - ArchWiki for more information.

You can also create other datasets for different mountpoints (e.g. /home, or even specific users download /home/username/Downloads).

Finally, export the zfs pool and then import them again with sudo. Do not skip this or you’ll have problems importing them later.

sudo zpool export <zfs-pool-name>
sudo zpool import -d /dev/disk/by-id -R /mnt <zfs-pool-name> -N

With the argument -R /mnt, when you mount your zfs dataset for the system later, it will automatically use /mnt as the root mountpoint and mount to the relative path under /mnt (e.g. if you set the mountpoint of the dataset to /home, then it will be mounted at /mnt/home).

4. Mount and configure the root filesystem

Mount the zfs datasets with sudo zfs mount, for example:

sudo zfs mount <zfs-pool-name>/<zfs-dataset-for-root>

This should mount the root dataset (in the above example, <zfs-pool-name>/manjaro/root) to /mnt. You can mount other zfs datasets if there is any. Finally, mount your ESP.
If you use GRUB, mount it at /mnt/boot/efi by

sudo mkdir -p /mnt/boot/efi
sudo mount <your-ESP> /mnt/boot/efi

If you use systemd-boot, then mount it at /mnt/boot by

sudo mkdir /mnt/boot
sudo mount <your-ESP> /mnt/boot

Set the bootfs property on the descendant root filesystem with sudo so the boot loader knows where to find the operating system.

sudo zpool set bootfs=<zfs-pool-name>/<zfs-dataset-for-root> <zfs-pool-name>
sudo zpool set cachefile=/etc/zfs/zpool.cache <zfs-pool-name>
sudo mkdir -p /mnt/etc/zfs
sudo cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

If you have many zfs pools, make sure to set cachefile property for all of them before copying the zpool.cache file, so that they can be mounted automatically during the boot process.

Now, your system is ready for installation.

5. Install the full Manjaro using basestrap and Manjaro official iso-profile

If you like default settings of the official editions from Manjaro team, then this step is for you:

First, we need to clone the latest iso-profile from Manjaro.

git clone https://gitlab.manjaro.org/profiles-and-settings/iso-profiles.git ~/iso-profiles

And under the folder, you’ll find the iso-profiles for various editions. Let’s say we would like to install the KDE edition, then we can go to ~/iso-profiles/manjaro/kde, then you should find the files structure like this:

desktop-overlay
live-overlay -> ../../shared/manjaro/live-overlay
Packages-Desktop
Packages-Live -> ../../shared/Packages-Live
Packages-Mhwd -> ../../shared/Packages-Mhwd
Packages-Root -> ../../shared/Packages-Root
profile.conf

The most important items are Packages-Desktop, Packages-Root, and the folder desktop-overlay. The two Packages-* files indicate what packages will be installed, while the files in desktop-overlay are the settings that Manjaro team tweak for the edition. It is noted that the KERNEL packages from Packages-Root is the Linux kernel that the users should specify themselves (e.g. linux510).

Use whatever methods to copy all the packages from those two Packages-* files, and install them to the candidate system. Remember to check if KERNEL-zfs (i.e. linux510-zfs) and zfs-utils are included for zfs support. It is noted that KERNEL-zfs should be used instead of zfs-dkms to prevent future breakage due to invalid zfs kernel module.

sudo basestrap /mnt <all-needed-Packages>

Once the installation finishes, copy everything from the desktop-overlay folder to /mnt to apply the settings from Manjaro team.

After that, we need to chroot into /mnt to configure the system.

sudo manjaro-chroot /mnt /bin/bash

Once we’re in the chroot environment, we first edit the /etc/fstab file for automatically mounting non-zfs filesystems and swap, and then modify /etc/mkinitcpio.conf for zfs support.

Use the UUID to identify your partitions to mount for /etc/fstab for persistent device identification. You can look them up by

ls -lh /dev/disk/by-uuid/

If you can’t see them in the chroot environment, exit the chroot environment and execute the above ls command should work.

Below, an example, /etc/fstab firl:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system>                            <dir> <type> <options> <dump> <pass>
UUID=1A66-24E1                             /boot vfat   defaults  0      0
UUID=5af3f596-1068-457f-9d64-4db5aa649caa  none  swap   defaults  0      0
UUID=388de0df-1d0b-4f78-b6a7-2888859d3fa1  none  swap   defaults  0      0

You can use this as a template for your need. You don’t need to add the zfs dataset here, as the ZFS module will handle it.

Modify the /etc/mkinitcpio.conf file and add zfs after keyboard and before filesystem in HOOKS. You can remove fsck If you don’t have non-zfs filesystems. The HOOKS line should look like this.
HOOKS=(base udev autodetect modconf block keyboard zfs filesystems)
Then regenerate the initramfs by mkinitcpio -P.

Once we have the initramfs, we need a bootloader to boot the system. See Install Arch Linux on ZFS - ArchWiki for more details.
For GRUB, execute

grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Manjaro
grub-mkconfig -o /boot/grub/grub.cfg

For systemd-boot, execute

bootctl install

and create an entry config file (e.g. /boot/loader/entries/manjaro5.10.conf) with the content similar to

title   Manjaro Linux 5.10
linux   /vmlinuz-5.10-x86_64
initrd  /amd-ucode.img
initrd  /intel-ucode.img
initrd  /initramfs-5.10-x86_64.img
options zfs=zroot/manjaro/root rw

You can install systemd-boot-manager and execute sdboot-manage gen to automatically create the entry config file for you.

Once we have the bootloader, we need to enable some necessary services for automatically mounting zfs datasets

systemctl enable zfs.target
systemctl enable zfs-import-cache
systemctl enable zfs-mount
systemctl enable zfs-import.target

We also need to enable the display manager for the system to boot directly into the desktop environment.
For example, in KDE, we need to enable sddm.

systemctl enable sddm.service

Finally, we need to generate the hostid. Do not skip this, otherwise the system will failed to unmount the zfs pool during shutdown or restart process

zgenhostid $(hostid)

Now your system is correctly configured. It’s time for more customisation.

6. Configure the system and create user account

Follows the 7th step of [root tip] Do a manual Manjaro installation to configure the system, including Console keyboard, Locale, Timezone, Clock, Hostname, Hosts configuration, System administration, Network, Time syncronization, and Root password. For the network configuration, I recommend using NetworkManager.
Once the above settings are done, create a administrator user for DE login.

useradd -m -G lp,network,power,sys,wheel -s /bin/bash <USERNAME>
passwd <USERNAME>

If you don’t want the users to have administrator privilege (i.e. sudoer), do not add wheel when creating the user.

Now you should have everything set! You can exit the chroot environment by exit.

7. Export zfs pool and reboot

Do not skip this, or you’ll have problem mounting the zfs datasets later
First, unmount all the non-zfs datasets (e.g. /boot)

sudo umount <mountpoint>

Then, unmount all zfs datasets

sudo zfs unmount -a

Finally, export all zfs pools

sudo zpool export <zfs-pool-name>

Once it’s done, you can safely reboot the system. You should have a working Manjaro full installation with ZFS support now!

ZFS pool Maintenance

Regular data scrubbing is necessary to keep the zfs pool healthy (see ZFS - ArchWiki for more information). There is an AUR package called systemd-zpool-scrub that create scrubbing systemd service and timer for you. The default timer is set to weekly. After installing that package, enable the scrubbing timer by

sudo systemctl enable zpool-scrub@<zfs-pool-name>.timer

If you use SSD for your zfs pool vdev, regular TRIM is important for sustained long-term performance.
To query which zfs pools support trimming, use

sudo zpool status -t

And you should get something like

  pool: zbigdata
 state: ONLINE
  scan: scrub repaired 4K in 01:25:36 with 0 errors on Mon Jul 18 01:25:40 2022
config:

        NAME                               STATE     READ WRITE CKSUM
        zbigdata                           ONLINE       0     0     0
          ata-ST2000DX002-2DV164_Z4Z9XARA  ONLINE       0     0     0  (trim unsupported)

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:56 with 0 errors on Mon Jul 18 00:02:00 2022
config:

        NAME                                             STATE     READ WRITE CKSUM
        zroot                                            ONLINE       0     0     0
          mirror-0                                       ONLINE       0     0     0
            nvme-Force_MP600_21038229000128553384-part3  ONLINE       0     0     0  (untrimmed)
            nvme-Force_MP600_21048229000128553DBB-part3  ONLINE       0     0     0  (untrimmed)

errors: No known data errors

We only need to enable regular TRIM on those zfs pools which have vdev that can be trimmed
zfs provides a zpool trim command for manual TRIM as well as a autotrim argument for zfs pools. Because how the automatic TRIM and a full zpool trim differ in operations, it makes sense to set both automatic trim for zfs pools as well as systemd timers for occasional full TRIM service.
To set the autotrim argument for zfs pools, use the below command:

sudo zpool set autotrim=on <zfs-pool-name>

To enable regular full zpool trim service, we need to create the below systemd timer and service:
/etc/systemd/system/zfs-trim@.timer

[Unit]
Description=Monthly zpool trim on %i

[Timer]
OnCalendar=monthly
AccuracySec=1h
Persistent=true

[Install]
WantedBy=multi-user.target

/etc/systemd/system/zfs-trim@.service

[Unit]
Description=zpool trim on %i

[Service]
Nice=19
IOSchedulingClass=idle
KillSignal=SIGINT
ExecStart=/usr/bin/zpool trim %i

[Install]
WantedBy=multi-user.target

And enable the timer for the supported zfs pools by:

sudo systemctl enable zfs-trim@<zfs-pool-name>.timer

Summary

ZFS is a really advanced filesystem and has lots of features to prevent data loss. The current automatic way to install Manjaro on ZFS is through Manjaro-Architect, but it doesn’t allow the users to customise their ZFS partitioning or using the existing ZFS pools for installation. I hope that this tutorial can give a simple indication of installing Manjaro on ZFS root without too many hassles to configure the afterwards installation.

5 Likes

Erm…I think that should be modprobe. :grin:

2 Likes

Nice work!

A few things I would add:

  • There is really no right or wrong way to configure your datasets. You can set them up how you want to have them.
  • Some options can only be set at the time of pool/dataset creation. Be sure to get those right. Especially setting ashift=12 on your pool for most disks. Since many disks misreport, if you let it autodetect it will usually be wrong negatively impacting performance.
  • More datasets is usually better then fewer datasets. There is not much downside to having more datasets but there are many benefits
    • The ability to set separate snapshot policies on different datasets
    • Ease of moving around your data later if you decide you want to change things
    • Flexibility of a backup strategy that involves snapshot replication
    • Flexibility to allow sharing of resources between installs or re-use of resources when you reinstall
    • As a point of reference, I currently have 56 datasets on my main workstation
  • Make sure not to forget about doing regular scrubs on your pools. I use a systemd timer that automatically runs them monthly. If you prefer to do them manually I would do them at minimal quarterly.
2 Likes

2 posts were split to a new topic: Is there any advantage using zfs in a system for day to day basic usage