What is NILFS?
NILFS is an abbreviation for New Implementation of a Log-structured File System.
It is a file system for Linux, developed in Japan at the NTT Cyber Space Laboratories. The initial developers were Ryusuke KONISHI, Koji SATO, Seiji KIHARA, Yoshiji AMAGAI, Hisashi HIFUMI and Satoshi MORIAI. (A full list of the main contributors can be found here.)
The first publications about NILFS date back to 2005. Unfortunately they were in Japanese, which did not help in promoting NILFS outside the Japanese speaking community, but it is included in the Linux kernel since version 2.6.13 and was published under the GNU General Public License.
What is it about?
NILFS was designed to minimize the likelyhood of data loss caused by filesystem corruption or human error. Therefore the developers chose the approach of an incremental round robin log-structured copy on write file system with a single place of writing using CRC32 for data and metadata.
(More about that topic you can find in this presentation by Ryusuke Konishi )
What does that mean?
Log-structured filesystem: The whole filesystem is a chronological journal.
Copy on Write: Nothing gets deleted, untill the partition runs out of space or the garbage collecter is run on purpose. If you edit a file, the original version stays untouched and the changes will be saved saparetely.
Incremental: In order to gain efficency, only altered blocks will be written.
Round robin: NILFS regards a partition as an infinite circle that consists of a single linear chronological sequence, similar to a circular buffer. The writing process starts at the first block of the partition. As soon as the end of the FS is reached, a garbage collection must be performed. After that the writing process will continue at the first free block.
CRC32: Checksums are generated as a protection against silent data corruption of data and metadata.
Single place of writing: reduces complexity.
Why would I want that?
From the description above we can conclude that NILFS is actually a sequence of incremental checkpoints, that can be flagged as snapshots. Flagging as snapshot prevents the garbage collector from deleting a checkpoint. NILFS can thus be reverted to an older state if necessary. As nothing gets deleted as long as the garbage collector is not run, it is possible to undelete files that got deleted accidentially by the user. The structure of NILFS makes it also robust against power outages. There is no delayed allocation (cf. XFS) and no saparate journal that could become inconsistant in the case of a power outage. The round robin approach reduces the likelyhood of writing hot spots on SSDs.
NILFS2 is supported by the Linux kernel and by GRUB2. On some distributions the nilfs-tools are not preinstalled (Linux Mint e.g.). If this is the case, just open your package manager and install them.
I’m using NILFS2 for my ‘/’ (root), my ‘/home’ and even my ‘/boot’ partition, since it is supported by both, by GRUB2 and the Linux kernel.
Unfortunately (January 2017) Calamares, the graphical Manjaro installer has a bug, that prevents it from installing to a NILFS ‘/’ (root) partition.
But luckily I was able to find a solution.
How to install Manjaro on NILFS2
1.) You need a Manjaro ISO (I recommend USB) + the Manjaro ISO root password, which should be “manjaro”
2.) Boot the ISO
3.) Open Gparted. It will ask for the root password, which should be “manjaro”.
4.) Create a partition scheme like the following one:
Partition Label Mountpoint Filesystem Recommended Size Creation order Recommended UUID command /dev/sda1 EFI ('/boot/efi') FAT16 or FAT32 betwenn 8 and 512 MiB¹ 1 /dev/sda2 BOOT ('/boot') NILFS2 2048 MiB² 2 sudo nilfs-tune -U cafebabe-cafe-cafe-cafe-cafebabe0002 /dev/sda2 /dev/sda5 ROOT_nilfs2 ('/') NILFS2 about 20 GiB 5 sudo nilfs-tune -U cafebabe-cafe-cafe-cafe-cafebabe0005 /dev/sda5 /dev/sda6 ROOT_ext4 ('/') EXT4 about 20 GiB 6 sudo tune2fs -U cafebabe-cafe-cafe-cafe-cafebabe0006 /dev/sda6 /dev/sda4 HOME ('/home') NILFS2 as much space as you have ;) 4 sudo nilfs-tune -U cafebabe-cafe-cafe-cafe-cafebabe0004 /dev/sda4 /dev/sda3 SWAP ('/swap') SWAP the same size as your RAM 3 sudo swaplabel -U cafebabe-cafe-cafe-cafe-cafebabe0003 /dev/sda3
¹ For a Windows installation on EFI the EFI-partition must not be smaller than 512 MiB and in FAT32.
² 2048 might seem a lot, but because of the snapshotting nature of NILFS2 and the the Linux kernel images, that will be saved in /boot, this partition should not be too small, else you might run into trouble with the garbage collector.
Hint: Create the ext4 partition as the last one so it gets automatically the highest number, because in the end you might want to delete it.
5.) Optionally you can change the UUIDs of the partitions. The commands and example UUIDs are already listed in step 4.
6.) Open Calamares and start the install procedure.
7.) When Calamares asks for partitioning, choose “Manual partitioning” and select the mountpoints for the partitions. They are already listed in step 4 of this tutorial. IMPORTANT: For now, choose the ROOT_ext4 partition for ‘/’. DON’T select the ROOT_nilfs2 partition!
9.) After the install, boot your Manjaro, in order to check, if it works.
10.) Boot again the ISO.
11.) Mount the partions ROOT_nilfs2 and ROOT_ext4.
12.) Copy all the data from ext4 to nilfs2 with the following command (you have to look up the paths of the mounted partitions):
sudo cp -Ra /THE-EXT4-ROOT-PARTITION/* /THE-NILFS2-ROOT-PARTITION
Ignore the journal copying error.
13.) Open /etc/fstab on the NILFS2 root partition as root and replace the root partition.
It should look like that:
# <file system> <mount point> <type> <options> <dump> <pass> UUID=550E-7218 /boot/efi vfat defaults,noatime 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0002 /boot nilfs2 defaults,noatime 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0006 / ext4 defaults,noatime,discard 0 1 UUID=cafebabe-cafe-cafe-cafe-cafebabe0004 /home nilfs2 defaults,noatime,discard 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0003 swap swap defaults,noatime,discard 0 0 tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
And make it look like that:
# <file system> <mount point> <type> <options> <dump> <pass> UUID=550E-7218 /boot/efi vfat defaults,noatime 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0002 /boot nilfs2 defaults,noatime 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0005 / nilfs2 defaults,noatime,discard 0 1 UUID=cafebabe-cafe-cafe-cafe-cafebabe0004 /home nilfs2 defaults,noatime,discard 0 2 UUID=cafebabe-cafe-cafe-cafe-cafebabe0003 swap swap defaults,noatime,discard 0 0 tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
14.) Mount the boot partition.
15.) Open the /boot/grub/grub.cfg as root.
16.) Find and replace:
cafebabe-cafe-cafe-cafe-cafebabe0006 -> cafebabe-cafe-cafe-cafe-cafebabe0005
gpt6 -> gpt5
ext2 -> nilfs2
17.) Save the changes, unmount the partitions and reboot.
18.) Boot will fail.
19.) No problem! Boot the fallback image from the grub menu.
20.) Success? Check with gparted! Yes? Congratulations!!!
21.) Now you need to execute sudo update-grub, in order to be able to boot normaly.
22.) The “sudo update-grub” didn’t help? Then install another kernel with the Manjaro settings tool. Boot into the new kernel. Remove the old one with the Manjaro settings tool. Then you can reinstall it, if you wish.
23.) Congratulations! Now you have a Manjaro, running completely on NILFS2.
If you wish, you may delete the ext4 partition now and use the free space for another purpose.
A life with NILFS2
How do I get rid of checkpoints to get more free diskspace?
Use the garbage collector:
sudo nilfs-clean /dev/sda2; sudo nilfs-clean /dev/sda5; sudo nilfs-clean /dev/sda4
For further information about converting checkpoints into snapshots, how to mount them and administration in general, please take a look at the manual: http://nilfs.sourceforge.net/en/manual.html
Recover a partition
Some technical details
How is NILFS integrated into the Linux kernel?
Actually NILFS is a separate kernel module and the garbage collector nilfs-clean is a userland demon. This is a very modular approach, which has the big advantage, that NILFS2 will stay compatible with the Linux kernel, even when the kernel developers will decide to make bigger internal changes.
"What is the role of “.nilfs” file? Is it deletable?"
In the highest directory of your NILFS-partitions you will find a file named .nilfs which is usually empty. This is why:
“The .nilfs file was used for locking between cleanerd and other NILFS utilities. This file is now obsolete, and you can delete it if you are using nilfs-utils 2.1 or later. mkfs.nilfs2 still creates this file to avoid troubles on older nilfs-utils environment.
The .nilfs is a regular file, and the kernel module of NILFS2 does not depend on this file.”
More about NILFS2
- Ryusuke KONISHI: Utilizing NILFS2 Fine‐grained Snapshots: http://raid6.com.au/posts/files/NILFS2_lcj2011_konishi.pdf
- TimeBrowse, a GUI snapshot manager for NILFS2 and Nautilus: https://sourceforge.net/projects/timebrowse/
- BE AWARE, that nilfs.org is not the official website anymore and that it was pwned after the NILFS-team gave the domain away. The new official site is: http://nilfs.sourceforge.net/en/ BUT: Even if nilfs.org was pwned, you might find there still some older official publications about NILFS (like the first publications from 2005 in Japanese) which are missing on the new site.
- A NILFS2-plugin for Thunar, showing a list of checkpoints: https://aur.archlinux.org/packages/thunar-nilfs2-plugin-git/