[Primer] NTFS on Linux

soundofthunder · 7 March 2024 07:11

NTFS on Linux

CONVENTIONS

Examples and references use the following conventions:

/base/volume: indicates a generic mount point to an NTFS volume.
/dev/sdXY: a storage device X, and partition Y.
volume: generic reference to a partition containing a filesystem.
nano: the author’s text editor of choice. Substitute as needed.

NTFS mount errors

[Wrong fs type, bad option, bad superblock]

RECOGNISING MOUNT ERRORS

When trying to mount an NTFS volume, you may receive an error similar to the following;
and dmesg further assists in recognising the problem:

mount: _/media_: wrong fs type, bad option, bad superblock on _/dev/sdXY_, 
missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.
_sdXY_: volume is dirty and "force" flag is not set!

This is by far the most common error when an indicator of damage to an NTFS volume is found. The NTFS3 kernel mode driver will not mount an NTFS volume if is marked dirty.

You can check dmesg for any additional clues:

sudo dmesg -H | grep -i ntfs3

The Dirty Bit

The Linux NTFS filesystem driver – ntfs3 – returns a mount failure error when damage, or rather, an indicator of damage to an NTFS filesystem is found. This indicator is commonly referred to as a dirty bit.

When a dirty bit is detected, ntfs3 prevents an NTFS filesystem from being mounted. This is by design, and is a safety feature of ntfs3 intended to prevent further damage through continued use of the NTFS filesystem. Any damage must be properly repaired, and the dirty bit cleared, before the partition is allowed to mount.

Clearing the dirty bit does not repair any damage to an NTFS volume.

Although a dirty bit can be cleared within Linux, using specific Linux tools, this action is not recommended; unless as a precursor to using a tool specifically designed to diagnose and repair Windows filesystem errors, such as chkdsk.

Linux utilities and NTFS

Resetting the dirty bit on an NTFS volume (in Linux) is inherently unsafe – it’s a recipe for trouble if the NTFS filesystem has not first been correctly repaired. If an NTFS filesystem has crashed you absolutely must run the required recovery tools before using the volume again; and that requires a Windows environment.

NTFSFIX

The Linux tool ntfsfix is part of the ntfsprogs package, which in turn is part of the ntfs-3g package.

Blog posts and other resources in the wild actively suggest using ntfsfix to repair an NTFS volume in Linux; however, ntfsfix is limited when compared to Microsoft’s chkdsk, and only performs a minimal set of checks. It does little more than clear the dirty bit, which in itself, can create more issues than solutions.

Clearing the dirty bit in this fashion effectively hides deeper NTFS problems which normally would be caught and fixed by chkdsk. If the dirty bit has been wrongfully cleared prior to then using chkdsk, the check/repair process may also be impeded in Windows.

WARNING: NTFSFIX IS NOT CHKDSK

Windows chkdsk is recommended to safely check/repair an NTFS volume, however, if you wish to accept the risk despite the warnings, here’s how:

Before using any ntfsfix action, the NTFS volume must first be unmounted:

sudo umount /dev/sdXY

The Linux command ntfsfix (without options) will perform a minimal set of recovery actions. If any superficial damage is repaired the dirty bit may also be cleared.

sudo ntfsfix /dev/sdXY

As the usage information suggests, ntfsfix can also be used to clear the dirty bit or clear the list of bad sectors. In either case, these options do not actually fix anything.

Usage: ntfsfix [options] device  
   Attempt to fix an NTFS partition.  
  
   -b, --clear-bad-sectors Clear the bad sector list  
   -d, --clear-dirty       Clear the volume dirty flag  
   -h, --help              Display this help  
   -n, --no-action         Do not write anything  
   -V, --version           Display version information

To clear the dirty bit using ntfsfix, pass the -d, --clear-dirty argument; this will allow the NTFS volume to be mounted temporarily (to save personal files):

sudo ntfsfix -d /dev/sdXY
sudo mount -t ntfs3 /dev/sdXY /base/volume

Repairing NTFS volumes

Despite some Linux tools being capable of repairing NTFS to some extent, always choose a tool specifically designed to diagnose and repair Windows filesystems, such as chkdsk.

If damage is indicated on an NTFS volume this should be checked/repaired by the Windows chkdsk utility; from an administrative command prompt, within a Windows environment.

Use /f to fix errors on an NTFS volume:

chkdsk x: /f

x: denotes the drive letter of the volume as it appears in Windows.

This combination would fix the NTFS volume; perform a check/repair of bad sectors; and print the name of each file to screen as it is processed:

chkdsk x: /r /x

The /r argument also includes the functionality of /f.
This combination will take an extremely long time to complete.

Windows CHKDSK for NTFS

CHKDSK can clear a dirty bit when either of these conditions are met:

A check/repair is successfully performed on an NTFS volume.
Damage to an NTFS volume is not found after a successful check/repair.

CHKDSK options for use with NTFS only:

Option	Description
`/f`	Fixes errors on the disk.
`/v`	Displays the name of each file as the disk is checked.
`/r`	Locates bad sectors and recovers readable data. (/r includes /f, with extra analysis).
`/x`	Forces the volume to dismount, if needed; all open handles are invalidated. (/x includes /f).
`/?`	Displays help at the command prompt.

If /f is not used, chkdsk does not fix any errors.
Use /r to find physical errors and attempt to recover data.
indicates that the disk must be locked to use the option.

CHKDSK exit codes

Exit codes reported by chkdsk after completion:

Exit code	Description
0	No errors were found.
1	Errors were found and fixed.
2	Did not perform cleanup (/f not specified).
3	Could not check the disk, errors could not be fixed (/f not specified).

WINDOWS FSUTIL

The Windows utility FSUTIL can check for and set a dirty bit. This can be useful if the dirty bit has been previously cleared by a Linux tool and chkdsk is unable to properly determine whether damage is present and repair it.

Check for a dirty bit: fsutil dirty query x:
Set the dirty bit: fsutil dirty set x:
A dirty bit cannot be cleared using fsutil

What if I don’t have Windows?

RUN CHKDSK FROM BOOTABLE MEDIA

CHKDSK can also be run from bootable media:

Windows 7/8/8.1/10/11 DVD/USB installer
Windows Repair Disc, such as Hiren’s BootCD PE
Windows To Go DVD/USB
Using Rufus To Create Windows To Go USB Drive

Note: An OEM Windows Recovery Disc of any kind is not suitable.

Boot with a Ventoy USB, and ISO files are automatically listed in the Ventoy menu, and can be booted directly. A 32GB (minimum) capacity USB should allow ample space to store several ISOs of your choice.

Ventoy is available from the official Manjaro extra repository:

sudo pacman -S ventoy

Type ventoy (without arguments) to see usage information:

Usage:  Ventoy2Disk.sh CMD [ OPTION ] /dev/sdX  
 CMD:  
  -i  install Ventoy to sdX (fails if disk already installed with Ventoy)  
  -I  force install Ventoy to sdX (no matter if installed or not)  
  -u  update Ventoy in sdX  
  -l  list Ventoy information in sdX  
  
 OPTION: (optional)  
  -r SIZE_MB  preserve some space at the bottom of the disk (only for install)  
  -s/-S       enable/disable secure boot support (default is enabled)  
  -g          use GPT partition style, default is MBR (only for install)  
  -L          Label of the 1st exfat partition (default is Ventoy)  
  -n          try non-destructive installation (only for install)

Write the Ventoy system to an empty USB drive;
use /dev/sdX to target the device itself, and not a partition:

sudo sh ventoy -i -r 100 -S -g -L VOLUME /dev/sdX

Substitute VOLUME for a volume label name to use.
Substitute /dev/sdX for the location of your USB device.
Preserve some space on the target device (example allows 100mb).

ChkDsk fixed my disk. What now?

If chkdsk has successfully checked/repaired your NTFS volume, then no further action is needed. Continue using it as before, but remember to regularly check/repair the drive.

What if my disk still won’t mount?

If NTFS3 still refuses to mount your NTFS volume; if chkdsk was unsuccessful; this could indicate premature clearing of the dirty bit, or deeper issues with the disk itself.

The NTFS volume can likely be mounted with ntfs-3g as a temporary workaround; to allow recovering important files; but this carries with it some potential risk.

A Dirty Workaround

If an NTFS volume is marked as dirty, ntfs3 will not mount it. This is safety feature of the kernel mode driver, and the refusal to mount is by design.

Important to note is that ntfs-3g user mode driver works around this by ignoring the dirty bit. Any existing damage to the NTFS volume is therefore also ignored.

What does this mean?

NTFS-3G should be considered only a temporary workaround to allow saving your data to another location. Regular use of the Windows chkdsk utility is recommended, regardless of which NTFS driver is used. Ignoring this could result in data loss.

Mounting NTFS

Persistent Naming

Automatic mounting requires that a device is referenced by an attribute that will not, or is unlikely to, change during its foreseeable lifespan. Because of the way in which devices are enumerated, referencing by the device node is unsuitable.

Referencing an NTFS volume by its UUID is the most accepted method, and will be the main focus; although it is also possible to reference by volume label, if that name will not change.

A volume UUID is often referenced using /dev/disk/by-uuid/XXXX-XXXX notation, however, can vary according to specific usage requirements, such as for fstab, for example.

You can find attributes of an NTFS volume you wish to mount, using either BLKID or LSBLK.

BLKID and LSBLK

Both BLKID and LSBLK are part of the standard util-linux package included in most distributions:

# Check that the util-linux package is installed
pacman -Qi util-linux
# Install the util-linux package (if needed)
sudo pacman -S util-linux

THE BLKID COMMAND

The blkid command (without arguments) is functionally equivalent to using the -o full option; both commands output full details of all available devices.

Any of these options are appropriate for finding the UUID of an NTFS volume:

Option	Description
`/dev/sdXY`	output details of a specific device
`-o`, `--output <format>`	output details in either `list` or `full` format
`-s`, `--match-tag <TAG>`	output details of the `UUID` tag
`-t`, `--match-token <NAME=value>`	output using `LABEL=foo` or `TYPE=ntfs` tokens

The output from the following commands will include the UUID of your NTFS volume; Note that elevated privileges are required for most BLKID commands.

Output details in a tabular list:

sudo blkid -o list

Output details as an unordered list:

sudo blkid -o full

Output details of all available UUIDs:

sudo blkid -s UUID

Output details based on a known volume LABEL:

sudo blkid -t LABEL=foo

Output details based on ntfs filesystem TYPE:

sudo blkid -t TYPE=ntfs

See also: [Nix Tools] BLKID for usage and examples.

THE LSBLK COMMAND

The lsblk command (without arguments) is almost identical to using the -a option, which additionally outputs details of empty devices without partitions or mount points.

Any of these options are appropriate for finding the UUID of an NTFS volume:

Option	Description
`-f`, `--fs`	outputs sorted details with these predefined columns:`NAME`, `FSTYPE`, `FSVER`, `LABEL`, `UUID`, `FSAVAIL`, `FSUSE%`, `MOUNTPOINTS`
`-o`, `--output <list>`	outputs details from user-selected columns; example: `NAME`, `TYPE`, `FSTYPE`, `UUID`, `LABEL`, `MOUNTPOINT`

The output of the following commands will include the UUID of your NTFS volume; Note that elevated privileges are required for some LSBLK commands.

Output predefined and sorted columns:

sudo lsblk -f

Output only the columns you specify:

sudo lsblk -o NAME,FSTYPE,UUID,LABEL,MOUNTPOINT

See also: [Nix Tools] LSBLK for usage and examples.

Find the UUID= value of your NTFS volume from the BLKID or LSBLK, and keep it handy for reference. This UUID will be required for either the fstab or systemd unit mounting methods mentioned further in this article.

Note: The UUID should be uppercase for NTFS (lowercase for Linux filesystems).

See also: Persistent naming; Util-Linux; man blkid; man lsblk.

Mount Points

Traditionally, both /media and /mnt were used as base directories for mounting volumes; /media was reserved for external storage devices (such as USB and optical media); while /mnt was used for most everything else that wasn’t part of the main system structure.

In Manjaro (and other systemd-based distributions); removable storage (USB/Optical) is automatically mounted by the system to /run/media by default; and /mnt is instead used as a base for mounting non-system removable/temporary storage.

This leaves /media free to be used for additional non-system fixed (static) volumes (those foreign to the system hierarchy), however, this directory must be created manually.

1. To manually create the /media base directory:

sudo mkdir /media

→ Set root permissions for base directory (/media):

sudo chmod 755 /media

2. To create a mount point in an existing base directory:

sudo mkdir /base/volume

→ Set permissions for mount point directories (base/volume):

sudo chmod 700 /base/volume

FSTAB

A volume may not mount (automatically) if it is not properly referenced in /etc/fstab.
Removable drives (such as USB) do not need an fstab entry; use the taskbar widget instead.

View fstab with cat /etc/fstab
Edit fstab with sudo nano /etc/fstab

FSTAB ATTRIBUTES:

<file system> <mount point> <type> <options> <dump> <pass>

One character of empty space (minimum) is required between each attribute in an fstab entry.

NOTES ON FSTAB ENTRIES

Each fstab entry contains a reference to:

Attribute	Description
file system / block device	Denotes a block device or a remote file system to be mounted. A block device can be referenced by its node location (`/dev/sdXY`), LABEL or UUID; and on GPT partitioned disks, also the PARTUUID or PARTLABEL. Due to the dynamic nature of disk enumeration, node location is considered unreliable for consistent mounting.
mount point / dir	The mount directory.
type	Denotes the filesystem type: `ntfs`, `exfat` or `vfat`. (Linux types excluded for brevity).
options	The literal default option is shorthand for: `async,auto,dev,exec,nouser,rw,suid`. The nofail option allows booting when the drive is not present (useful for removable media), and may be added after defaults using a comma separator (`defaults,nofail`). See also: man mount ‘FILESYSTEM-INDEPENDENT_MOUNT_OPTIONS’.
dump	This field is usually set to 0, which disables the check by the dump(8) utility.
pass	Sets the order for file system checks at boot time; For the root device it should be 1. For other partitions it should be 2, or 0 to disable checking. See also: man fsck.

For those wishing to mount hot swappable devices, please consider the systemd unit procedure outlined in Hot Swapping SATA devices in /etc/fstab.

DEFAULT FSTAB ENTRIES FOR NTFS:

When using NTFS-3G:

UUID=XXXX-XXXX /base/volume ntfs-3g defaults 0 0

When using NTFS3:

UUID=XXXX-XXXX /base/volume ntfs3 defaults 0 0

Remount

Note: After changing mount options in /etc/fstab, a remount may be needed:

sudo mount -o remount /base/volume

See also: fstab; fstab; How fstab works.

Disks (gnome-disk-utility)

Volumes can be mounted automatically using Disks (gnome-disk-utility), which is available for most Linux distributions, including Manjaro:

sudo pacman -S gnome-disk-utility

The Disks GUI tool is best used for volumes to be mounted in addition to the usual Linux system partitions. Volumes mounted by Disks will automatically be added to /etc/fstab, after any pre-existing entries; in a format resembling:

/dev/disk/by-uuid/XXXXXXXXXXXXXXXX /base/XXXXXXXXXXXXXXXX auto nosuid,nodev,nofail,noauto 0 0
/dev/disk/by-uuid/XXXXXXXXXXXXXXXX /base/XXXXXXXXXXXXXXXX auto nosuid,nodev,nofail,x-gvfs-show,noauto 0 0

Systemd Units

MOUNT/AUTOMOUNT UNITS

Using fstab entries for volumes that may (or may not) be available at boot, is not ideal. It is possible to replace the fstab method with systemd mount/automount units.

A mount point must also exist prior to being used with Mount/Automount units. If a mount point was previously used in /etc/fstab, then we can presume it does.

If the desired mount point does not exist, create it, and set the necessary permissions (as previously discussed).

Mount/Automount templates

The following templates may be useful when creating the configuration files needed to mount and automount your NTFS volumes.

SYSTEMD MOUNT UNIT (TEMPLATE)

1. Create the mount unit configuration file:

Substitute base-volume for the desired base directory and volume mount point:

sudo nano /etc/systemd/system/base-volume.mount

2. Paste the following content into the newly opened nano session, and edit parameters in accordance with your needs:

[Unit]
# Substitute {VolumeName} for the human readable volume name.
# Substitute {base} for the desired base directory.
Description=Mount VolumeName partition on /base

[Mount]
# Wait time for device to become available (default: 10 seconds).
# This period must elapse before a mount error dialog appears.
TimeoutSec=10
# Substitute XXXX's for the volume UUID (Uppercase for NTFS).
# Referencing an unique volume LABEL name is also valid.
What=/dev/disk/by-uuid/XXXXXXXXXXXXXXXX
# Substitute {base} for the desired base directory.
# Substitute {volume} for the mount point (lowercase).
Where=/base/volume
# Use the appropriate filesystem type {ntfs|ntfs3|ntfs-3g}
Type=ntfs
# Mount options, as used in /etc/fstab.
Options=defaults,rw,noatime

# This should not be needed when activated by an automount.
# [Install]
# WantedBy=multi-user.target

Exit Nano, and Save.

3. Enable the mount unit

Substitute base-volume for the desired base directory and volume mount point:

sudo systemctl enable --now base-volume.mount

SYSTEMD AUTOMOUNT UNIT (TEMPLATE)

1. Create the automount unit configuration file:

Substitute base-volume for the desired base directory and volume mount point:

sudo nano /etc/systemd/system/base-volume.automount

2. Paste the following content into the newly opened nano session, and edit parameters in accordance with your needs:

[Unit]
# Substitute {VolumeName} for the human readable volume name.
Description=Automount VolumeName partition on /base

[Automount]
# Substitute {base} for the desired base directory.
# Substitute {volume} for the mount point (lowercase).
Where=/base/volume
# Idle time before device is unmounted (default: 10 seconds).
# Period of inactivity must elapse before device is unmounted.
TimeoutIdleSec=10

[Install]
WantedBy=multi-user.target

Exit Nano, and Save.

3. Enable the automount unit:

Substitute base-volume for the desired base directory and volume mount point:

sudo systemctl enable --now base-volume.automount

The system will mount the device on access and unmount it after (default: 10) seconds of inactivity. The mount point will be accessible only if the device is both present and accessible.

Alternative Filesystems

EXT4

EXT4 is perhaps the most widely supported Linux filesystem, and a reasonable choice to replace the Microsoft proprietary NTFS filesystem.

Other Linux filesystems such as BTRFS, XFS and ZFS are also possible candidates, however, remain beyond the scope of this article.

FSTAB usage (Lowercase UUID):

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /base/volume ext4 defaults 0 0

NTFS

Yes, NTFS, you read correctly. There are times when NTFS is simply the right choice:

If you multiboot, and wish to share data between Linux and Windows using a journaled filesystem on a dedicated disk volume (HDD/SSD); or, if you prefer the general robustness of NTFS, and are willing to deal with ongoing and sometimes inconvenient maintenance tasks.

FSTAB usage (Uppercase UUID):

UUID=XXXXXXXXXXXXXXXX /base/volume ntfs defaults 0 0

Note: The previous caveat (ntfs3, the dirty bit, chkdsk) remains relevant.

EXFAT

EXFAT is similar to FAT32, but without the 4 GB file size limit. You can use exFAT drives on Linux with full read-write support. In comparison to NTFS, EXFAT isn’t as resilient; however, some users consider it less troublesome when sharing data between Linux and Windows.

EXFAT is well suited if you use a removable USB disk as temporary, rather than long term, common storage between Linux and Windows.

FSTAB usage (Upppercase UUID):

UUID=XXXX-XXXX /base/volume exfat defaults,nofail 0 0

VFAT

VFAT filesystem formats are generally not recommended. This following information is listed mainly for historical context.

The vfat kernel module supports FAT12/FAT16/FAT32 filesystems.

Of these, FAT32 is the most appropriate for use in Linux, however, is generally not recommended due to inherent file size limitations. Traditionally, FAT32 has been used for USB thumb drives, though this trend is being superseded in favour of exfat. FAT32 is also the underlying filesystem of ESP partitions (with the exception of MacOS, which uses HFS+).

FSTAB usage (Uppercase UUID):

UUID=XXXX-XXXX /base/volume vfat defaults,nofail 0 0