[Primer] NTFS on Linux

NTFS on Linux


CONVENTIONS

Examples and references use the following conventions:

  • /base/volume: indicates a generic mount point to an NTFS volume.
  • /dev/sdXY: a storage device X, and partition Y.
  • volume: generic reference to a partition containing a filesystem.
  • nano: the author’s text editor of choice. Substitute as needed.

NTFS mount errors

[Wrong fs type, bad option, bad superblock]

RECOGNISING MOUNT ERRORS

When trying to mount an NTFS volume, you may receive an error similar to the following;
and dmesg further assists in recognising the problem:

mount: _/media_: wrong fs type, bad option, bad superblock on _/dev/sdXY_, 
missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.
_sdXY_: volume is dirty and "force" flag is not set!

This is by far the most common error when an indicator of damage to an NTFS volume is found. The NTFS3 kernel mode driver will not mount an NTFS volume if is marked dirty.

You can check dmesg for any additional clues:

sudo dmesg -H | grep -i ntfs3

See also: NTFS3; dmesg


The Dirty Bit

The Linux NTFS filesystem driverntfs3 – returns a mount failure error when damage, or rather, an indicator of damage to an NTFS filesystem is found. This indicator is commonly referred to as a dirty bit.

When a dirty bit is detected, ntfs3 prevents an NTFS filesystem from being mounted. This is by design, and is a safety feature of ntfs3 intended to prevent further damage through continued use of the NTFS filesystem. Any damage must be properly repaired, and the dirty bit cleared, before the partition is allowed to mount.

Clearing the dirty bit does not repair any damage to an NTFS volume.

Although a dirty bit can be cleared within Linux, using specific Linux tools, this action is not recommended; unless as a precursor to using a tool specifically designed to diagnose and repair Windows filesystem errors, such as chkdsk.

See also: Dirty Bit;


Linux utilities and NTFS

Resetting the dirty bit on an NTFS volume (in Linux) is inherently unsafe – it’s a recipe for trouble if the NTFS filesystem has not first been correctly repaired. If an NTFS filesystem has crashed you absolutely must run the required recovery tools before using the volume again; and that requires a Windows environment.

NTFSFIX

The Linux tool ntfsfix is part of the ntfsprogs package, which in turn is part of the ntfs-3g package.

Blog posts and other resources in the wild actively suggest using ntfsfix to repair an NTFS volume in Linux; however, ntfsfix is limited when compared to Microsoft’s chkdsk, and only performs a minimal set of checks. It does little more than clear the dirty bit, which in itself, can create more issues than solutions.

Clearing the dirty bit in this fashion effectively hides deeper NTFS problems which normally would be caught and fixed by chkdsk. If the dirty bit has been wrongfully cleared prior to then using chkdsk, the check/repair process may also be impeded in Windows.


WARNING: NTFSFIX IS NOT CHKDSK

Windows chkdsk is recommended to safely check/repair an NTFS volume, however, if you wish to accept the risk despite the warnings, here’s how:

Before using any ntfsfix action, the NTFS volume must first be unmounted:

sudo umount /dev/sdXY

The Linux command ntfsfix (without options) will perform a minimal set of recovery actions. If any superficial damage is repaired the dirty bit may also be cleared.

sudo ntfsfix /dev/sdXY

As the usage information suggests, ntfsfix can also be used to clear the dirty bit or clear the list of bad sectors. In either case, these options do not actually fix anything.

Usage: ntfsfix [options] device  
   Attempt to fix an NTFS partition.  
  
   -b, --clear-bad-sectors Clear the bad sector list  
   -d, --clear-dirty       Clear the volume dirty flag  
   -h, --help              Display this help  
   -n, --no-action         Do not write anything  
   -V, --version           Display version information

To clear the dirty bit using ntfsfix, pass the -d, --clear-dirty argument; this will allow the NTFS volume to be mounted temporarily (to save personal files):

sudo ntfsfix -d /dev/sdXY
sudo mount -t ntfs3 /dev/sdXY /base/volume

See also: NTFS; man ntfsfix;


Repairing NTFS volumes

Despite some Linux tools being capable of repairing NTFS to some extent, always choose a tool specifically designed to diagnose and repair Windows filesystems, such as chkdsk.

If damage is indicated on an NTFS volume this should be checked/repaired by the Windows chkdsk utility; from an administrative command prompt, within a Windows environment.

Use /f to fix errors on an NTFS volume:

chkdsk x: /f
  • x: denotes the drive letter of the volume as it appears in Windows.

This combination would fix the NTFS volume; perform a check/repair of bad sectors; and print the name of each file to screen as it is processed:

chkdsk x: /r /x
  • The /r argument also includes the functionality of /f.
  • This combination will take an extremely long time to complete.

Windows CHKDSK for NTFS


CHKDSK can clear a dirty bit when either of these conditions are met:

  • A check/repair is successfully performed on an NTFS volume.
  • Damage to an NTFS volume is not found after a successful check/repair.

CHKDSK options for use with NTFS only:

Option Description
/f :lock: Fixes errors on the disk.
/v Displays the name of each file as the disk is checked.
/r :lock: Locates bad sectors and recovers readable data. (/r includes /f, with extra analysis).
/x Forces the volume to dismount, if needed; all open handles are invalidated. (/x includes /f).
/? Displays help at the command prompt.
  • If /f is not used, chkdsk does not fix any errors.
  • Use /r to find physical errors and attempt to recover data.
  • :lock: indicates that the disk must be locked to use the option.

CHKDSK exit codes

Exit codes reported by chkdsk after completion:

Exit code Description
0 No errors were found.
1 Errors were found and fixed.
2 Did not perform cleanup (/f not specified).
3 Could not check the disk, errors could not be fixed (/f not specified).

WINDOWS FSUTIL

The Windows utility FSUTIL can check for and set a dirty bit. This can be useful if the dirty bit has been previously cleared by a Linux tool and chkdsk is unable to properly determine whether damage is present and repair it.

  • Check for a dirty bit: fsutil dirty query x:
  • Set the dirty bit: fsutil dirty set x:
  • A dirty bit cannot be cleared using fsutil

See also: CHKDSK; FSUTIL;


What if I don’t have Windows?


:one: RUN CHKDSK FROM BOOTABLE MEDIA

Option 1. Windows 7/8/8.1/10/11 DVD/ISO/USB installer:

Boot with the DVD/ISO/USB, set your Language, choose ‘Repair your Computer’, then ‘Troubleshoot’, and finally ‘Command Prompt’’. Windows command-line tools will be immediately available for use.

If you miss the ‘Repair your computer’ option, you can also launch a command prompt when reaching the partitioning stage of Windows Setup; use the Shift+F10 key combination.

See also: Download Windows 10;

Option 2. Windows To Go USB:

This is a fully functional version of Windows that you install to a USB disk. Windows command-line tools will be immediately available for use. See Using Rufus To Create Windows To Go USB Drive to learn how to create a Windows To Go USB.

Option 3. Hiren’s BootCD PE:

Hiren’s BootCD PE ISO contains official the expected Microsoft tools found in a Windows PE environment. Among them is chkdsk which should be used to repair you NTFS volume.

Option 4. Windows Repair Disc:

There exist many other options, likely an Internet search away…


Note: An OEM Windows Recovery Disc of any kind is not suitable.


Boot with a Ventoy USB, and ISO files are automatically listed in the Ventoy menu, and can be booted directly. A 32GB capacity USB should allow ample space to store several ISOs of your choice.

Ventoy is available from the official Manjaro extra repository:

sudo pacman -S ventoy
Ventoy Usage:

Type ventoy (without arguments) to see usage information:

Usage:  Ventoy2Disk.sh CMD [ OPTION ] /dev/sdX  
 CMD:  
  -i  install Ventoy to sdX (fails if disk already installed with Ventoy)  
  -I  force install Ventoy to sdX (no matter if installed or not)  
  -u  update Ventoy in sdX  
  -l  list Ventoy information in sdX  
  
 OPTION: (optional)  
  -r SIZE_MB  preserve some space at the bottom of the disk (only for install)  
  -s/-S       enable/disable secure boot support (default is enabled)  
  -g          use GPT partition style, default is MBR (only for install)  
  -L          Label of the 1st exfat partition (default is Ventoy)  
  -n          try non-destructive installation (only for install)

Creating the Ventoy USB:

Write the Ventoy system to an empty USB drive;
use /dev/sdX to target the device itself, and not a partition:

sudo sh ventoy -i -r 100 -S -g -L VOLUME /dev/sdX
  • Enable -s or disable -S Secure Boot.
  • Substitute VOLUME for a volume label name to use.
  • Substitute /dev/sdX for the location of your USB device.
  • Preserve some space on the target device (example allows 100mb).

Updating the Ventoy USB:

Update the Ventoy system on an existing Ventoy USB;
ensure the updated Ventoy version is available in Manjaro; and then:

sudo ventoy -u /dev/sdX
  • The Ventoy USB update process is non-destructive.

See also: Ventoy (GitHub);


:two: RUN CHKDSK FROM A VIRTUAL MACHINE

Chkdsk can be run from a Windows Virtual Machine:

If the affected volume is a USB device, such as a thumb drive or a HDD/SSD/NVMe in a removable USB enclosure, you can try installing Windows as a VirtualBox guest. The volume can then be mounted directly to a Windows virtual environment, complete with all tools necessary.

See VirtualBox in the Manjaro Wiki for information on correctly installing VirtualBox.

The author suggests Windows 7/8/8.1/10 as a VM. Windows versions before this range may contain outdated or incompatible of tools; and Windows 11 may not install in VirtualBox due to strict requirements imposed by Microsoft. Also be aware that there are licensing considerations when installing Windows in any virtual environment.

See Windows 10 on VirtualBox.

Having a Windows OS available as a virtual machine makes repairing NTFS volumes on USB relatively easy. Simply mount a USB device in the Windows Guest OS and then run chkdsk normally. This method is also convenient for defragmenting an NTFS volume, as needed.

MY NTFS VOLUME IS NOT ON USB

If the affected NTFS volume is directly connected to your computer (HDD/SSD/NVMe), then the author suggests running CHKDSK from bootable media is the safest option. :eyes:

VIRTUALBOX RAW HARD DISK ACCESS

VirtualBox can allow raw hard disk access to (non-USB) physical disks or partitions as virtual disks. Raw hard disk access in VirtualBox is implemented as part of VMDK image support.

This involves creating a special VMDK file which defines where the data is to be stored. With raw partition support, any partitioning information is stored inside the VMDK image, and the host’s partitioning information is not affected.

The process is potentially dangerous in inexperienced hands, and is mentioned only to acknowledge that the option exists. Please conduct your own research to determine its suitability, and naturally, you accept all associated risks.

See also: Oracle VM VirtualBox Manual;


:three: MULTIBOOT MANJARO AND WINDOWS

If you are already multibooting Manjaro and Windows, this is perhaps the easiest scenario in which to manage your NTFS volumes; simply reboot to Windows and perform the necessary tasks as needed.

If multibooting is something you’ve perhaps read about but haven’t tried, and wish to experience it for yourself;

See Multiboot Linux and Windows on Separate Disks for a tutorial. :eyes:


ChkDsk fixed my disk. What now?

If chkdsk has successfully checked/repaired your NTFS volume, then no further action is needed. Continue using it as before, but remember to regularly check/repair the drive.


What if my disk still won’t mount?

If NTFS3 still refuses to mount your NTFS volume; if chkdsk was unsuccessful; this could indicate premature clearing of the dirty bit, or deeper issues with the disk itself.

The NTFS volume can likely be mounted with ntfs-3g as a temporary workaround; to allow recovering important files; but this carries with it some potential risk.


A Dirty Workaround

If an NTFS volume is marked as dirty, ntfs3 will not mount it. This is safety feature of the kernel mode driver, and the refusal to mount is by design.

Important to note is that ntfs-3g user mode driver works around this by ignoring the dirty bit. Any existing damage to the NTFS volume is therefore also ignored. :eyes:

What does this mean?

NTFS-3G should be considered only a temporary workaround to allow saving your data to another location. Regular use of the Windows chkdsk utility is recommended, regardless of which NTFS driver is used. Ignoring this could result in data loss.

See also: NTFS-3G; NTFS-3G


BLACKLIST NTFS3 AND USE NTFS-3G

1. Install NTFS-3G:

# Check whether the `ntfs-3g` package is installed
pacman -Qi ntfs-3g

# Install the `ntfs-3g` package, if needed
sudo pacman -S ntfs-3g

2. Blacklist NTFS3:

echo 'blacklist ntfs3' | sudo tee /etc/modprobe.d/blacklist-ntfs3.conf

Note: To reverse this decision at any time, delete blacklist-ntfs3.conf:

sudo rm /etc/modprobe.d/blacklist-ntfs3.conf
  • NTFS3 will again take precedence when no longer blacklisted.
  • NTFS-3G can optionally remain installed for manual mounting.

Mounting NTFS

Persistent Naming

Automatic mounting requires that a device is referenced by an attribute that will not, or is unlikely to, change during its foreseeable lifespan. Because of the way in which devices are enumerated, referencing by the device node is unsuitable.

Referencing an NTFS volume by its UUID is the most accepted method, and will be the main focus; although it is also possible to reference by volume label, if that name will not change.

A volume UUID is often referenced using /dev/disk/by-uuid/XXXX-XXXX notation, however, can vary according to specific usage requirements, such as for fstab, for example.

You can find attributes of an NTFS volume you wish to mount, using either BLKID or LSBLK.

BLKID and LSBLK

Both BLKID and LSBLK are part of the standard util-linux package included in most distributions:

# Check that the util-linux package is installed
pacman -Qi util-linux
# Install the util-linux package (if needed)
sudo pacman -S util-linux

THE BLKID COMMAND

The blkid command (without arguments) is functionally equivalent to using the -o full option; both commands output full details of all available devices.

Any of these options are appropriate for finding the UUID of an NTFS volume:

Option Description
/dev/sdXY output details of a specific device
-o, --output <format> output details in either list or full format
-s, --match-tag <TAG> output details of the UUID tag
-t, --match-token <NAME=value> output using LABEL=foo or TYPE=ntfs tokens

The output from the following commands will include the UUID of your NTFS volume; Note that elevated privileges are required for most BLKID commands.

Output details in a tabular list:

sudo blkid -o list

Output details as an unordered list:

sudo blkid -o full

Output details of all available UUIDs:

sudo blkid -s UUID

Output details based on a known volume LABEL:

sudo blkid -t LABEL=foo

Output details based on ntfs filesystem TYPE:

sudo blkid -t TYPE=ntfs

See also: [Nix Tools] BLKID for usage and examples. :eyes:


THE LSBLK COMMAND

The lsblk command (without arguments) is almost identical to using the -a option, which additionally outputs details of empty devices without partitions or mount points.

Any of these options are appropriate for finding the UUID of an NTFS volume:

Option Description
-f, --fs outputs sorted details with these predefined columns:NAME, FSTYPE, FSVER, LABEL, UUID, FSAVAIL, FSUSE%, MOUNTPOINTS
-o, --output <list> outputs details from user-selected columns; example: NAME, TYPE, FSTYPE, UUID, LABEL, MOUNTPOINT

The output of the following commands will include the UUID of your NTFS volume; Note that elevated privileges are required for some LSBLK commands.

Output predefined and sorted columns:

sudo lsblk -f

Output only the columns you specify:

sudo lsblk -o NAME,FSTYPE,UUID,LABEL,MOUNTPOINT

See also: [Nix Tools] LSBLK for usage and examples. :eyes:


Find the UUID= value of your NTFS volume from the BLKID or LSBLK, and keep it handy for reference. This UUID will be required for either the fstab or systemd unit mounting methods mentioned further in this article.

Note: The UUID should be uppercase for NTFS (lowercase for Linux filesystems).

See also: Persistent naming; Util-Linux; man blkid; man lsblk.


Mount Points

Traditionally, both /media and /mnt were used as base directories for mounting volumes; /media was reserved for external storage devices (such as USB and optical media); while /mnt was used for most everything else that wasn’t part of the main system structure.

In Manjaro (and other systemd-based distributions); removable storage (USB/Optical) is automatically mounted by the system to /run/media by default; and /mnt is instead used as a base for mounting non-system removable/temporary storage.

This leaves /media free to be used for additional non-system fixed (static) volumes (those foreign to the system hierarchy), however, this directory must be created manually.

1. To manually create the /media base directory:

sudo mkdir /media

→ Set root permissions for base directory (/media):

sudo chmod 755 /media

2. To create a mount point in an existing base directory:

sudo mkdir /base/volume

→ Set permissions for mount point directories (base/volume):

sudo chmod 700 /base/volume

FSTAB

A volume may not mount (automatically) if it is not properly referenced in /etc/fstab.
Removable drives (such as USB) do not need an fstab entry; use the taskbar widget instead.

  • View fstab with cat /etc/fstab
  • Edit fstab with sudo nano /etc/fstab

FSTAB ATTRIBUTES:

<file system> <mount point> <type> <options> <dump> <pass>
  • One character of empty space (minimum) is required between each attribute in an fstab entry.

NOTES ON FSTAB ENTRIES

Each fstab entry contains a reference to:

Attribute Description
file system / block device Denotes a block device or a remote file system to be mounted. A block device can be referenced by its node location (/dev/sdXY), LABEL or UUID; and on GPT partitioned disks, also the PARTUUID or PARTLABEL. Due to the dynamic nature of disk enumeration, node location is considered unreliable for consistent mounting.
mount point / dir The mount directory.
type Denotes the filesystem type: ntfs, exfat or vfat. (Linux types excluded for brevity).
options The literal default option is shorthand for:
async,auto,dev,exec,nouser,rw,suid.
The nofail option allows booting when the drive is not present (useful for removable media), and may be added after defaults using a comma separator (defaults,nofail).
See also: man mount ‘FILESYSTEM-INDEPENDENT_MOUNT_OPTIONS’.
dump This field is usually set to 0, which disables the check by the dump(8) utility.
pass Sets the order for file system checks at boot time;
For the root device it should be 1.
For other partitions it should be 2, or 0 to disable checking.
See also: man fsck.

DEFAULT FSTAB ENTRIES FOR NTFS:

When using NTFS-3G:

UUID=XXXX-XXXX /base/volume ntfs-3g defaults 0 0

When using NTFS3:

UUID=XXXX-XXXX /base/volume ntfs3 defaults 0 0

Remount

Note: After changing mount options in /etc/fstab, a remount may be needed:

sudo mount -o remount /base/volume

See also: fstab; fstab; How fstab works.


Disks (gnome-disk-utility)

Volumes can be mounted automatically using Disks (gnome-disk-utility), which is available for most Linux distributions, including Manjaro:

sudo pacman -S gnome-disk-utility

The Disks GUI tool - despite having gnome- in its name - is a standalone package which can be used for mounting volumes in addition to the usual Linux system partitions. Volumes mounted by Disks will automatically be added to /etc/fstab, after any pre-existing entries; in a format resembling:

/dev/disk/by-uuid/XXXXXXXXXXXXXXXX /base/XXXXXXXXXXXXXXXX auto nosuid,nodev,nofail,noauto 0 0
/dev/disk/by-uuid/XXXXXXXXXXXXXXXX /base/XXXXXXXXXXXXXXXX auto nosuid,nodev,nofail,x-gvfs-show,noauto 0 0

Systemd Units

MOUNT/AUTOMOUNT UNITS

Using fstab entries for volumes that may (or may not) be available at boot, is not ideal. It is possible to replace the fstab method with systemd mount/automount units.

A mount point must also exist prior to being used with Mount/Automount units. If a mount point was previously used in /etc/fstab, then we can presume it does.

  • If the desired mount point does not exist, create it, and set the necessary permissions (as previously discussed).

Mount/Automount templates

The following templates may be useful when creating the configuration files needed to mount and automount your NTFS volumes.


SYSTEMD MOUNT UNIT (TEMPLATE)

1. Create the mount unit configuration file:

  • Substitute base-volume for the desired base directory and volume mount point:
sudo nano /etc/systemd/system/base-volume.mount

2. Paste the following content into the newly opened nano session, and edit parameters in accordance with your needs:

[Unit]
# Substitute {VolumeName} for the human readable volume name.
# Substitute {base} for the desired base directory.
Description=Mount VolumeName partition on /base

[Mount]
# Wait time for device to become available (default: 10 seconds).
# This period must elapse before a mount error dialog appears.
TimeoutSec=10
# Substitute XXXX's for the volume UUID (Uppercase for NTFS).
# Referencing an unique volume LABEL name is also valid.
What=/dev/disk/by-uuid/XXXXXXXXXXXXXXXX
# Substitute {base} for the desired base directory.
# Substitute {volume} for the mount point (lowercase).
Where=/base/volume
# Use the appropriate filesystem type {ntfs|ntfs3|ntfs-3g}
Type=ntfs
# Mount options, as used in /etc/fstab.
Options=defaults,rw,noatime

# This should not be needed when activated by an automount.
# [Install]
# WantedBy=multi-user.target
  • Exit Nano, and Save.

3. Enable the mount unit

  • Substitute base-volume for the desired base directory and volume mount point:
sudo systemctl enable --now base-volume.mount

SYSTEMD AUTOMOUNT UNIT (TEMPLATE)

1. Create the automount unit configuration file:

  • Substitute base-volume for the desired base directory and volume mount point:
sudo nano /etc/systemd/system/base-volume.automount

2. Paste the following content into the newly opened nano session, and edit parameters in accordance with your needs:

[Unit]
# Substitute {VolumeName} for the human readable volume name.
Description=Automount VolumeName partition on /base

[Automount]
# Substitute {base} for the desired base directory.
# Substitute {volume} for the mount point (lowercase).
Where=/base/volume
# Idle time before device is unmounted (default: 10 seconds).
# Period of inactivity must elapse before device is unmounted.
TimeoutIdleSec=10

[Install]
WantedBy=multi-user.target
  • Exit Nano, and Save.

3. Enable the automount unit:

  • Substitute base-volume for the desired base directory and volume mount point:
sudo systemctl enable --now base-volume.automount

The system will mount the device on access and unmount it after (default: 10) seconds of inactivity. The mount point will be accessible only if the device is both present and accessible.


Alternative Filesystems


EXT4

EXT4 is perhaps the most widely supported Linux filesystem, and a reasonable choice to replace the Microsoft proprietary NTFS filesystem.

Other Linux filesystems such as BTRFS, XFS and ZFS are also possible candidates, however, remain beyond the scope of this article.

FSTAB usage (Lowercase UUID):

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /base/volume ext4 defaults 0 0

NTFS

Yes, NTFS, you read correctly. There are times when NTFS is simply the right choice:

If you multiboot, and wish to share data between Linux and Windows using a journaled filesystem on a dedicated disk volume (HDD/SSD); or, if you prefer the general robustness of NTFS, and are willing to deal with ongoing and sometimes inconvenient maintenance tasks.

FSTAB usage (Uppercase UUID):

UUID=XXXXXXXXXXXXXXXX /base/volume ntfs defaults 0 0

Note: The previous caveat (ntfs3, the dirty bit, chkdsk) remains relevant.


EXFAT

EXFAT is similar to FAT32, but without the 4 GB file size limit. You can use exFAT drives on Linux with full read-write support. In comparison to NTFS, EXFAT isn’t as resilient; however, some users consider it less troublesome when sharing data between Linux and Windows.

EXFAT is well suited if you use a removable USB disk as temporary, rather than long term, common storage between Linux and Windows.

FSTAB usage (Upppercase UUID):

UUID=XXXX-XXXX /base/volume exfat defaults,nofail 0 0

VFAT

VFAT filesystem formats are generally not recommended. This following information is listed mainly for historical context.

The vfat kernel module supports FAT12/FAT16/FAT32 filesystems.

Of these, FAT32 is the most appropriate for use in Linux, however, is generally not recommended due to inherent file size limitations. Traditionally, FAT32 has been used for USB thumb drives, though this trend is being superseded in favour of exfat. FAT32 is also the underlying filesystem of ESP partitions (with the exception of MacOS, which uses HFS+).

FSTAB usage (Uppercase UUID):

UUID=XXXX-XXXX /base/volume vfat defaults,nofail 0 0

See also: FAT filesystem and Linux


Contributors:

6 Likes

That covers just about everything and then some :grin:

3 Likes

The only comment is if you are only sharing data between windows and linux ntfs is not the best file system to use as you have stated above.
I prefere exfat native to both windows and linux.

7 posts were split to a new topic: Questions regarding [Primer] NTFS on Linux