Files missing during file transfer / renaming process

Support
1

Hi.

I have been plagued by a weird behavior, since the update in May 2023.

When I perform rename in NTFS drive, files in same directory may gone missing/hidden.
When I transfer files from one drive (EXT4 or NTFS) to another drive (NTFS), some files may gone missing.

eg. If my source has 100 files, the destination may only have 50 files.
According to folder properties, the size occupied is indeed reflecting the presence of 50 files only - so the missing 50 files is not there.
But some times, after some manipulation, the missing 50 files reappeared, and the folder properties reported size of 100 files now.
However, there is no sure way to recover the missing files.

I have tested the issue, and i am certain that:

  1. it is not dependent on file manager (I used Dolphin and Nemo),
  2. it is not dependent on renaming app (KRename and F2 function),
  3. it can happen to any hard drive - it is not a HDD corruption issue.
  4. the issue will only happen on NTFS file system.

And when the issue happens, the NTFS hard drive usually become non-responsive, and cannot be un-mount.
After reboot, the hard drive also cannot be mounted due to unknown error.
A fix using command NTFSFIX showed that the “$MFTMirr does not match $MFT”.
And though NTFSFIX can resolve the un-mountable issue, it does not recover the files lost.

My temporary solution has been to transfer the files out to a EXT4 drive, perform batch rename, and transfer them back to NTFS drive.
But today, 3/6 files were lost during the transfer, so it seems that there is no safe solution anymore.

Can anyone point me the right direction to resolve this issue?

Thanks!

My versions:
Operating System: Manjaro Linux
KDE Plasma Version: 5.27.6
KDE Frameworks Version: 5.108.0
Qt Version: 5.15.10
Kernel Version: 6.4.3-1-MANJARO (64-bit)
Graphics Platform: X11

2

I suspect that the symptoms originate from the fact that UNIX filesystems are inherently case-sensitive — or at the very least, by default, because ext4 now supports case retention without case-sensitivity if specified as such in the mount options — whereas ntfs is inherently case-retentive but case-insensitive by default.

As such, copying two separate files on a UNIX filesystem whose name only differs in case to ntfs (or another Microsoft-style filesystem) will cause the file copied second to overwrite the file that was copied first, and therefore, the number of copies left on the target medium after the operation to differ from the number of files at the source.

As an example, on an ext4 or btrfs filesystem, you can have two separate files that are located in the same directory, one being called Letter.txt and the other called LEttEr.TxT, and if you copy them both over to an ntfs volume, then you will have only one file on ntfs, with the case of its filename characters being one of those two spellings — most commonly (albeit not necessarily) the spelling of the first of the two files.

It is not unthinkable that this sort of thing occurs when copying large quantities of files.

3

I’m aware of the different letters supported in EXT4 and NTFS.

In my scenario, the list of files is like this:
ABC E01.mkv
ABC E02.mkv
ABC E03.mkv
ABC E04.mkv
and so on.

So as you can see: none of the files overlap/duplicate in file name.

In fact, as I transfer the files from NTFS to EXT4 to perform batch rename, it means the files have no issue in NTFS previously.
And my batch rename operation merely include a constant XYZ to the end of the file name:
ABC E01 XYZ.mkv
ABC E02 XYZ.mkv
ABC E03 XYZ.mkv
ABC E04 XYZ.mkv

So I presume this XYZ should not change the file name “requirement” by NTFS.

4

What are the mount options on the ntfs volume, and which filesystem type is it being mounted as? ntfs, ntfs-3g or ntfs3?

5

image
image585×522 28.2 KB

6

That doesn’t tell me much. Try this… :arrow_down:

mount | grep ntfs
7

image

8

Sorry for the delayed reply — I had to go away from the keyboard for a while.

First of all, don’t use screenshots for terminal output. Select the text in the terminal with the mouse, then click the </> button in the toolbar of the post editor and paste the copied output in between the two lines of three backticks.

Secondly, run the command with sudo privileges. :arrow_down:

sudo mount | grep ntfs

I need to see what driver is being used to mount your ntfs volume.

9

No worry.

Here is what you asked.

/dev/sda1 on /run/media/wind77/R+SaSu type ntfs3 (rw,nosuid,nodev,relatime,uid=1000,gid=1001,windows_names,iocharset=utf8,uhelper=udisks2)

10

This could possibly be the culprit. That driver is still not very mature. Try mounting it manually with the ntfs-3g driver instead. :arrow_down:

sudo umount /dev/sda1
sudo mount -t ntfs-3g -o rw,nosuid,nodev,relatime,uid=1000,gid=1001,windows_names,iocharset=utf8 /dev/sda1 /run/media/wind77/R+SaSu
11

I will try your approach tonight - right now the NTFS drive is transferring out all content, which needs 24 hrs.

I remember when I installed Manjaro earlier this year, the NTFS used was still ntfs-3g.
I have no idea it has been swapped to ntfs3.
Any idea when ntfs-3g was replaced by ntfs3 during update?

12

It wasn’t, really, but the ntfs3 driver was at some point added to the kernel, whereas ntfs-3g is a driver that runs in userspace, via the FUSE framework. So I’m guessing that if you’re running a later kernel, it will use the in-kernel ntfs3 driver instead of the userspace ntfs-3g driver. :thinking:

13

https://www.reddit.com/r/archlinux/comments/13ona5u/ntfs3_driver_keeps_corrupting_ntfs_filesystem_on/

Seems like NTFS3 is creating lots of issue to other users as well.

In my usage, I plug in different NTFS drives, after each reboot.
So, does that mean I need to run the following command, after each reboot?

Or the system will auto use NTFS-3g for all my other NTFS drives, henceforth?

14

Well, if the partition and mountpoint are always the same between your various drives, then could create a designated mountpoint in your $HOME and add a line to /etc/fstab — something along the lines of… :arrow_down:

/dev/sda1   /home/wind77/NTFS  ntfs-3g  noauto,nofail,rw,nosuid,nodev,relatime,uid=1000,gid=1001,windows_names,iocharset=utf8   0   0

You will then need to manually mount the filesystem — you should be able to do that from within Dolphin — and make sure you unmount it again before unplugging it.

15

fstab only contains the settings for non-default (User session defaults) device.
My NTFS drives are all set as “default”, and their settings are not in fstab.

Is there another place to look for “default” device?

16

It doesn’t matter. If you add that line to /etc/fstab, then that will take precedence. Without it, the system uses the udisks2 framework of systemd to autodetect and automount your filesystem, and as you could see, it’ll then use the buggy ntfs3 driver by default because your kernel happens to support that.

17

So as to say, if the 1st ntfs device in fstab is mounted using ntfs-3g, then all other ntfs devices (user session default or not) will be following ntfs-3g?
ie. the system will use either ntfs-3g or ntfs3 at any time, but not both at the same time?

18

No, you misunderstand. That line in /etc/fstab will allow you to easily mount an NTFS partition, as long as it’s the first partition on the drive that the kernel sees as /dev/sda.

So unless you’re also mounting NTFS partitions which are not the first partition on the drive and/or they are on another drive than /dev/sda, you can use this one line for sequentially using and mounting an NTFS partition. Dolphin will check /etc/fstab when you tell it to mount the volume, and if you mount it manually, then you only need to issue the command… :arrow_down:

sudo mount /home/wind77/NTFS

… provided that the folder /home/wind77/NTFS exists, of course.

Perhaps some light instructional reading is at the order. :stuck_out_tongue: :arrow_down:

19

Hmm…
I have more than 20 HDDs in NTFS, and I rotate them to connect to my PC.
By default, when they are connected, they are found under /media.

These 20 HDDS are not listed in the fstab.

So, I guess I will have to type the command 20x?
:cry:

20

If they are all the first partition on the drive that the kernel sees as /dev/sda, then they don’t have to be, because you (presumably) won’t be plugging in more than one drive at the time, and each time you plug in a drive, it’ll use that one line.