NTFS - NTFS file transfer across partitions - resulting in different file numbers and file size

[wind77]

I have 2 HDDs, both are NTFS in GPT.
Both are mounted using ntfs3.

When I finished copying files from Source partition (S) to Destination partition (D), I realized there was something weird.
The (S) has 9116 files, and (D) only has 8996 files.
And in terms of file size, there is a 45,006,036,093 byte difference.

As reported by Dolphin, (Left is Source; Right is Destination)

I was worry if the file transfer was exited prematurely, so I run FreeFileSync to check:
a. Compare by File time and size - both (S) and (D) are identical - no difference.
b. Compare by File content - both (S) and (D) are identical - no difference.
c. Compare by File size - both (S) and (D) are identical - no difference.

There is no symbolic link in the partitions.

I’m now stuck - I cannot be certain if 100% of (S) has been copied to (D), and thus I cannot re-purpose (S) for other purpose.

Is there a way for me to troubleshoot this issue?

My setup:

Operating System: Manjaro Linux 
KDE Plasma Version: 5.27.7
KDE Frameworks Version: 5.108.0
Qt Version: 5.15.10
Kernel Version: 6.4.9-1-MANJARO (64-bit)

[Mirdarthos]

Hi @wind77,

In Dolphin, make sure Show hidden files are enabled:

https://i.imgur.com/pfGZq6Y.png

Or check in the terminal with:

ls -l

Hidden files/directories start with a . (dot).

[wind77]

Show hidden files is enabled.

[Mirdarthos]

Try diff. From the tldr page:

[...]
Compare directories, only showing the names of files that differ:
diff --recursive --brief old_directory new_directory
[...]

So something like:

diff --recursive --brief </dev/dribeA> </dev/driveB>

Orr a mountpoint, or something.

[Aragorn]

@wind77

Both are mounted using ntfs3.

Don’t use ntfs3. It is known to screw up your NTFS filesystems. Use ntfs-3g instead.

[Mirdarthos]

Or better yet, ext4.

[wind77]

@Aragorn

Don’t use ntfs3. It is known to screw up your NTFS filesystems.

On a side note:
I tried renaming some 50 files using ntfs3 yesterday.
And none of them gone missing, like what happened previously.

Files missing during file transfer / renaming process

Hi. I have been plagued by a weird behavior, since the update in May 2023. When I perform rename in NTFS drive, files in same directory may gone missing/hidden. When I transfer files from one drive (EXT4 or NTFS) to another drive (NTFS), some files may gone missing. eg. If my source has 100 files, the destination may only have 50 files. According to folder properties, the size occupied is indeed reflecting the presence of 50 files only - so the missing 50 files is not there. But some times…

I suppose the issue has been fixed in kernel 6.4.9?

[Aragorn]

@wind77

I suppose the issue has been fixed in kernel 6.4.9?

I have no idea — I don’t use either Microsoft Windows or NTFS. But the advice I gave you still stands: if you want reliable NTFS support, use ntfs-3g instead of ntfs3. ntfs-3g actually works.

[wind77]

@Mirdarthos

Or better yet, ext4.

My storage HDDs are mostly media files.

ext4 would occupy >5% of storage space for journaling, and I dun require it for media storage.
and 5% means 1TB of space in a 20TB HDD.

@Aragorn

if you want reliable NTFS support, use ntfs-3g instead of ntfs3. ntfs-3g actually works.

Roger that!
Need to find time to reconfig and test it out.

[Mirdarthos]

@wind77

and 5% means 1TB of space in a 20TB HDD.

And, as you can now see, NTFS can maybe use less, but the potential lose 20TB of data is that much more…

[wind77]

Hmm… I dunno what to say.

I spent 6 hrs to transfer these ~4TB of files, and 8+ hrs to troubleshoot with FreeFileSync, and all these while, Dolphin reported:

This was reported BEFORE and AFTER comparison by FreeFileSync.

And now, suddenly, Dolphin reported this :

I have no idea how the missing files resurfaced, as I have done nothing…

[megavolt]

@wind77

ext4 would occupy >5% of storage space for journaling, and I dun require it for media storage.

Actually it is

1 % → file system, also called metadata
5% → reserved space for root
5-10% → ext4 journal

The only difference here to ntfs is that it has 5% reserved space, ntfs also have a journal at about the same size. The reserved space can be disabled, since it makes only sense if ext4 would be the root partition and if you don’t care about security, then journal can bis disabled also.

Disable reserved space:

sudo tune2fs -m 0 /dev/sdXX

Disable journal:

sudo tune2fs -O ^has_journal /dev/sdXX

Then you have more space, but less security. At the end it would be as reliable as fat32/exfat/ext2.

[Mirdarthos]

Also, I don’t know how recommended this is, but the journaling can be disabled:

https://foxutech.com/how-to-disable-enable-journaling/

And:

https://wiki.archlinux.org/title/Ext4#Disabling_journaling

[wind77]

This is my little experiment to see how ext4 would consume the space in comparison with ntfs.

@megavolt so for storage purpose, ext4 would use 1% for metadata and 5-10% for journal, and the journal can be disabled? In short, the FS/raw ratio for ext4 would be similar to ntfs?

@Mirdarthos

I don’t know how recommended this is

For pure storage of media, is there any advantage of journal?

[Aragorn]

@wind77

I have no idea how the missing files resurfaced, as I have done nothing…

Compression? NTFS does support inline compression.

Or, stuff may still have been in the pipeline. You have to keep in mind that disk I/O is asynchronous. The files are first written to the virtual filesystem layer, and only later this is synced to the underlying filesystem.

Another thing to keep in mind is that in some cases — or at least, such is reported by people here — dolphin doesn’t always automatically update the view.

[Mirdarthos]

@wind77

For pure storage of media, is there any advantage of journal?

I have not the faintest of ideas my dude…

[wind77]

@Aragorn

Compression?

I did not perform compression, nor get ntfs to perform inline compression.
And the files are mostly mkv.

[Aragorn]

@wind77

@Aragorn

Compression?

I did not perform compression, nor get ntfs to perform inline compression.
And the files are mostly mkv.

.mkv is a compressed format, albeit that this in and of itself doesn’t have anything to do with NTFS compression.

---

@wind77

For pure storage of media, is there any advantage of journal?

Yes, of course. It ensures the integrity of your files during copy/move/modify operations.

[freggel.doe]

@wind77

For pure storage of media, is there any advantage of journal?

If you think you do not need a journal - you can disable it for any ext4 filesystem.

[Aragorn]

Actually, one can disable it on just about every filesystem, except for xfs.