Dolphin large file transfer to usb stick indicates completed after caching is done, transfer still ongoing

Thanks very much for the informative reply. To clarify, I do appreciate the need for the “safely remove” option, and use it in other operating systems. I do object to being notified that the transfer has completed before it actually has been completed. Is it possible for the notification to be based on data written to the drive itself without killing IO performance?

1 Like

I have to echo the last post. I hate when the copy / move dialog finishes and closes, and knowing just from watching the light on the flash drive continue to flash for another minute or two after the dialog closed tells me that data is still being copied / moved to the flash drive. There must be a way to have the copy / move dialog not close til all data is finished copying / moving.

2 Likes

I don’t think so. The notification is a userspace process, while the actual physical I/O is a kernelspace process. But perhaps you can file a feature request at bugs.kde.org.

As Aragorn said you can kind of get around this “issue” but in the end, it will still take the same amount of time to complete the task. A little patience goes a long way.

1 Like

It’s not a matter of the time the issue was and is the fact that even though it seems to be done copying / moving it isn’t. At least as far as I’m concerned.

1 Like

Perhaps this will help.

OK maybe this will help you USB Write Caching Explained - Windows, Mac, & Linux

The filesystem used on the USB stick itself makes a big difference here.

I use a lot of different USB sticks and these days they tend to come formatted either as FAT32 or exFAT, with larger drives tending more toward exFAT. Caching behaviour differs significantly on sticks with different formats.

I found several of my 64GB sticks came exFAT formatted and transfer was insanely fast (because it just loaded up the cache) and then everything just hung - nasty.

I reformatted the stick to FAT32 and transfer was much slower (because it was real) and the cache effect was very small (10 - 20 seconds).

This isn’t really a Linux issue as such either - Certainly with exFAT - Windows used to get very upset with exFAT if anything goes slightly untoward during transfer - I’ve had some nasty exFAT experiences there, but I think it’s been rectified more recently; MacOS too.

Yeah exactly, im experience the same problem… the copy+move dialog for a finished task comes much to early… and its not only related to a flash drive, i also has this with external usb drives… i just cant trust this dialogs… its not a reliable info.

Ideally you want it to sort of “wait” for a bit so you can see the cache is still active and go away when it’s finished the actual write. I’ve not tried writing to an external ext4 system, so I don’t know how that would look from a caching perspective - I’ve found that, from my Manjaro box, FAT32 USB Sticks are OK and are “done” around 10 seconds after it says they are done, exFAT USB sticks are absolutely crazy (see above) and my larger drive (which I think is NTFS) seems to be fine.

Consistent it is not, but does that depend to an extent on the hardware on the other side of the USB interface?

Then look at the link I posted. If I remember correctly, it suggests changing the amount of data that is stored before a flush/write is triggered. The idea is that when the dialog disappears there’s not a lot of data left to be written and it finishes quickly.

Not sure why it matters anyway, if you use a terminal then you don’t even have the dialog, just sync and umount. If you use the GUI you should be using the safely remove feature (or sync if appropriate) anyway.

@fbt89

I think that article is confusing the write cache (buffer really) on storage devices and the virtual filesystem that @Aragorn talked about which exists in the system’s memory.

It talks about the cache being in system RAM but then tells you to edit a non-existent config file for hdparm. However hdparm deals with storage devices, AFAIK it can only turn off the write cache on the drive(s).

1 Like

to sum up my efforts to tackle the issue, stop wasting time to make this right. you can pull all hairs you got only return to the way things are, all that will be left will be the whining you did. all OS’ including windows (to some degree) has this issue you just notice it more on KDE since there is this notification at the end of supposed file transfer.

1 Like

This is the correct method.

When the device name/id is removed from dolphin the transfer is complete.

Linux and by that inheritance Manjaro uses filecaching more aggressively than Windows which counter for the delay between the message and the actual completion.

You can fiddle with your system settings - there is plenty topics on the subject - just use the search function :mag: in top right corner or top right corner of manjaro.org.

1 Like

BULL this day and age the USB key should be treated no differently than a internal drive, meaning when the file dialog closes it should be done and one should be able to just remove the key right then and not have to wait for use a special program to tell one that it’s safe to remove the key.

They’re all the same. There are 2 differences, the speed of the device, and the fact you don’t usually remove an internal drive whilst the machine is running.

You actually seem to want them treated differently.

You can’t just write to an internal drive and then pull it out. Here’s how you write to a storage device:

  1. Mount it.
  2. Write the data.
  3. Sync.
  4. Unmount.
  5. It’s now safe to remove.

It doesn’t matter if you’re using the GUI or the terminal, or if the drive is internal or external the process is the same. Assuming SATA hotplug is enabled, otherwise you’ll need to turn off the machine for internal drives.

Safely remove syncs, unmounts and detaches (deletes the device file for) the device. It’s not special software, it’s how you use the GUI to safely remove (ie without data loss) a storage device.

You can sometimes get away with an unsafe removal, but that doesn’t mean you should do it.

EDIT:

You’ve already been given 2 options.

  1. Mount using sync.
  2. Change dirty bytes so less data needs to be written after the dialog closes.

They’re the closest you’re gonna get to what you want. I’d just remove it properly if I were you.

2 Likes

Lets try that again way too many times I copy my data to my flash drive rather Linux or Windows and when I’m ready to go I just unplug it. Never have had an issue with it.

Perhaps someone should invent teeth that grabs the metal connector of the USB stick from the inside and refuses to let go until data is written?
Wouldn’t that be fun? Bite marks on your USB stick! The kids go: “I am not touching that computer again!”

I noticed this issue when moving linux .iso files onto a ventoy usb stick. (I just checked, and the .iso partition is formatted exfat, thanks tiefling!) I copied and pasted the .iso from an internal ssd to an older 32gb usb 3.0 drive, the transfer dialogue appeared for about 2 seconds, then indicated completion. Being not completely tech illiterate, I suspected what was happening and decided to use the safely remove function to see how out of sync the dialogue and reality were. It varies with the size of the file of course…but it can be 10 minutes early.

After reading the replies in the thread it has become clear that this isn’t a DE problem, but actually is an issue with the way that the filesystem is abstracted from userspace in linux in general. (I might be very wrong here?) I’m going to try some of the suggestions folks here have made to reduce the amount of error in the completion dialogue.

Now I imagine my parents, or my neighbour, or some of my friends running one of these new and polished distros, and I know this would result in data loss/corruption, probably making them averse to running linux at all. I went to the kde bug report page, where I found this issue had been reported late 2011, and the only thing that has been done is to add a warning not to unplug the device on a page that you actively have to seek out.

Imagine if I could plug in a removable device, it mounts automatically, I transfer a file to it, and when that transfer is complete, and the device is in a state where it is safe to yank without unmounting, it tells me that the transfer is complete. (I say without unmounting because if the typical user didn’t manually mount it in the first place, and if the OS automatically mounts a removeable device, IMO it better assume that the typical user will not unmount it manually.)

That said, I have been extremely impressed by the ease of installation and the amount of polish and QOL enhancements that have gone into Manjaro, and some other distros that I’ve been trying out.
I hope to settle on a “daily-driver” distro shortly, because windows is becoming increasingly less attractive.

EDIT: I just read the write caching article that fbt89 linked (Thanks!) I think that the windows default disabling of caching on removeable drives is much saner, especially if adoption by less technical users is a goal.

I’ve been doing some testing with different usb drive formats, and also the maxperfwiz script by cscs, and it does look like the exfat format is the most problematic with stock settings. (Completion notification after seconds, actual write completion long afterward)
At least with fat32, while the progress bar reaches 100 percent almost immediately, I don’t get the completion notification until the file is actually written to the device.
I haven’t tried any other formats yet.

With the maxperfwiz settings (yes to all tweaks) behaviour is much improved with actually realistic transfer rates after the buffering to ram is done, even with exfat, which still “completes” slightly early. I haven’t noticed any major system load increase with htop or any transfer speed reduction.

1 Like

For those of you that remember the Zip drives I had one I swear you wanted a hardhat every time you ejected the disk, cause it would be shot across the room if you didn’t have your hand there to stop it.