Hello, I’m new to Manjaro, (Latest stable) and am having an issue with large file transfers (Dolphin) to usb drives incorrectly being marked as completed after the caching phase is complete, not when the file is actually successfully written to the drive. I’ve been using the “safely remove” feature to give me some indication when the transfer is complete, but this seems a poor workaround to an issue that shouldn’t exist. Is there something I can configure to give me a proper file transfer dialog?
Dolphin large file transfer to usb stick indicates completed after caching is done, transfer still ongoing
Welcome to the forum!
That’s quite judgmental for someone obviously new to GNU/Linux.
You have to understand how these things work. The Linux kernel uses a virtual filesystem, onto which all physical filesystems are mounted.
So when I/O occurs, then what really happens is that this I/O takes place within the virtual filesystem layer — in what is called buffers, the opposite of cache memory — and gets synchronized only with the physical filesystem underneath at a later point, when the system is less busy.
Other factors in this regard are the speed of the bus through which the physical device is connected, as well as the I/O throughput of the physical device itself. USB sticks are not known for high throughput, because they use a different kind of flash storage when compared to an SSD.
Now, there is a way to cut out (most of) the memory buffering during writes, but for that, you would need to manually mount the USB stick, instead of relying on the automount feature, because you can mount a filesystem with the
sync option. This option means that all I/O must forego the buffers and be immediately written to the underlying physical medium.
However, in many cases, this actually makes performance worse, because now all of the data to be written to the device is being pushed down the USB bus, and your device cannot handle all that data at once. This is going to hamper the overall performance of your computer, because I/O is a high-priority process and cannot be interrupted.
So your computer is going to crawl to a halt until all of the data has made it onto your USB stick. And even then still, it’ll have to buffer some of it due to the USB stick itself being the I/O bottleneck.
This is why the option to safely unmount a connected filesystem exists. So instead of your claim that it shouldn’t exist, it very much should exist after all.
Thanks very much for the informative reply. To clarify, I do appreciate the need for the “safely remove” option, and use it in other operating systems. I do object to being notified that the transfer has completed before it actually has been completed. Is it possible for the notification to be based on data written to the drive itself without killing IO performance?
I have to echo the last post. I hate when the copy / move dialog finishes and closes, and knowing just from watching the light on the flash drive continue to flash for another minute or two after the dialog closed tells me that data is still being copied / moved to the flash drive. There must be a way to have the copy / move dialog not close til all data is finished copying / moving.
I don’t think so. The notification is a userspace process, while the actual physical I/O is a kernelspace process. But perhaps you can file a feature request at bugs.kde.org.
As Aragorn said you can kind of get around this “issue” but in the end, it will still take the same amount of time to complete the task. A little patience goes a long way.
It’s not a matter of the time the issue was and is the fact that even though it seems to be done copying / moving it isn’t. At least as far as I’m concerned.
Perhaps this will help.
OK maybe this will help you USB Write Caching Explained - Windows, Mac, & Linux
The filesystem used on the USB stick itself makes a big difference here.
I use a lot of different USB sticks and these days they tend to come formatted either as FAT32 or exFAT, with larger drives tending more toward exFAT. Caching behaviour differs significantly on sticks with different formats.
I found several of my 64GB sticks came exFAT formatted and transfer was insanely fast (because it just loaded up the cache) and then everything just hung - nasty.
I reformatted the stick to FAT32 and transfer was much slower (because it was real) and the cache effect was very small (10 - 20 seconds).
This isn’t really a Linux issue as such either - Certainly with exFAT - Windows used to get very upset with exFAT if anything goes slightly untoward during transfer - I’ve had some nasty exFAT experiences there, but I think it’s been rectified more recently; MacOS too.
Yeah exactly, im experience the same problem… the copy+move dialog for a finished task comes much to early… and its not only related to a flash drive, i also has this with external usb drives… i just cant trust this dialogs… its not a reliable info.
Ideally you want it to sort of “wait” for a bit so you can see the cache is still active and go away when it’s finished the actual write. I’ve not tried writing to an external ext4 system, so I don’t know how that would look from a caching perspective - I’ve found that, from my Manjaro box, FAT32 USB Sticks are OK and are “done” around 10 seconds after it says they are done, exFAT USB sticks are absolutely crazy (see above) and my larger drive (which I think is NTFS) seems to be fine.
Consistent it is not, but does that depend to an extent on the hardware on the other side of the USB interface?
Then look at the link I posted. If I remember correctly, it suggests changing the amount of data that is stored before a flush/write is triggered. The idea is that when the dialog disappears there’s not a lot of data left to be written and it finishes quickly.
Not sure why it matters anyway, if you use a terminal then you don’t even have the dialog, just
umount. If you use the GUI you should be using the safely remove feature (or
sync if appropriate) anyway.
I think that article is confusing the write cache (buffer really) on storage devices and the virtual filesystem that @Aragorn talked about which exists in the system’s memory.
It talks about the cache being in system RAM but then tells you to edit a non-existent config file for
hdparm deals with storage devices, AFAIK it can only turn off the write cache on the drive(s).
to sum up my efforts to tackle the issue, stop wasting time to make this right. you can pull all hairs you got only return to the way things are, all that will be left will be the whining you did. all OS’ including windows (to some degree) has this issue you just notice it more on KDE since there is this notification at the end of supposed file transfer.
This is the correct method.
When the device name/id is removed from dolphin the transfer is complete.
Linux and by that inheritance Manjaro uses filecaching more aggressively than Windows which counter for the delay between the message and the actual completion.
You can fiddle with your system settings - there is plenty topics on the subject - just use the search function in top right corner or top right corner of manjaro.org.
BULL this day and age the USB key should be treated no differently than a internal drive, meaning when the file dialog closes it should be done and one should be able to just remove the key right then and not have to wait for use a special program to tell one that it’s safe to remove the key.
They’re all the same. There are 2 differences, the speed of the device, and the fact you don’t usually remove an internal drive whilst the machine is running.
You actually seem to want them treated differently.
You can’t just write to an internal drive and then pull it out. Here’s how you write to a storage device:
- Mount it.
- Write the data.
- It’s now safe to remove.
It doesn’t matter if you’re using the GUI or the terminal, or if the drive is internal or external the process is the same. Assuming SATA hotplug is enabled, otherwise you’ll need to turn off the machine for internal drives.
Safely remove syncs, unmounts and detaches (deletes the device file for) the device. It’s not special software, it’s how you use the GUI to safely remove (ie without data loss) a storage device.
You can sometimes get away with an unsafe removal, but that doesn’t mean you should do it.
You’ve already been given 2 options.
- Mount using sync.
- Change dirty bytes so less data needs to be written after the dialog closes.
They’re the closest you’re gonna get to what you want. I’d just remove it properly if I were you.
Lets try that again way too many times I copy my data to my flash drive rather Linux or Windows and when I’m ready to go I just unplug it. Never have had an issue with it.
Perhaps someone should invent teeth that grabs the metal connector of the USB stick from the inside and refuses to let go until data is written?
Wouldn’t that be fun? Bite marks on your USB stick! The kids go: “I am not touching that computer again!”
I noticed this issue when moving linux .iso files onto a ventoy usb stick. (I just checked, and the .iso partition is formatted exfat, thanks tiefling!) I copied and pasted the .iso from an internal ssd to an older 32gb usb 3.0 drive, the transfer dialogue appeared for about 2 seconds, then indicated completion. Being not completely tech illiterate, I suspected what was happening and decided to use the safely remove function to see how out of sync the dialogue and reality were. It varies with the size of the file of course…but it can be 10 minutes early.
After reading the replies in the thread it has become clear that this isn’t a DE problem, but actually is an issue with the way that the filesystem is abstracted from userspace in linux in general. (I might be very wrong here?) I’m going to try some of the suggestions folks here have made to reduce the amount of error in the completion dialogue.
Now I imagine my parents, or my neighbour, or some of my friends running one of these new and polished distros, and I know this would result in data loss/corruption, probably making them averse to running linux at all. I went to the kde bug report page, where I found this issue had been reported late 2011, and the only thing that has been done is to add a warning not to unplug the device on a page that you actively have to seek out.
Imagine if I could plug in a removable device, it mounts automatically, I transfer a file to it, and when that transfer is complete, and the device is in a state where it is safe to yank without unmounting, it tells me that the transfer is complete. (I say without unmounting because if the typical user didn’t manually mount it in the first place, and if the OS automatically mounts a removeable device, IMO it better assume that the typical user will not unmount it manually.)
That said, I have been extremely impressed by the ease of installation and the amount of polish and QOL enhancements that have gone into Manjaro, and some other distros that I’ve been trying out.
I hope to settle on a “daily-driver” distro shortly, because windows is becoming increasingly less attractive.
EDIT: I just read the write caching article that fbt89 linked (Thanks!) I think that the windows default disabling of caching on removeable drives is much saner, especially if adoption by less technical users is a goal.