Caught in a mount/umount loop that prevented reboot

[Daniel-I]

The issue started after selecting to “Safely Remove”/eject my Ventoy USB stick from Dolphin. Normally I’d use the plasma Disk & Devices system tray tool, but because I’d just finished using Dolphin to update the stick with a new Manjaro image, I worked with where my mouse was closest.

What I noticed was that the system (opening Disk & Devices system tray tool) got stuck in the “removing” process. Normally it completes almost instantly, but after 30 seconds of waiting, I knew something wasn’t right… and as I needed the stick to load Linux on my friend’s PC, I pulled it out and used it successfully.

What I noticed after coming back to my PC was that my system refused to remount the Ventoy stick (but could mount/umount a different one). After some investigation, I noticed 2 things:

1. lsblk still reported the stick (/dev/sdd) and it’s partitions (sdd1 and sdd2) even though the disk was removed
2. the /run/media/<userid>/Ventoy mountpoint still existed

So I figured I’d reboot my PC normally (from the app launcher) and found myself stuck at a black screen. I switched over to TTY2, logged in, and tried to reboot there. It informed me it couldn’t due to two processes/actions that were still running… mount and umount… and suggested I try systemctl reboot -i to try ignore/bypass those inhibitors, which I did.

Now the screen filled with more “shutdown” processes (I prefer grub with “quiet” removed to see all boot/shutdown messages), and entered a 90 second countdown while it toggled between a D-Bus and Disk Manager messages… likely related to the mount/umount “inhibitors”… and after I waited out the 90 second countdown, it restarted the countdown again.

At this point I felt I was out of options and pressed reset.

Has anyone else experienced an issue like this? Is it a Dolphin Issue, or do USB3 sticks sometimes take more that 30 seconds to umount and I should have waited longer? Was there another command I could have tried using to release the drive or stop the inhibiting mount/umount? When I got stuck during shutdown/TTY2, was this the time to learn about and use REISUB, or would it have hit the same wall?

Here is an excerpt from the log post reset/reboot:

$ journalctl --priority err --boot -1
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158083088 op 0x1:(WRITE) flags 0x800 phys_seg 49 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760130, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760131, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760132, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760133, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760134, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760135, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760136, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760137, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760138, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: Buffer I/O error on dev sdd1, logical block 19760139, lost async page write
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158083480 op 0x1:(WRITE) flags 0x100000 phys_seg 43 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158083824 op 0x1:(WRITE) flags 0x800 phys_seg 34 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158084096 op 0x1:(WRITE) flags 0x100000 phys_seg 75 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158084696 op 0x1:(WRITE) flags 0x800 phys_seg 90 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158085416 op 0x1:(WRITE) flags 0x100000 phys_seg 12 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158085512 op 0x1:(WRITE) flags 0x800 phys_seg 17 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158085648 op 0x1:(WRITE) flags 0x100000 phys_seg 12 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158085744 op 0x1:(WRITE) flags 0x800 phys_seg 108 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux kernel: blk_update_request: I/O error, dev sdd, sector 158086608 op 0x1:(WRITE) flags 0x100000 phys_seg 68 prio class 0
Jan 04 13:36:17 AM4-5600X-Linux mount.exfat[2651602]: fsync failed: Input/output error
Jan 04 13:39:18 AM4-5600X-Linux systemd-udevd[383]: sdd1: Worker [2667044] processing SEQNUM=5528 killed
Jan 04 13:39:18 AM4-5600X-Linux systemd-udevd[383]: sdd1: Worker [2667044] failed
Jan 04 13:40:06 AM4-5600X-Linux kernel: INFO: task kworker/5:2:2329433 blocked for more than 122 seconds.
Jan 04 13:40:06 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:40:06 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:40:06 AM4-5600X-Linux kernel: INFO: task mount.exfat:2651602 blocked for more than 122 seconds.
Jan 04 13:40:06 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:40:06 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:42:09 AM4-5600X-Linux kernel: INFO: task kworker/5:2:2329433 blocked for more than 245 seconds.
Jan 04 13:42:09 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:42:09 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:42:09 AM4-5600X-Linux kernel: INFO: task mount.exfat:2651602 blocked for more than 245 seconds.
Jan 04 13:42:09 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:42:09 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:44:12 AM4-5600X-Linux kernel: INFO: task kworker/5:2:2329433 blocked for more than 368 seconds.
Jan 04 13:44:12 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:44:12 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:44:12 AM4-5600X-Linux kernel: INFO: task mount.exfat:2651602 blocked for more than 368 seconds.
Jan 04 13:44:12 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:44:12 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:46:15 AM4-5600X-Linux kernel: INFO: task kworker/5:2:2329433 blocked for more than 491 seconds.
Jan 04 13:46:15 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:46:15 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:46:15 AM4-5600X-Linux kernel: INFO: task mount.exfat:2651602 blocked for more than 491 seconds.
Jan 04 13:46:15 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:46:15 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:48:18 AM4-5600X-Linux kernel: INFO: task kworker/5:2:2329433 blocked for more than 614 seconds.
Jan 04 13:48:18 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:48:18 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 04 13:48:18 AM4-5600X-Linux kernel: INFO: task mount.exfat:2651602 blocked for more than 614 seconds.
Jan 04 13:48:18 AM4-5600X-Linux kernel:       Tainted: G           OE     5.15.12-1-MANJARO #1
Jan 04 13:48:18 AM4-5600X-Linux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[andreas85]

Gentle shut down a suddenly hanged PC to minimize a chance of getting a broken filesystem and data loss:

[HowTo] reboot / turn off your frozen computer: REISUB/REISUO

Difficulty: ★★☆☆☆ This tutorial was created because people are using a hard power off during freezes, so this will teach you how to turn off your computer when everything is frozen using softer means. So whenever you see someone saying they had to “Power Off” or “Hard Reset” their system without doing a REISUB, please send them here! If you’re reading this in response to a question, please click the green link above this text to bring you to the full unabridged text of the tutorial…

Provide Information:

[HowTo] Provide System Information

Welcome to Manjaro! To allow all of us helpful people on this forum TO HELP YOU, we need some basic information about your system. This probably means you described the symptoms of your problem, but now we need some more information to know where the origin of your issue is exactly. Someone else probably linked you to this tutorial, so if you’re reading this in response to a question, please click the link above this text to bring you to the full and unabridged text of the tutorial…

[Daniel-I]

Thank you for confirming this was a call for REISUB @andreas85

I’ve followed that post and enabled the SysRq key via GRUB… but I have a question about the key sequence.

Once you’ve located your SysRq key, please keep the Alt key pressed.

Now lightly tap these keys waiting between 1 second (fast, new machines) and 6 seconds (older or resource-starved machines¹) in-between keypresses : R E I S U B

Am I…

1. Pressing and holding Alt first and maintaining it held the whole time?
2. When I press SysRq, is it a tap, or am I holding it together with Alt the whole time?
3. If I’ve released (tapped) SysRq (while keeping Alt held), do I spell out REISUB slowly, or do I need to tap SysRq between each letter/command?

EDIT: I found a video @ The Magic SysRQ Key on the Keyboard - YouTube where he is following a guide @ Linux Magic System Request Key Hacks — The Linux Kernel documentation… the directions there state:

On x86

You press the key combo ALT-SysRq-.

Note: Some keyboards may not have a key labeled ‘SysRq’. The ‘SysRq’ key is also known as the ‘Print Screen’ key. Also some keyboards cannot handle so many keys being pressed at the same time, so you might have better luck with press Alt, press SysRq, release SysRq, press command key, release everything.

In the video it sounds like he is following the pre-Note directions, which I understood as being that each key combination is one command, so we need to do 6 combinations in total:

1. press/hold Alt + SysRq + R (in sequence) then release everything
2. press/hold Alt + SysRq + E (in sequence) then release everything
3. press/hold Alt + SysRq + I (in sequence) then release everything
4. press/hold Alt + SysRq + S (in sequence) then release everything
5. press/hold Alt + SysRq + U (in sequence) then release everything
6. press/hold Alt + SysRq + B (in sequence) then release everything

With the Note/exception being if your keyboard doesn’t like 3 keys held down at once to do this instead:

1. press/hold Alt, press/release (tap) SysRq, press R, release everything
2. press/hold Alt, press/release (tap) SysRq, press E, release everything
3. press/hold Alt, press/release (tap) SysRq, press I, release everything
4. press/hold Alt, press/release (tap) SysRq, press S, release everything
5. press/hold Alt, press/release (tap) SysRq, press U, release everything
6. press/hold Alt, press/release (tap) SysRq, press B, release everything

Is this verbosity more correct?

[andreas85]

@Daniel-I

• Pressing and holding Alt first and maintaining it held the whole time?
• When I press SysRq, is it a tap, or am I holding it together with Alt the whole time?
• If I’ve released (tapped) SysRq (while keeping Alt held), do I spell out REISUB slowly, or do I need to tap SysRq between each letter/command?

1. Yes
2. Only tap SysRq while holding Alt
3. While holding Alt spell out REISUB slowly (without SysRq )

On my keyboard SysRq is labeled Drucken

[Daniel-I]

Well I was able to repeat the issue, and learned I was just impatient and created my own problem.

I think some file transfers via Dolphin are still in process (not 100% finalized) once it says the transfer is complete. This time I’d copied 242 files @ 37GB over to an SD within a USB toaster/adapter. When Dolphin said it was finished, this time I used the plasma systray Disks & Devices to “Safely remove” and the Remove progress wheel spun for somewhere between 2-3 minutes until it finally completed, closed the plasma widget, and refreshed Dolphin.

So ultimately the Solution was a circumstantial two-parter:

1. don’t be impatient and yank the USB after 30 seconds… as even though the norm is to complete it in seconds, sometimes it can take a few minutes.
2. If you do get impatient and yank the drive anyway, expect issues that you’ll likely need to use Alt+SysRq+R-E-I-S-U-B to get out of cleanly

Note: I think the next time I do USB file transfers, I’m going to check the USB LED for flashing (and let that stop/catch-up) before racing to “Eject” the disk… because I feel that if it was still working/flashing (despite the plasma notification that the Dolphin transfer was complete), I would be creating contention for the disk between the in progress write finalization and the new ejecting/umount request.

[andreas85]

@Daniel-I

When Dolphin said it was finished,

Dolphin will say it is finished, after it has performed the complete copy.
But part of this is not jet written to disk. It is only in write-cache. A sync will write it to disk.