"Unmounting timed out." 90 sec wait for my mounts to try and unmount during shutdown

jackdinn · 1 October 2023 16:59

I have had this problem before on a completely different system, but i can’t for the life of me remember how I fixed it.
I have searched and read, but variations of this problem go back for years, with all sorts of suggestions.

Im sure there was a simple remedy, but like an idiot i don’t seem to have taken a note of what it was at the time?

My info & question:-

❱inxi --full --verbosity=7 --filter --no-host
System:
  Kernel: 6.5.3-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc Desktop: KDE Plasma v: 5.27.8 tk: Qt v: 5.15.10
    wm: kwin_x11 vt: 2 dm: SDDM Distro: Manjaro Linux base: Arch Linux
Machine:
  Type: Desktop Mobo: Gigabyte model: B85-HD3 serial: <superuser required>
    UEFI: American Megatrends v: FC date: 01/17/2014
Battery:
  Message: No system battery data found. Is one present?
Memory:
  System RAM: total: 16 GiB note: est. available: 15.44 GiB
    used: 5.06 GiB (32.7%)
  RAM Report: permissions: Unable to run dmidecode. Root privileges
    required.
CPU:
  Info: quad core model: Intel Core i7-4790 bits: 64 type: MT MCP smt: enabled
    arch: Haswell rev: 3 cache: L1: 256 KiB L2: 1024 KiB L3: 8 MiB
  Speed (MHz): avg: 2379 high: 3991 min/max: 800/4000 cores: 1: 3991 2: 3899
    3: 800 4: 800 5: 3972 6: 800 7: 3975 8: 800 bogomips: 57493
  Flags: abm acpi aes aperfmperf apic arat arch_perfmon avx avx2 bmi1 bmi2
    bts clflush cmov constant_tsc cpuid cpuid_fault cx16 cx8 de ds_cpl dtes64
    dtherm dts ept ept_ad erms est f16c flexpriority flush_l1d fma fpu
    fsgsbase fxsr ht ibpb ibrs ida invpcid invpcid_single lahf_lm lm mca mce
    md_clear mmx monitor movbe msr mtrr nonstop_tsc nopl nx pae pat pbe pcid
    pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pti pts rdrand
    rdtscp rep_good sdbg sep smep smx ss ssbd sse sse2 sse4_1 sse4_2 ssse3
    stibp syscall tm tm2 tpr_shadow tsc tsc_adjust tsc_deadline_timer vme vmx
    vnmi vpid x2apic xsave xsaveopt xtopology xtpr
Graphics:
  Device-1: Intel Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics
    vendor: Gigabyte driver: i915 v: kernel arch: Gen-7.5 ports: active: VGA-1
    empty: HDMI-A-1,HDMI-A-2 bus-ID: 00:02.0 chip-ID: 8086:0412 class-ID: 0300
  Display: x11 server: X.Org v: 21.1.8 with: Xwayland v: 23.2.0
    compositor: kwin_x11 driver: X: loaded: modesetting alternate: fbdev,vesa
    dri: crocus gpu: i915 display-ID: :0 screens: 1
  Screen-1: 0 s-res: 1920x1080 s-dpi: 96 s-size: 508x285mm (20.00x11.22")
    s-diag: 582mm (22.93")
  Monitor-1: VGA-1 model: VA24D serial: <filter> res: 1920x1080 hz: 60
    dpi: 93 size: 527x296mm (20.75x11.65") diag: 604mm (23.8") modes:
    max: 1920x1080 min: 720x400
  API: EGL v: 1.5 hw: drv: intel crocus platforms: device: 0 drv: crocus
    device: 1 drv: swrast surfaceless: drv: crocus x11: drv: crocus
    inactive: gbm,wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: intel mesa v: 23.1.7-1 glx-v: 1.4
    direct-render: yes renderer: Mesa Intel HD Graphics 4600 (HSW GT2)
    device-ID: 8086:0412
  API: Vulkan v: 1.3.264 layers: 1 surfaces: xcb,xlib device: 0
    type: integrated-gpu driver: mesa intel device-ID: 8086:0412
Audio:
  Device-1: Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio
    driver: snd_hda_intel v: kernel bus-ID: 00:03.0 chip-ID: 8086:0c0c
    class-ID: 0403
  Device-2: Intel 8 Series/C220 Series High Definition Audio
    vendor: Gigabyte 8 driver: snd_hda_intel v: kernel bus-ID: 00:1b.0
    chip-ID: 8086:8c20 class-ID: 0403
  API: ALSA v: k6.5.3-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator
  Server-1: sndiod v: N/A status: off
  Server-2: JACK v: 1.9.22 status: off
  Server-3: PipeWire v: 0.3.80 status: off with: pipewire-media-session
    status: active
  Server-4: PulseAudio v: 16.1 status: active with: pulseaudio-alsa
    type: plugin
Network:
  Device-1: Intel Wireless 8260 driver: iwlwifi v: kernel pcie:
    speed: 2.5 GT/s lanes: 1 bus-ID: 02:00.0 chip-ID: 8086:24f3 class-ID: 0280
  IF: wlp2s0 state: up mac: <filter>
  IP v4: <filter> type: dynamic noprefixroute scope: global
    broadcast: <filter>
  IP v6: <filter> type: noprefixroute scope: link
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: Gigabyte driver: r8169 v: kernel pcie: speed: 2.5 GT/s lanes: 1
    port: e000 bus-ID: 03:00.0 chip-ID: 10ec:8168 class-ID: 0200
  IF: enp3s0 state: down mac: <filter>
  WAN IP: <filter>
Bluetooth:
  Device-1: Edimax Bluetooth Adapter driver: btusb v: 0.8 type: USB rev: 1.1
    speed: 12 Mb/s lanes: 1 bus-ID: 3-10.3.1:5 chip-ID: 7392:c611 class-ID: e001
    serial: <filter>
  Report: btmgmt ID: hci1 rfk-id: 1 state: up address: <filter> bt-v: 5.1
    lmp-v: 10 class-ID: 7c0104
  Device-2: Intel Bluetooth wireless interface driver: btusb v: 0.8
    type: USB rev: 2.0 speed: 12 Mb/s lanes: 1 bus-ID: 3-8:2 chip-ID: 8087:0a2b
    class-ID: e001
  Report: ID: hci0 rfk-id: 0 state: up address: <filter> bt-v: 4.2 lmp-v: 8
    class-ID: 7c0104
Logical:
  Message: No logical block device data found.
RAID:
  Message: No RAID data found.
Drives:
  Local Storage: total: 2.73 TiB used: 112.69 GiB (4.0%)
  ID-1: /dev/sda vendor: Samsung model: SSD 870 EVO 1TB size: 931.51 GiB
    speed: 6.0 Gb/s tech: SSD serial: <filter> fw-rev: 3B6Q scheme: GPT
  ID-2: /dev/sdb vendor: Western Digital model: WD20EZRX-00D8PB0
    size: 1.82 TiB speed: 6.0 Gb/s tech: HDD rpm: 5400 serial: <filter>
    fw-rev: 0A80 scheme: MBR
  Message: No optical or floppy data found.
Partition:
  ID-1: / size: 915.53 GiB used: 112.69 GiB (12.3%) fs: ext4 dev: /dev/sda2
    label: N/A uuid: 98072a66-4fb4-4f36-b74d-77005a71c528
  ID-2: /boot/efi size: 299.4 MiB used: 288 KiB (0.1%) fs: vfat
    dev: /dev/sda1 label: N/A uuid: 80B1-86C0
  ID-3: /mnt/TRUENAS-cavern size: 5.79 TiB used: 242.42 GiB (4.1%) fs: cifs
    dev: /dev/cavern
  ID-4: /mnt/TRUENAS-home size: 5.58 TiB used: 24.63 GiB (0.4%) fs: cifs
    dev: /dev/home
  ID-5: /mnt/TRUENAS-manjaro-home size: 5.71 TiB used: 157.82 GiB (2.7%)
    fs: cifs dev: /dev/manjaro-home
  ID-6: /mnt/TRUENAS-media size: 5.92 TiB used: 379.08 GiB (6.2%) fs: cifs
    dev: /dev/media
Swap:
  ID-1: swap-1 type: file size: 29.3 GiB used: 0 KiB (0.0%) priority: -2
    file: /swapfile
Unmounted:
  ID-1: /dev/sdb1 size: 1.72 TiB fs: ntfs label: E_Drive
    uuid: 5C9420E29420C07E
USB:
  Hub-1: 1-0:1 info: full speed or root hub ports: 2 rev: 2.0 speed: 480 Mb/s
    lanes: 1 chip-ID: 1d6b:0002 class-ID: 0900
  Hub-2: 1-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0
    speed: 480 Mb/s lanes: 1 chip-ID: 8087:8000 class-ID: 0900
  Hub-3: 2-0:1 info: full speed or root hub ports: 2 rev: 2.0
    speed: 480 Mb/s lanes: 1 chip-ID: 1d6b:0002 class-ID: 0900
  Hub-4: 2-1:2 info: Intel Integrated Rate Matching Hub ports: 6 rev: 2.0
    speed: 480 Mb/s lanes: 1 chip-ID: 8087:8008 class-ID: 0900
  Hub-5: 3-0:1 info: hi-speed hub with single TT ports: 12 rev: 2.0
    speed: 480 Mb/s lanes: 1 chip-ID: 1d6b:0002 class-ID: 0900
  Device-1: 3-8:2 info: Intel Bluetooth wireless interface type: bluetooth
    driver: btusb interfaces: 2 rev: 2.0 speed: 12 Mb/s lanes: 1 power: 100mA
    chip-ID: 8087:0a2b class-ID: e001
  Hub-6: 3-10:3 info: Terminus Hub ports: 4 rev: 2.0 speed: 480 Mb/s
    lanes: 1 power: 100mA chip-ID: 1a40:0101 class-ID: 0900
  Hub-7: 3-10.3:4 info: Terminus Hub ports: 4 rev: 2.0 speed: 480 Mb/s
    lanes: 1 power: 100mA chip-ID: 1a40:0101 class-ID: 0900
  Device-1: 3-10.3.1:5 info: Edimax Bluetooth Adapter type: bluetooth
    driver: btusb interfaces: 2 rev: 1.1 speed: 12 Mb/s lanes: 1 power: 500mA
    chip-ID: 7392:c611 class-ID: e001 serial: <filter>
  Device-2: 3-10.3.2:6 info: Integrated Express HTX HID Device
    type: keyboard,mouse driver: hid-generic,hid-multitouch,usbhid interfaces: 2
    rev: 2.0 speed: 12 Mb/s lanes: 1 power: 100mA chip-ID: 048d:8911
    class-ID: 0301 serial: <filter>
  Device-3: 3-10.3.3:7 info: Dell USB Wired Entry Keyboard
    type: keyboard,HID driver: hid-generic,usbhid interfaces: 2 rev: 1.1
    speed: 1.5 Mb/s lanes: 1 power: 100mA chip-ID: 413c:2111 class-ID: 0300
  Hub-8: 4-0:1 info: super-speed hub ports: 6 rev: 3.0 speed: 5 Gb/s
    lanes: 1 chip-ID: 1d6b:0003 class-ID: 0900
Sensors:
  System Temperatures: cpu: 36.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Processes: 249 Uptime: 2h 11m wakeups: 0 Init: systemd v: 254
  default: graphical Compilers: gcc: 13.2.1 clang: 16.0.6 Packages: pm: pacman
  pkgs: 1352 Shell: fish v: 3.6.1 default: Bash v: 5.1.16 running-in: yakuake
  inxi: 3.3.30

I have several NAS mounts mounted at boot with fstab

❱cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a device; this may
# be used with UUID= as a more robust way to name devices that works even if
# disks are added and removed. See fstab(5).
#
# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=xxxxxxxx                            /boot/efi      vfat    umask=0077 0 2
UUID=xxxxxxxxxxxxxxxxxxxxxx /              ext4    defaults,noatime 0 1
/swapfile none swap defaults 0 0
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0
//192.168.1.100/manjaro-home /mnt/TRUENAS-manjaro-home cifs vers=2.0,user,nofail,cache=loose,credentials=/home/greg/.smbcredentials-nas,iocharset=utf8,uid=1000,gid=1000,noperm,_netdev 0 0
//192.168.1.100/cavern /mnt/TRUENAS-cavern cifs vers=2.0,user,nofail,cache=loose,credentials=/home/greg/.smbcredentials-nas,iocharset=utf8,uid=1000,gid=1000,noperm,_netdev 0 0
//192.168.1.100/home /mnt/TRUENAS-home cifs vers=2.0,user,nofail,cache=loose,credentials=/home/greg/.smbcredentials-nas,iocharset=utf8,uid=1000,gid=1000,noperm,_netdev 0 0
//192.168.1.100/media /mnt/TRUENAS-media cifs vers=2.0,user,nofail,cache=loose,credentials=/home/greg/.smbcredentials-media,iocharset=utf8,uid=1000,gid=1000,noperm,_netdev 0 0

>mount
...
//192.168.1.100/manjaro-home on /mnt/TRUENAS-manjaro-home type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=loose,username=jack,uid=1000,noforceuid,gid=1000,noforcegid,addr=192.168.1.100,file_mode=0755,dir_mode=0755,iocharset=utf
8,soft,nounix,serverino,mapposix,noperm,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1,user,_netdev)
//192.168.1.100/cavern on /mnt/TRUENAS-cavern type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=loose,username=jack,uid=1000,noforceuid,gid=1000,noforcegid,addr=192.168.1.100,file_mode=0755,dir_mode=0755,iocharset=utf8,soft,nouni
x,serverino,mapposix,noperm,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1,user,_netdev)
//192.168.1.100/home on /mnt/TRUENAS-home type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=loose,username=jack,uid=1000,noforceuid,gid=1000,noforcegid,addr=192.168.1.100,file_mode=0755,dir_mode=0755,iocharset=utf8,soft,nounix,se
rverino,mapposix,noperm,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1,user,_netdev)
//192.168.1.100/media on /mnt/TRUENAS-media type cifs (rw,nosuid,nodev,relatime,vers=2.0,cache=loose,username=qbit,uid=1000,noforceuid,gid=1000,noforcegid,addr=192.168.1.100,file_mode=0755,dir_mode=0755,iocharset=utf8,soft,nounix,serveri
no,mapposix,noperm,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

This is journalctl with just the 90 sec delay area shown between the time 15:22 & 15:24

❱journalctl -b -1 
...
Oct 01 15:22:37 greg-b85hd3 systemd[1]: run-user-1000.mount: Deactivated successfully.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: Unmounted /run/user/1000.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: user-runtime-dir@1000.service: Deactivated successfully.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: Stopped User Runtime Directory /run/user/1000.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: Removed slice User Slice of UID 1000.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: user-1000.slice: Consumed 1h 34min 51.176s CPU time.
Oct 01 15:22:37 greg-b85hd3 systemd[1]: Stopped target Remote File Systems.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dhome.mount: Unmounting timed out. Terminating.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dmedia.mount: Unmounting timed out. Terminating.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dhome.mount: Mount process exited, code=killed, status=15/TERM
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dhome.mount: Failed with result 'timeout'.
Oct 01 15:24:06 greg-b85hd3 kernel: CIFS: VFS: \\192.168.1.100 __cifs_put_smb_ses: Session Logoff failure rc=-512
Oct 01 15:24:06 greg-b85hd3 kernel: CIFS: VFS: \\192.168.1.100 __cifs_put_smb_ses: Session Logoff failure rc=-512
Oct 01 15:24:06 greg-b85hd3 systemd[1]: Unmounted /mnt/TRUENAS-home.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dmedia.mount: Mount process exited, code=killed, status=15/TERM
Oct 01 15:24:06 greg-b85hd3 systemd[1]: mnt-TRUENAS\x2dmedia.mount: Failed with result 'timeout'.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: Unmounted /mnt/TRUENAS-media.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: Stopped target Network is Online.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: Stopped target Network.
Oct 01 15:24:06 greg-b85hd3 systemd[1]: NetworkManager-wait-online.service: Deactivated successfully.

so Its failing to umount some of the NAS. The NAS is idle, im 100% its not being used by anything at the time. How to fix?

soundofthunder · 1 October 2023 17:34

This is only a wild guess – I can’t help beyond that – but, it seems like your remote system (NAS) is dropping the connection; which might explain the unmounting timeout and the subsequent logoff session failure. It might be time to check your cables and connections for intermittent faults. Cheers.

jackdinn · 1 October 2023 17:45

Thx for the response but it not anything to do with the nas, Its when im trying to shut this system down, it hangs with the message “Cant unmount xxxxxx” wait xx seconds" , I dont have any trouble other than when i try t shutdown/restart this system.

Iv seen it before on another system of mine but i found a fix, and now it does not do it any more on that other system but i must have forgotten to write it down as i can’t find what i did to fix it in any of my notes.

soundofthunder · 1 October 2023 17:48

It seemed a possibility, at least, that the unmount doesn’t occur fundamentally due to not receiving either the instruction, or the confirmation back from the NAS. Good luck.

jackdinn · 1 October 2023 17:51

I can umount them all manually and then shutdown fine, its only when the system tried to shutdown and umount itself that it fails.

Thx.

Xephon · 1 October 2023 17:57

You could try a script executed on shutdown that will perform unmount of your network drives with --lazy option.
Or try setting DefaultTimeoutStartSec= to 10-15s instead of 90s in /etc/systemd/system.conf.d like described here

jackdinn · 1 October 2023 18:01

well i just tried this and then it shutdown fine.

❱sudo umount /mnt/TRUENAS-manjaro-home; sudo umount /mnt/TRUENAS-cavern; sudo umount /mnt/TRUENAS-home; sudo umount /mnt/TRUENAS-media

Not sure what/where the --lazy option goes or where to put a script to automate it at shutdown?
maybe make a script in /lib/systemd/system-shutdown/ ?

soundofthunder · 1 October 2023 18:04

Welcome.

Edit:- Sorry, wrong page. It should have been this, specifically under the heading “The problem is journald”.

Regarding running a script at shutdown, maybe this post will be helpful;

Cheers.

jackdinn · 1 October 2023 18:37

well i tried

Edit /etc/systemd/journald.conf to change the Storage= line to

Storage=volatile

It didn’t work, still get this at shutdown (this is after a reboot (well 2 reboots)).

I shall try this next but I’m not sure how?

Add a shutdown hook to mkinitcpio’s HOOKS= array, e.g. :

HOOKS=“base udev autodetect block keymap encrypt lvm2 filesystems usbinput fsck var shutdown”

jackdinn · 1 October 2023 19:00

that didnt work either should i just leave /etc/mkinitcpio.conf as it is with “shutdown” at the end of the list or what?

Anyhow, Ill try just reducing the timeout. Is DefaultTimeoutStartSec all i need to change? what about the others, like:-
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultTimeoutAbortSec=
#DefaultDeviceTimeoutSec=90s

EDIT:-

Ok so DefaultTimeoutStartSec=10 works ^^ , its not really a good solution though as its just hiding and forcing the unmount. What if there was actual data flowing at the time it might be corrupt. I dont know what else to do though.

O and it /etc/systemd/system.conf not /etc/systemd/system.conf.d

megavolt · 1 October 2023 19:22

Hello @jackdinn

Looks to me that it fails because Wifi is offline first and then tries to unmount a network share when it’s offline. Obviously, that will not work.

Probably your wifi profile is not a global one. That means: after graphical logout, it will be offline first.

Make it global in NetworkManager: Tick “All users may connect to this network” when you use network shares globally in fstab.

jackdinn · 1 October 2023 19:31

Not sure why you are looking at my either-net connection enp3s0 as i dont use it. Anyhow, I have changed it for wlp2s0 and shall give it a go.

❱ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 74:d4:35:99:41:76 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:ed:5c:a2:d3:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.106/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 85558sec preferred_lft 85558sec
    inet6 fe80::d364:d83c:cd7d:1b36/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

If this works (or not even) id still like to know if i should undo all the things i have tried so for? Especially for mkinitcpio.

Thx

soundofthunder · 1 October 2023 19:32

What happened after the stop job finished? Did it fail, or did it just continue shutting down?

Yes. If you try something and it doesn’t work – reverse it – always.

Ergo, the connection is dropped and the command isn’t received.

jackdinn · 1 October 2023 19:44

Ahh, yes this is the answer. I do remember doing this before (great i remember it now OFC) I shall go write it down in my Cherrytree list of hundreds of other “bang you head against the nearest wall” items after installing whatever distro.

Thank you @megavolt

system · 4 October 2023 09:44

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.