Booting very slow SSD -- hints for systemd-analyze needed (modules)

Hi,
I have a problem with my boot-times (out of nowhere). The system is booting from a SSD.

if I systemd-analyze blame:

systemd-analyze blame
43.084s systemd-modules-load.service
18.321s systemd-random-seed.service
17.129s systemd-binfmt.service
12.125s systemd-timesyncd.service
12.124s systemd-update-utmp.service
 6.581s dev-sda2.device
 6.190s apparmor.service
 6.179s lvm2-monitor.service
 6.090s systemd-tmpfiles-setup-dev.service
 6.083s systemd-vconsole-setup.service
 4.170s pkgfile-update.service
 4.026s NetworkManager-wait-online.service
  609ms dev-loop0.device

I wonder what the first 5 services do and how I can figure out what is running that long? Especially the modules-load-service.

It seems not to be normal that the module-loader is working for 40secs+

systemd-analyse

Startup finished in 10.987s (firmware) + 2.718s (loader) + 14.701s (kernel) + 54.669s (userspace) = 1min 23.076s 
graphical.target reached after 46.580s in userspace.

OS-Specs:

Kernel: 5.10.129-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 12.1.0
    parameters: BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64
    root=UUID=78c62a92-a9d6-4778-9395-75882551638b rw quiet splash apparmor=1
    security=apparmor resume=UUID=79e5ac27-7be2-44a7-9ffb-61561e7007a3
    udev.log_priority=3 console=tty6
  Desktop: GNOME v: 42.3.1 tk: GTK v: 3.24.34 wm: gnome-shell dm: 1: GDM
    v: 42.0 note: stopped 2: LightDM v: 1.30.0 Distro: Manjaro Linux
    base: Arch Linux

Thanks a lot!

If you run Gnome i honestly recommend to make sure that GDM is the only DM installed and enabled. According to your forum profile you have:
CPU: AMD Ryzen 7 5800X
GPU: NVIDIA Gforce RTX 3090
And according to inxi you run

You would be better with 5.15 kernel …
Also, it would be much more useful for us if you shared the entire inxi -Fazy output.

3 Likes

This is highly unusual and it could be the system has problems loading the modules required.

Sample system
$ inxi -SCMmG
System:
  Host: tiger Kernel: 5.18.12-3-MANJARO arch: x86_64 bits: 64
    Desktop: KDE Plasma v: 5.25.3 Distro: Manjaro Linux
Machine:
  Type: Desktop System: LENOVO product: 30CY006WGE v: ThinkStation P330
    serial: <superuser required>
  Mobo: LENOVO model: 314F v: SDK0T08861 WIN 3305275981957
    serial: <superuser required> UEFI: LENOVO v: M1VKT65A date: 03/03/2021
Memory:
  RAM: total: 62.68 GiB used: 3.21 GiB (5.1%)
  RAM Report:
    permissions: Unable to run dmidecode. Root privileges required.
CPU:
  Info: 8-core model: Intel Core i9-9900K bits: 64 type: MT MCP cache:
    L2: 2 MiB
  Speed (MHz): avg: 800 min/max: 800/3600 cores: 1: 800 2: 800 3: 800
    4: 800 5: 800 6: 800 7: 800 8: 800 9: 800 10: 800 11: 800 12: 800 13: 800
    14: 800 15: 800 16: 800
Graphics:
  Device-1: AMD Ellesmere [Radeon Pro WX 7100] driver: amdgpu v: kernel
  Display: x11 server: X.Org v: 21.1.4 with: Xwayland v: 22.1.3 driver: X:
    loaded: amdgpu unloaded: modesetting gpu: amdgpu
    resolution: 5120x1440~120Hz
  OpenGL: renderer: AMD Radeon Pro WX 7100 Graphics (polaris10 LLVM 14.0.6
    DRM 3.46 5.18.12-3-MANJARO)
    v: 4.6 Mesa 22.1.3
$ systemd-analyze blame
5.229s NetworkManager-wait-online.service
2.054s systemd-tmpfiles-clean.service
1.992s boot-efi.mount
1.982s pkgfile-update.service
 433ms udisks2.service
 338ms a-private.mount
 328ms a-iso.mount
 224ms dev-nvme0n1p2.device
 155ms user@1000.service
 148ms systemd-journal-flush.service
 107ms lvm2-monitor.service
  85ms ldconfig.service
  79ms systemd-modules-load.service
  63ms var-lib-nfs-rpc_pipefs.mount
  60ms systemd-udev-trigger.service
  46ms upower.service
  36ms a-virtualbox.mount
  35ms systemd-udevd.service
  35ms pamac-daemon.service
  29ms systemd-journald.service
  28ms systemd-tmpfiles-setup.service
  27ms logrotate.service
  24ms NetworkManager.service
  22ms systemd-logind.service
  21ms a-projects.mount
  20ms avahi-daemon.service
  18ms systemd-sysctl.service
  17ms dbus.service
  16ms systemd-sysusers.service
  15ms systemd-tmpfiles-setup-dev.service
  13ms modprobe@fuse.service
  11ms systemd-journal-catalog-update.service
  11ms systemd-fsck@dev-disk-by\x2duuid-37C3\x2d8F1C.service
  10ms polkit.service
  10ms gssproxy.service
   9ms systemd-random-seed.service
   6ms rpc-statd-notify.service
   6ms user-runtime-dir@1000.service
   6ms dev-hugepages.mount
   5ms dev-mqueue.mount
   5ms sys-kernel-debug.mount
   5ms sys-kernel-tracing.mount
   4ms kmod-static-nodes.service
   4ms systemd-update-utmp.service
   4ms dev-disk-by\x2duuid-c3ce6c90\x2d8a5e\x2d4e28\x2d8580\x2d1f109642a880.swap
   4ms alsa-restore.service
   3ms modprobe@configfs.service
   3ms systemd-remount-fs.service
   3ms modprobe@drm.service
   2ms systemd-update-done.service
   2ms systemd-user-sessions.service
   2ms rtkit-daemon.service
   1ms sys-kernel-config.mount
   1ms tmp.mount
   1ms sys-fs-fuse-connections.mount

As you say - out of nowhere - I suspect file system errors - perhaps caused by beginning disk failure - but that is a guess and your profile suggests this is fairly recent system perhaps 12-18 month?

Which file system has been setup for your root device?

Is your root device/partition running out of space? e.g. btrfs with a bulk load of snapshots or your logs running full or …

Why are you using 5.10 kernel - likely 5.15 or 5.18 supports your hardware better than 5.10.

2 Likes

I will put the output in here, when I am at home again.

To be honest, I am unsure how to upgrade (not destroying my nvidia-configuration). Last time I decided to upgrade to a newer kernel, the GUI stopped working. I think it was a problem with the nvidia-module. So I switched back to this kernel again.

Btw. Manjaro is a great distribution, which helped me to finally go “away” from MS Windows.

I also have to say, that I am quite good at debian, but very new to Manjaro / Arch. So if something brakes, I cannot fix it without some advise (from the “internet”) :wink:

The root-device is 85% full.

ext4

well, I usually switch components from time to time, but not all. Mainboard and CPU are recent. The SSD (with the Rootfs on it) is about 4 years old. So it might really be possible that the SSD is not that good (begin to fail)

I suggest you do some housekeeping :slight_smile:

Move some stuff of the disk - perhaps even - dare I say it - delete stuff?

I have a bad habit of keeping ISO and other obsolete data around - now I regularly purge them - perhaps not all but the obsolete.

Vacuum your log

journalctl --vacuum-size=50M

Cleanup your pacman cache

sudo paccache -ruk0

Run

sudo fstrim -a

Clear downloads folder

rm -f ~/Downloads/*.iso ~/Downloads/*.zip

Use mhwd-kernel - it will sync the necessary packages required your nvidia setup

Perhaps - run a timeshift snapshot before doing anything - then you have the option to rollback if something fails.

2 Likes

@linux-aarhus I will give it a try when I am at home and come back to you.

Thanks!

Before I upgraded the Kernel!

System:
  Kernel: 5.10.131-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 12.1.0
    parameters: BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64
    root=UUID=78c62a92-a9d6-4778-9395-75882551638b rw quiet splash apparmor=1
    security=apparmor resume=UUID=79e5ac27-7be2-44a7-9ffb-61561e7007a3
    udev.log_priority=3 console=tty6
  Desktop: GNOME v: 42.3.1 tk: GTK v: 3.24.34 wm: gnome-shell dm: 1: GDM
    v: 42.0 note: stopped 2: LightDM v: 1.30.0 Distro: Manjaro Linux
    base: Arch Linux
Machine:
  Type: Desktop Mobo: Micro-Star model: MPG X570 GAMING EDGE WIFI (MS-7C37)
    v: 1.0 serial: <filter> UEFI: American Megatrends LLC. v: 1.I0
    date: 05/19/2022
CPU:
  Info: model: AMD Ryzen 7 5800X socket: AM4 bits: 64 type: MT MCP arch: Zen 3
    gen: 4 built: 2021-22 process: TSMC n7 (7nm) family: 0x19 (25)
    model-id: 0x21 (33) stepping: 0 microcode: 0xA201016
  Topology: cpus: 1x cores: 8 tpc: 2 threads: 16 smt: enabled cache:
    L1: 512 KiB desc: d-8x32 KiB; i-8x32 KiB L2: 4 MiB desc: 8x512 KiB
    L3: 32 MiB desc: 1x32 MiB
  Speed (MHz): avg: 2337 high: 3800
    min/max: 2200/4995:5437:4850:5726:5143:5581:5288 boost: enabled
    base/boost: 3800/4850 scaling: driver: acpi-cpufreq governor: schedutil
    volts: 1.1 V ext-clock: 100 MHz cores: 1: 2200 2: 2200 3: 2800 4: 2200
    5: 2200 6: 2200 7: 2200 8: 2200 9: 2200 10: 2200 11: 3800 12: 2200
    13: 2200 14: 2200 15: 2200 16: 2200 bogomips: 121654
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
  Vulnerabilities:
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: spec_store_bypass
    mitigation: Speculative Store Bypass disabled via prctl and seccomp
  Type: spectre_v1
    mitigation: usercopy/swapgs barriers and __user pointer sanitization
  Type: spectre_v2 mitigation: Retpolines, IBPB: conditional, IBRS_FW,
    STIBP: always-on, RSB filling
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: NVIDIA GA102 [GeForce RTX 3090] vendor: ZOTAC driver: nvidia
    v: 515.57 alternate: nouveau,nvidia_drm non-free: 515.xx+
    status: current (as of 2022-06) arch: Ampere process: TSMC n7 (7nm)
    built: 2020-22 pcie: gen: 1 speed: 2.5 GT/s lanes: 16 link-max: gen: 4
    speed: 16 GT/s bus-ID: 2d:00.0 chip-ID: 10de:2204 class-ID: 0300
  Display: x11 server: X.org v: 1.21.1.4 with: Xwayland v: 22.1.3
    compositor: gnome-shell driver: X: loaded: nvidia gpu: nvidia display-ID: :0
    screens: 1
  Screen-1: 0 s-res: 3440x1440 s-size: <missing: xdpyinfo>
  Monitor-1: DP-0 res: 3440x1440 hz: 85 dpi: 109
    size: 800x335mm (31.5x13.19") diag: 867mm (34.15") modes: N/A
  OpenGL: renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2 v: 4.6.0 NVIDIA 515.57
    direct render: Yes
Audio:
  Device-1: NVIDIA GA102 High Definition Audio vendor: ZOTAC
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 2d:00.1 chip-ID: 10de:1aef class-ID: 0403
  Device-2: AMD Starship/Matisse HD Audio vendor: Micro-Star MSI
    driver: snd_hda_intel v: kernel pcie: gen: 4 speed: 16 GT/s lanes: 16
    bus-ID: 2f:00.4 chip-ID: 1022:1487 class-ID: 0403
  Sound Server-1: ALSA v: k5.10.131-1-MANJARO running: yes
  Sound Server-2: JACK v: 1.9.21 running: no
  Sound Server-3: PulseAudio v: 16.1 running: yes
  Sound Server-4: PipeWire v: 0.3.55 running: yes
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    vendor: Micro-Star MSI X570-A PRO driver: r8169 v: kernel pcie: gen: 1
    speed: 2.5 GT/s lanes: 1 port: d000 bus-ID: 27:00.0 chip-ID: 10ec:8168
    class-ID: 0200
  IF: enp39s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  Device-2: Intel Dual Band Wireless-AC 3168NGW [Stone Peak] driver: iwlwifi
    v: kernel pcie: gen: 1 speed: 2.5 GT/s lanes: 1 bus-ID: 29:00.0
    chip-ID: 8086:24fb class-ID: 0280
  IF: wlp41s0 state: down mac: <filter>
Bluetooth:
  Device-1: Intel Wireless-AC 3168 Bluetooth type: USB driver: btusb v: 0.8
    bus-ID: 1-4:2 chip-ID: 8087:0aa7 class-ID: e001
  Report: rfkill ID: hci0 rfk-id: 1 state: up address: see --recommends
Drives:
  Local Storage: total: 3.17 TiB used: 361.4 GiB (11.1%)
  SMART Message: Required tool smartctl not installed. Check --recommends
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Sabrent model: Rocket 4.0 1TB
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 63.2 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: RKT401.2 temp: 27.9 C scheme: GPT
  ID-2: /dev/nvme1n1 maj-min: 259:6 vendor: Sabrent model: Rocket 4.0 1TB
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 63.2 Gb/s
    lanes: 4 type: SSD serial: <filter> rev: RKT401.2 temp: 25.9 C scheme: MBR
  ID-3: /dev/sda maj-min: 8:0 vendor: Toshiba model: TR150 size: 447.13 GiB
    block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s type: SSD
    serial: <filter> rev: 12.3 scheme: GPT
  ID-4: /dev/sdb maj-min: 8:16 vendor: Samsung model: SSD 860 EVO 1TB
    size: 931.51 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s
    type: SSD serial: <filter> rev: 3B6Q scheme: GPT
Partition:
  ID-1: / raw-size: 438.03 GiB size: 430.09 GiB (98.19%)
    used: 361.4 GiB (84.0%) fs: ext4 block-size: 4096 B dev: /dev/sda2
    maj-min: 8:2
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%)
    used: 288 KiB (0.1%) fs: vfat block-size: 512 B dev: /dev/sda1 maj-min: 8:1
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default)
  ID-1: swap-1 type: partition size: 8.8 GiB used: 0 KiB (0.0%) priority: -2
    dev: /dev/sda3 maj-min: 8:3
Sensors:
  System Temperatures: cpu: 41.2 C mobo: N/A gpu: nvidia temp: 35 C
  Fan Speeds (RPM): N/A gpu: nvidia fan: 0%
Info:
  Processes: 367 Uptime: 2m wakeups: 0 Memory: 31.34 GiB used: 2.29 GiB (7.3%)
  Init: systemd v: 251 default: graphical tool: systemctl Compilers:
  gcc: 12.1.0 clang: 14.0.6 Packages: pacman: 1536 lib: 479 flatpak: 0
  Shell: Bash (su) v: 5.1.16 running-in: gnome-terminal inxi: 3.3.19

I am at 5.19 now (it seems it is rc1). Nvida config not broken. Did it with mhwd-kernel.

I also used the commands you mentioned (fstrim, journalctl, pacman …) to clean up the system. I still have 88% used discspace, so it might have been even fuller before.

My bootingtimes are far better at the moment:

1.878s systemd-modules-load.service
 689ms dev-loop1.device
 687ms dev-loop0.device
 322ms systemd-rfkill.service
 312ms dev-sda2.device
 243ms apparmor.service
 223ms ModemManager.service
 204ms tlp.service
 178ms cups.service
 153ms lvm2-monitor.service
  94ms boot-efi.mount
  81ms udisks2.service
  77ms user@1000.service
  57ms systemd-udev-trigger.service
  55ms systemd-tmpfiles-setup-dev.service
  52ms systemd-vconsole-setup.service
  50ms systemd-udevd.service
  45ms NetworkManager.service
  44ms accounts-daemon.service
  43ms polkit.service
  36ms snapd.apparmor.service
  35ms bluetooth.service
  35ms fprintd.service
  33ms upower.service

Unfortunatly we will probably never know, what did the trick :wink: cause I did all at once…