NetworkManager does not start on freshly installed system

I just installed a fresh image of Manjaro Gnome minimal on a DeskMini 600 PC with Ryzen 7 8700G / 32 GB / 1 TB. When the live system was running from the USB stick, I could access (wired) Network without any issues. After booting into the new system, NetworkManager does not start up. Any hints how to fix this?

[blue ~]# uname -a
Linux blue 6.9.10-1-MANJARO #1 SMP PREEMPT_DYNAMIC Fri Jul 19 15:08:51 UTC 2024 x86_64 GNU/Linux
[blue ~]# systemctl start NetworkManager 
Job for NetworkManager.service failed because the control process exited with error code.
See "systemctl status NetworkManager.service" and "journalctl -xeu NetworkManager.service" for details.
[blue ~]# systemctl status NetworkManager.service
Ă— NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Sat 2024-08-10 12:29:57 CEST; 7s ago
 Invocation: dd65b90370594a7ea6b66e27c5f07a0f
       Docs: man:NetworkManager(8)
    Process: 2207 ExecStart=/usr/bin/NetworkManager --no-daemon (code=exited, status=203/EXEC)
   Main PID: 2207 (code=exited, status=203/EXEC)

Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Start request repeated too quickly.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 12:29:57 blue systemd[1]: Failed to start Network Manager.



[blue ~]# journalctl -xeu NetworkManager.service

â–‘â–‘ A start job for unit NetworkManager.service has begun execution.
â–‘â–‘ 
â–‘â–‘ The job identifier is 8480.
Aug 10 12:29:57 blue (kManager)[2207]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 12:29:57 blue (kManager)[2207]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
â–‘â–‘ Subject: Process /usr/bin/NetworkManager could not be executed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ The process /usr/bin/NetworkManager could not be executed and failed.
â–‘â–‘ 
â–‘â–‘ The error number returned by this process is ERRNO.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Main process exited, code=exited, status=203/EXEC
â–‘â–‘ Subject: Unit process exited
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ An ExecStart= process belonging to unit NetworkManager.service has exited.
â–‘â–‘ 
â–‘â–‘ The process' exit code is 'exited' and its exit status is 203.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
â–‘â–‘ Subject: Unit failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ The unit NetworkManager.service has entered the 'failed' state with result 'exit-code'.
Aug 10 12:29:57 blue systemd[1]: Failed to start Network Manager.
â–‘â–‘ Subject: A start job for unit NetworkManager.service has failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ A start job for unit NetworkManager.service has finished with a failure.
â–‘â–‘ 
â–‘â–‘ The job identifier is 8480 and the job result is failed.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
â–‘â–‘ Subject: Automatic restarting of a unit has been scheduled
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ Automatic restarting of the unit NetworkManager.service has been scheduled, as the result for
â–‘â–‘ the configured Restart= setting for the unit.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Start request repeated too quickly.
Aug 10 12:29:57 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
â–‘â–‘ Subject: Unit failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ The unit NetworkManager.service has entered the 'failed' state with result 'exit-code'.
Aug 10 12:29:57 blue systemd[1]: Failed to start Network Manager.
â–‘â–‘ Subject: A start job for unit NetworkManager.service has failed
â–‘â–‘ Defined-By: systemd
â–‘â–‘ Support: https://forum.manjaro.org/c/support
â–‘â–‘ 
â–‘â–‘ A start job for unit NetworkManager.service has finished with a failure.
â–‘â–‘ 
â–‘â–‘ The job identifier is 8614 and the job result is failed.

Hi @InBetweenDreams,

Sounds like a kernel module, or driver, is missing there.

Please boot the Live ISO and run the command:

inxi --verbosity=5 --filter

…and provide the output here.

1 Like

Hello @Mirdarthos ,

here’s what it says:

System:
  Kernel: 6.9.10-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 14.1.1
  Desktop: GNOME v: 46.3.1 Distro: Manjaro base: Arch Linux
Machine:
  Type: Desktop Mobo: ASRock model: X600M-STX serial: <superuser required>
    uuid: <superuser required> UEFI: American Megatrends LLC. v: 4.03
    date: 07/11/2024
Memory:
  System RAM: total: 32 GiB available: 30.44 GiB used: 2.36 GiB (7.8%)
  Array-1: capacity: 128 GiB slots: 2 modules: 2 EC: None
    max-module-size: 64 GiB note: est.
  Device-1: Channel-A DIMM 0 type: DDR5 size: 16 GiB speed: 5600 MT/s
  Device-2: Channel-B DIMM 0 type: DDR5 size: 16 GiB speed: 5600 MT/s
CPU:
  Info: 8-core model: AMD Ryzen 7 8700G w/ Radeon 780M Graphics bits: 64
    type: MT MCP arch: Zen 4 rev: 2 cache: L1: 512 KiB L2: 8 MiB L3: 16 MiB
  Speed (MHz): avg: 1270 high: 3978 min/max: 400/5176 cores: 1: 400 2: 400
    3: 400 4: 2999 5: 2945 6: 400 7: 3000 8: 400 9: 2999 10: 400 11: 400 12: 400
    13: 400 14: 400 15: 3978 16: 400 bogomips: 134450
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: AMD Phoenix1 driver: amdgpu v: kernel arch: RDNA-3 bus-ID: 04:00.0
    temp: 34.0 C
  Display: wayland server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.1
    compositor: gnome-shell driver: gpu: amdgpu resolution: 2560x1440~60Hz
  API: EGL v: 1.5 drivers: radeonsi,swrast platforms:
    active: wayland,x11,surfaceless,device inactive: gbm
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.1.3-manjaro1.1
    glx-v: 1.4 direct-render: yes renderer: AMD Radeon Graphics (radeonsi
    gfx1103_r1 LLVM 18.1.8 DRM 3.57 6.9.10-1-MANJARO)
Audio:
  Device-1: AMD Rembrandt Radeon High Definition Audio driver: snd_hda_intel
    v: kernel bus-ID: 04:00.1
  Device-2: AMD Family 17h/19h HD Audio vendor: ASRock driver: snd_hda_intel
    v: kernel bus-ID: 04:00.6
  Device-3: Realtek USB SPDIF Adapter
    driver: hid-generic,snd-usb-audio,usbhid type: USB bus-ID: 1-2.2:3
  API: ALSA v: k6.9.10-1-MANJARO status: kernel-api
  Server-1: JACK v: 1.9.22 status: off
  Server-2: PipeWire v: 1.2.1 status: active
Network:
  Device-1: Realtek RTL8125 2.5GbE vendor: ASRock driver: r8169 v: kernel
    port: e000 bus-ID: 03:00.0
  IF: enp3s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
Drives:
  Local Storage: total: 2.75 TiB used: 1.9 MiB (0.0%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 990 PRO 1TB size: 931.51 GiB
    temp: 36.9 C
  ID-2: /dev/sda vendor: Samsung model: SSD 870 EVO 2TB size: 1.82 TiB
  ID-3: /dev/sdb vendor: Kingston model: DataTraveler 3.0 size: 14.65 GiB
    type: USB
  ID-4: /dev/sdc vendor: Actions model: HS USB FlashDisk size: 7.86 GiB
    type: USB
  Message: No optical or floppy data found.
Partition:
  Message: No partition data found.
Swap:
  Alert: No swap data was found.
Sensors:
  System Temperatures: cpu: 39.1 C mobo: N/A gpu: amdgpu temp: 34.0 C
  Fan Speeds (rpm): N/A
Info:
  Processes: 338 Uptime: 1m Init: systemd
  Packages: 1016 Compilers: N/A Shell: Zsh v: 5.9 inxi: 3.3.35

There it shows as using the r8169 kernel module/driver when, according to you it is working. So it suggests the module isn’t being loaded when using the installation. So boot into the installed version, and try loading it. (This isn’t persistent, it’s just to test if this works.) Run:

sudo modprobe r8169

If this works, we’ll make it permanent. If it doesn’t reboot and afterwards please provide the output of:

sudo inxi --verbosity=5 --filter

And

ping 1.1.1.1

And

systemctl status NetworkManager.service
1 Like

Sadly, loading the module did not work

[blue ~]# inxi --verbosity=5 --filter
System:
  Kernel: 6.9.10-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 14.1.1
  Console: pty pts/0 Distro: Manjaro base: Arch Linux
Machine:
  Type: Desktop Mobo: ASRock model: X600M-STX serial: <filter>
    uuid: 62006b9c-2208-0000-0000-000000000000 UEFI: American Megatrends LLC. v: 4.03
    date: 07/11/2024
Memory:
  System RAM: total: 32 GiB available: 30.44 GiB used: 1.42 GiB (4.7%)
  Array-1: capacity: 128 GiB slots: 2 modules: 2 EC: None max-module-size: 64 GiB note: est.
  Device-1: Channel-A DIMM 0 type: DDR5 size: 16 GiB speed: 5600 MT/s
  Device-2: Channel-B DIMM 0 type: DDR5 size: 16 GiB speed: 5600 MT/s
CPU:
  Info: 8-core model: AMD Ryzen 7 8700G w/ Radeon 780M Graphics bits: 64 type: MT MCP arch: Zen 4
    rev: 2 cache: L1: 512 KiB L2: 8 MiB L3: 16 MiB
  Speed (MHz): avg: 724 high: 2998 min/max: 400/5176 cores: 1: 400 2: 400 3: 400 4: 2998 5: 400
    6: 400 7: 400 8: 400 9: 400 10: 400 11: 400 12: 400 13: 400 14: 2998 15: 400 16: 400
    bogomips: 134457
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: AMD Phoenix1 driver: amdgpu v: kernel arch: RDNA-3 bus-ID: 04:00.0 temp: 32.0 C
  Display: unspecified server: X.org v: 1.21.1.13 with: Xwayland v: 24.1.1 driver: gpu: amdgpu
    tty: 153x67 resolution: 2560x1440
  API: EGL v: 1.5 drivers: radeonsi,swrast platforms: active: surfaceless,device
    inactive: gbm,wayland,x11
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: mesa v: 24.1.3-manjaro1.1 note: console (EGL sourced)
    renderer: AMD Radeon Graphics (radeonsi gfx1103_r1 LLVM 18.1.8 DRM 3.57 6.9.10-1-MANJARO),
    llvmpipe (LLVM 18.1.8 256 bits)
Audio:
  Device-1: AMD Rembrandt Radeon High Definition Audio driver: snd_hda_intel v: kernel
    bus-ID: 04:00.1
  Device-2: AMD Family 17h/19h HD Audio vendor: ASRock driver: snd_hda_intel v: kernel
    bus-ID: 04:00.6
  Device-3: Realtek USB SPDIF Adapter driver: hid-generic,snd-usb-audio,usbhid type: USB
    bus-ID: 1-2.2:3
  API: ALSA v: k6.9.10-1-MANJARO status: kernel-api
  Server-1: JACK v: 1.9.22 status: off
  Server-2: PipeWire v: 1.2.1 status: n/a (root, process)
Network:
  Device-1: Realtek RTL8125 2.5GbE vendor: ASRock driver: r8169 v: kernel port: e000
    bus-ID: 03:00.0
  IF: enp3s0 state: down mac: <filter>
Drives:
  Local Storage: total: 2.74 TiB used: 6.28 GiB (0.2%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 990 PRO 1TB size: 931.51 GiB temp: 35.9 C
  ID-2: /dev/sda vendor: Samsung model: SSD 870 EVO 2TB size: 1.82 TiB
  ID-3: /dev/sdb vendor: Actions model: HS USB FlashDisk size: 7.86 GiB type: USB
  Message: No optical or floppy data found.
Partition:
  ID-1: / size: 852.37 GiB used: 6.28 GiB (0.7%) fs: ext4 dev: /dev/dm-0
    mapped: luks-a42b056d-e5c5-436c-aef0-6aa80f6b9209 label: N/A
    uuid: f8defb5c-6c53-4a30-b7f4-feaf697cd08d
  ID-2: /boot/efi size: 1.95 GiB used: 448 KiB (0.0%) fs: vfat dev: /dev/nvme0n1p1 label: N/A
    uuid: AA3E-834D
Swap:
  ID-1: swap-1 type: partition size: 62.5 GiB used: 0 KiB (0.0%) dev: /dev/nvme0n1p2 label: N/A
    uuid: adc770e5-63ee-4603-9faf-dd4b149dbff5
Sensors:
  System Temperatures: cpu: 36.8 C mobo: N/A gpu: amdgpu temp: 33.0 C
  Fan Speeds (rpm): N/A
Info:
  Processes: 314 Uptime: 5m Init: systemd
  Packages: 880 Compilers: N/A Shell: Bash v: 5.2.26 inxi: 3.3.35


[blue ~]# ping 1.1.1.1
ping: connect: Network is unreachable


[blue ~]# systemctl status NetworkManager
Ă— NetworkManager.service - Network Manager
     Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Sat 2024-08-10 13:39:05 CEST; 5min ago
 Invocation: f147e490389c48e5886c09618be95b14
       Docs: man:NetworkManager(8)
    Process: 687 ExecStart=/usr/bin/NetworkManager --no-daemon (code=exited, status=203/EXEC)
   Main PID: 687 (code=exited, status=203/EXEC)

Aug 10 13:39:05 blue systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Aug 10 13:39:05 blue systemd[1]: NetworkManager.service: Start request repeated too quickly.
Aug 10 13:39:05 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:05 blue systemd[1]: Failed to start Network Manager.


[blue ~]# 

Cool, cool. So now we have to dig a bit deeper. And this :point_down:

…gives us kind of a clue where to look. So let’s check the logs. Please provide the output of:

journalctl --priority=warning..crit --no-pager --boot=0 --unit=NetworkManager.service

Where:

  • The --priority=warning..err argument limits the output to warnings and errors only;
  • --no-pager formats the output nicely for use here, on the forum;
  • the --boot=0 argument limits the messages to be from the current boot; and
  • the --unit=NetworkManager.service arguments limits the output to specific messages about NetworkManager.
1 Like

here it is:

[blue ~]# journalctl --priority=warning..crit --no-pager --boot=0 --unit=NetworkManager.service
Aug 10 13:39:03 blue (kManager)[643]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue (kManager)[643]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:03 blue systemd[1]: Failed to start Network Manager.
Aug 10 13:39:03 blue (kManager)[678]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue (kManager)[678]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:03 blue systemd[1]: Failed to start Network Manager.
Aug 10 13:39:03 blue (kManager)[681]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue (kManager)[681]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:03 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:03 blue systemd[1]: Failed to start Network Manager.
Aug 10 13:39:04 blue (kManager)[684]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:04 blue (kManager)[684]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:04 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:04 blue systemd[1]: Failed to start Network Manager.
Aug 10 13:39:04 blue (kManager)[687]: NetworkManager.service: Failed to execute /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:04 blue (kManager)[687]: NetworkManager.service: Failed at step EXEC spawning /usr/bin/NetworkManager: Exec format error
Aug 10 13:39:04 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:04 blue systemd[1]: Failed to start Network Manager.
Aug 10 13:39:05 blue systemd[1]: NetworkManager.service: Start request repeated too quickly.
Aug 10 13:39:05 blue systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Aug 10 13:39:05 blue systemd[1]: Failed to start Network Manager.
[blue ~]# 

Have you edited /usr/bin/NetworkManager? Because that’s what it looks like…And it’s a binary file, so not something we should try to change…hmmm…I think you might need internet access on the machine to fix this…

No, I did not edit a single file. It really is a fresh install. My knowledge of linux’s lower levels after all these years is still pretty basic, but I’m smart enough not to mess with the system’s binaries. :wink:

I do have an old but reliable USB WLAN stick (see last output) but have no idea how to make it work.

[blue ~]# file /usr/bin/NetworkManager 
/usr/bin/NetworkManager: data
[blue ~]# ls -l /usr/bin/NetworkManager 
-rwxr-xr-x 1 root root 3607848  5. Jul 20:15 /usr/bin/NetworkManager
[blue ~]# /usr/bin/NetworkManager 
-bash: /usr/bin/NetworkManager: cannot execute binary file: Exec format error
[blue ~]# sha256sum /usr/bin/NetworkManager 
43af5b708bcb05e083be584dcae5fb0f90af4b78f2466d660cda31c48a424a52  /usr/bin/NetworkManager
[blue ~]# lsusb |grep Logitech
Bus 001 Device 005: ID 046d:c548 Logitech, Inc. Logi Bolt Receiver
[blue ~]# 

It has been corrupted somehow. Faulty disk drive? This is what I get;

$ file /usr/bin/NetworkManager
/usr/bin/NetworkManager: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=dbc0f06c328bc8579ab3bc4323a0eabe2cdfce14, for GNU/Linux 4.4.0, stripped
1 Like

I hope it isn’t the hardware. That computer is brand new and it was not cheap (but IMHO worth every cent). I get the same output on my “old” machine which I use for writing this.
Should I try and replace the binary on the new machine w/ a copy of the working one?

You can try, might work, might not. No way to really know…

:smiley:

1 Like

Okay, “what could possibly go wrong?” I thought and copied the NetworkManager binary from my private office computer to the new machine. It seems to work. :slight_smile:

Thanks everyone to pointing me to the faulty file. I know this is not a clean repair, but I give it a try. The new machine is for mainly made for surfing and media in the living room, no critical data on it. I keep watching how it performs.

Any suggestions how to check 1 TB NVMe for faults? (Maybe it is that old USB stick I used for quite some time…)

And that is why my next suggestion is to reinstall NetworkManager:

First, the cache might still contain the faulty file, so let’s remove it, if it’s there:

sudo rm -fr /var/cache/pacman/pkg/networkmanager-*

That should remove all NetworkManager related packages.

Next, reinstall NetworkManager, and overwrite everything it does in /usr/bin/:

pamac reinstall --overwrite='/usr/bin/*' $(pamac search --quiet --installed NetworkManager)

Well, that’s the theory anyway. Then try to reboot and see if it worked. If not, then you know how to get it working again. :wink:

So, I hope this helps!

1 Like

… as soon as the other SSD is erased, which may take a while because of encryption… I’ll keep you updated.

looks more like a bad ISO used for installation.
perhaps reinstall NetworkManager to hopefully fix it

but when one binary is bad - others likely are too
it’s totally unpredictable

check the ISO file you used to install for integrity

it will likely fail

1 Like

I checked the ISO after download and it was okay. At the moment I prefer to blame the old USB stick. Worst to happen is to have no YT & other media on the big screen in the living room for a few days. – I’ll take the risk.

That is fine - your decision.
The ISO can be fine - the copy that ended up on the stick might not be.

With Ventoy (for example) you would just copy the ISO to the stick
and could then do the integrity check on it again - because that is what you are actually using,
not the downloaded copy …

That is not the worst that can happen - but:
it’s your decision.

How did reinstalling NetworkManager go? :sunglasses:

You really have no idea why it failed or what caused it.
Blame the USB stick if you want, all you want.
It’s still not going to remedy the situation either way.

It’s a new system.
Just do it again - takes half an hour or less.

Or gamble and deal with unexpected behavior.

Takes hours of your time and of that of the helpers.

Your choice.

… but what? That machine is just a mediaplayer on steroids. :wink:

No. — If that machine shows any strange behavior again before new installation: I’ll keep quiet. I have to wait for the new USB stick to arrive next week. Now for the weekend I’m fine with how it works – and this is basically browser and music player. With the new stick, I’ll start over again w/ a clean install. Still hoping that I’m right it was the old stick and not the new hardware.

2 Likes

… or a less than perfect download -
but I myself had never had this happen - a “corrupted” download
but some people report just that

All the best!
Cheers!