NUC13 Intel I226-v R04 possible firmware-bug?

Hello there,

I need help with a really weird problem.

System is Intel NUC 13th gen running Manjaro i3/xfce

When I do power off the unit completely, I am in 99% of cases no longer able to boot back into Manjaro.

It always hangs when starting Network-Manager, sometimes it shows other network dependent services as ‘starting’ too, but the timeout never finishes and will add time to the max timeout.

Rebooting using ctrl+alt+del in this situation does nothing, as it hangs then still indefinitely at Network-Manager, even though other services will stop.

To get out of it I have to pull the power cord.

As I first expected some update for network-manager gone wrong, I chrooted into it and reinstalled network-manager, after that the boot worked again.

After doing this for countless times, I realized, that a re-install isn’t necessary.
It is just enough to send a single ICMP or any other network packet.
After that I can reboot into Manjaro and it will boot and work fine.

So maybe, this has something to do on how the network chip is initialized, maybe the NUC firmware itself is buggy as well.

Modules: igc/iwlwifi

filename:       /lib/modules/6.1.85-1-MANJARO/kernel/drivers/net/ethernet/intel/igc/igc.ko.zst
filename:       /lib/modules/6.1.85-1-MANJARO/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko.zst

Inxi:

System:
  Kernel: 6.1.85-1-MANJARO arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    clocksource: tsc avail: acpi_pm
    parameters: BOOT_IMAGE=/@/boot/vmlinuz-6.1-x86_64
    root=UUID=9394c1d4-9cf3-4f47-858d-3ada256e2cdc rw rootflags=subvol=@
    cryptdevice=UUID=e5c2a966-761c-4ab2-aeb7-965bb879c457:luks-e5c2a966-761c-4ab2-aeb7-965bb879c457
    root=/dev/mapper/luks-e5c2a966-761c-4ab2-aeb7-965bb879c457 apparmor=1
    security=apparmor udev.log_priority=3 fsck.mode=force net.ifnames=0
    bios.devname=0 sysrq_always_enabled=1
  Desktop: i3 v: 4.23 with: i3bar tools: avail: i3lock,light-locker,xautolock
    vt: 1 dm: 1: LightDM v: 1.32.0 note: stopped 2: LXDM Distro: Manjaro
    base: Arch Linux
Machine:
  Type: Mini-pc System: Intel Client Systems product: NUC13L3Ki3 v: N13836-302
    serial: <superuser required> Chassis: Intel Corporation type: 35 v: 2.0
    serial: <superuser required>
  Mobo: Intel model: NUC13L3Bi3 v: M99093-302 serial: <superuser required>
    part-nu: NUC13L3Ki3000 uuid: <superuser required> UEFI: Intel
    v: L3RPL357.0027.2023.0417.2327 date: 04/17/2023
CPU:
  Info: model: 13th Gen Intel Core i3-1315U bits: 64 type: MST AMCP
    arch: Raptor Lake level: v3 note: check built: 2022+ process: Intel 7 (10nm)
    family: 6 model-id: 0xBA (186) stepping: 3 microcode: 0x411C
  Topology: cpus: 1x cores: 6 mt: 2 tpc: 2 st: 4 threads: 8 smt: enabled
    cache: L1: 544 KiB desc: d-4x32 KiB, 2x48 KiB; i-2x32 KiB, 4x64 KiB
    L2: 4.5 MiB desc: 2x1.2 MiB, 1x2 MiB L3: 10 MiB desc: 1x10 MiB
  Speed (MHz): avg: 627 high: 999 min/max: 400/4500:3300 scaling:
    driver: intel_pstate governor: powersave cores: 1: 400 2: 400 3: 427 4: 400
    5: 949 6: 814 7: 999 8: 632 bogomips: 39944
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
  Vulnerabilities:
  Type: gather_data_sampling status: Not affected
  Type: itlb_multihit status: Not affected
  Type: l1tf status: Not affected
  Type: mds status: Not affected
  Type: meltdown status: Not affected
  Type: mmio_stale_data status: Not affected
  Type: reg_file_data_sampling status: Vulnerable: No microcode
  Type: retbleed status: Not affected
  Type: spec_rstack_overflow status: Not affected
  Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via
    prctl
  Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer
    sanitization
  Type: spectre_v2 mitigation: Enhanced / Automatic IBRS; IBPB: conditional;
    RSB filling; PBRSB-eIBRS: SW sequence; BHI: BHI_DIS_S
  Type: srbds status: Not affected
  Type: tsx_async_abort status: Not affected
Graphics:
  Device-1: Intel Raptor Lake-P [UHD Graphics] driver: i915 v: kernel
    arch: Gen-13 process: Intel 7 (10nm) built: 2022+ ports:
    active: HDMI-A-1,HDMI-A-2 empty: DP-1, DP-2, DP-3, DP-4 bus-ID: 00:02.0
    chip-ID: 8086:a7a9 class-ID: 0300
  Display: x11 server: X.Org v: 21.1.12 with: Xwayland v: 23.2.6 driver: X:
    loaded: modesetting alternate: fbdev,vesa dri: iris gpu: i915 display-ID: :0
    screens: 1
  Screen-1: 0 s-res: 5120x1440 s-dpi: 96 s-size: 1354x381mm (53.31x15.00")
    s-diag: 1407mm (55.38")
  Monitor-1: HDMI-A-1 mapped: HDMI-1 pos: right
    model: LG (GoldStar) ULTRAGEAR serial: <filter> built: 2023 res: 2560x1440
    hz: 120 dpi: 93 gamma: 1.2 size: 697x392mm (27.44x15.43")
    diag: 801mm (31.5") ratio: 16:9 modes: max: 2560x1440 min: 640x480
  Monitor-2: HDMI-A-2 mapped: HDMI-2 pos: primary,left
    model: LG (GoldStar) ULTRAGEAR serial: <filter> built: 2023 res: 2560x1440
    hz: 120 dpi: 93 gamma: 1.2 size: 697x392mm (27.44x15.43")
    diag: 801mm (31.5") ratio: 16:9 modes: max: 2560x1440 min: 640x480
  API: OpenGL Message: Unable to show GL data. glxinfo is missing.
Audio:
  Device-1: Intel Raptor Lake-P/U/H cAVS driver: snd_hda_intel v: kernel
    alternate: snd_sof_pci_intel_tgl bus-ID: 00:1f.3 chip-ID: 8086:51ca
    class-ID: 0401
  Device-2: ESI Audiotechnik GmbH Amber i1
    driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 2.0 speed: 480 Mb/s
    lanes: 1 mode: 2.0 bus-ID: 3-3.1:5 chip-ID: 2573:0023 class-ID: 0102
    serial: <filter>
  Device-3: Generalplus USB Audio Device
    driver: hid-generic,snd-usb-audio,usbhid type: USB rev: 1.1 speed: 12 Mb/s
    lanes: 1 mode: 1.1 bus-ID: 3-3.2:8 chip-ID: 1b3f:2008 class-ID: 0300
  Device-4: Nordic ASA Smart Control driver: hid-generic,snd-usb-audio,usbhid
    type: USB rev: 1.1 speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 3-7:4
    chip-ID: 1915:1012 class-ID: 0300
  API: ALSA v: k6.1.85-1-MANJARO status: kernel-api with: aoss
    type: oss-emulator tools: alsactl,alsamixer,amixer
  Server-1: sndiod v: N/A status: off tools: aucat,midicat,sndioctl
  Server-2: JACK v: 1.9.22 status: off tools: N/A
  Server-3: PipeWire v: 1.0.3 status: off tools: pw-cli
  Server-4: PulseAudio v: 17.0 status: active with: pulseaudio-alsa
    type: plugin tools: pacat,pactl,pavucontrol,pulsemixer
Network:
  Device-1: Intel Raptor Lake PCH CNVi WiFi driver: iwlwifi v: kernel
    bus-ID: 00:14.3 chip-ID: 8086:51f1 class-ID: 0280
  IF: wlan0 state: down mac: <filter>
  Device-2: Intel Ethernet I226-V driver: igc v: kernel pcie: gen: 2
    speed: 5 GT/s lanes: 1 port: N/A bus-ID: 56:00.0 chip-ID: 8086:125c
    class-ID: 0200
  IF: eth0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  Info: services: mld, NetworkManager, sshd, systemd-timesyncd
Bluetooth:
  Device-1: Intel AX211 Bluetooth driver: btusb v: 0.8 type: USB rev: 2.0
    speed: 12 Mb/s lanes: 1 mode: 1.1 bus-ID: 3-10:7 chip-ID: 8087:0033
    class-ID: e001
  Report: btmgmt ID: hci0 rfk-id: 0 state: up address: <filter> bt-v: 5.3
    lmp-v: 12 status: discoverable: no pairing: no class-ID: 6c0104
Drives:
  Local Storage: total: 7.33 TiB used: 4.35 TiB (59.3%)
  SMART Message: Unable to run smartctl. Root privileges required.
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Crucial model: CT4000P3PSSD8
    size: 3.64 TiB block-size: physical: 512 B logical: 512 B speed: 63.2 Gb/s
    lanes: 4 tech: SSD serial: <filter> fw-rev: P9CR40A temp: 41.9 C
    scheme: GPT
  ID-2: /dev/sda maj-min: 8:0 vendor: Crucial model: CT4000P3 SSD8
    size: 3.64 TiB block-size: physical: 4096 B logical: 512 B type: USB rev: 3.2
    spd: 10 Gb/s lanes: 1 mode: 3.2 gen-2x1 tech: SSD serial: <filter>
    fw-rev: 0214
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
  ID-3: /dev/sdb maj-min: 8:16 vendor: Transcend model: JetFlash 64GB
    size: 57.6 GiB block-size: physical: 512 B logical: 512 B type: USB rev: 3.0
    spd: 5 Gb/s lanes: 1 mode: 3.2 gen-1x1 tech: SSD serial: <filter>
    fw-rev: 1100 scheme: MBR
  SMART Message: Unknown USB bridge. Flash drive/Unsupported enclosure?
Partition:
  ID-1: / raw-size: 238.42 GiB size: 238.42 GiB (100.00%)
    used: 231.74 GiB (97.2%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
    mapped: luks-e5c2a966-761c-4ab2-aeb7-965bb879c457
  ID-2: /boot/efi raw-size: 953 MiB size: 951.1 MiB (99.80%)
    used: 752 KiB (0.1%) fs: vfat dev: /dev/nvme0n1p1 maj-min: 259:1
  ID-3: /home raw-size: 238.42 GiB size: 238.42 GiB (100.00%)
    used: 231.74 GiB (97.2%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
    mapped: luks-e5c2a966-761c-4ab2-aeb7-965bb879c457
  ID-4: /var/log raw-size: 238.42 GiB size: 238.42 GiB (100.00%)
    used: 231.74 GiB (97.2%) fs: btrfs dev: /dev/dm-0 maj-min: 254:0
    mapped: luks-e5c2a966-761c-4ab2-aeb7-965bb879c457
Swap:
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default) zswap: yes
    compressor: zstd max-pool: 20%
  ID-1: swap-1 type: zram size: 7.79 GiB used: 0 KiB (0.0%) priority: 100
    comp: zstd avail: lzo,lzo-rle,lz4,lz4hc,842 max-streams: 8 dev: /dev/zram0
Sensors:
  System Temperatures: cpu: 51.0 C mobo: N/A
  Fan Speeds (rpm): N/A
Info:
  Memory: total: 64 GiB note: est. available: 62.3 GiB used: 4.89 GiB (7.9%)
  Processes: 271 Power: uptime: 29m states: freeze,mem,disk suspend: s2idle
    avail: deep wakeups: 0 hibernate: platform avail: shutdown, reboot, suspend,
    test_resume image: 24.91 GiB services: upowerd Init: systemd v: 255
    default: graphical tool: systemctl
  Packages: 1669 pm: pacman pkgs: 1662 libs: 462 tools: pamac,yay pm: flatpak
    pkgs: 7 Compilers: clang: 16.0.6 gcc: 13.2.1 Shell: Bash v: 5.2.26
    running-in: urxvtd inxi: 3.3.34

journalctl -b-2 -p4 (includes just the failed boot)

Apr 29 18:21:58 ark kernel:  #5 #6 #7
Apr 29 18:21:58 ark kernel: ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
Apr 29 18:21:58 ark kernel: pci 0000:00:07.2: DPC: RP PIO log size 0 is invalid
Apr 29 18:21:58 ark kernel: hpet_acpi_add: no address or irqs in _CRS
Apr 29 18:21:58 ark kernel: nvme nvme0: missing or invalid SUBNQN field.
Apr 29 18:21:58 ark kernel: sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)
Apr 29 18:21:58 ark (sd-exec-[348]: /usr/lib/systemd/system-generators/zram-generator failed with exit status 1.
Apr 29 18:21:58 ark systemd-sysctl[394]: Couldn't write '0' to 'zswap/enabled', ignoring: No such file or directory
Apr 29 18:21:59 ark kernel: razermouse: loading out-of-tree module taints kernel.
Apr 29 18:21:59 ark kernel: iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver
Apr 29 18:22:00 ark kernel: thermal thermal_zone2: failed to read out thermal zone (-61)
Apr 29 18:22:02 ark kernel: Bluetooth: hci0: Malformed MSFT vendor event: 0x02
Apr 29 18:22:06 ark kernel: EXT4-fs (dm-1): mounting with "discard" option, but the device does not support discard
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support csip plugin
Apr 29 18:22:07 ark bluetoothd[1660]: profiles/audio/micp.c:micp_init() D-Bus experimental not enabled
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support micp plugin
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support vcp plugin
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support mcp plugin
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support bass plugin
Apr 29 18:22:07 ark bluetoothd[1660]: src/plugin.c:plugin_init() System does not support bap plugin
Apr 29 18:22:07 ark bluetoothd[1660]: src/adapter.c:reset_adv_monitors_complete() Failed to reset Adv Monitors: Failed (0x03)
Apr 29 18:22:07 ark bluetoothd[1660]: src/device.c:device_set_wake_support() Unable to set wake_support without RPA resolution
Apr 29 18:22:07 ark bluetoothd[1660]: Failed to clear UUIDs: Failed (0x03)
Apr 29 18:22:07 ark bluetoothd[1660]: Failed to add UUID: Failed (0x03)
Apr 29 18:22:07 ark bluetoothd[1660]: Failed to add UUID: Failed (0x03)
Apr 29 18:22:07 ark bluetoothd[1660]: Failed to add UUID: Failed (0x03)
Apr 29 18:22:07 ark bluetoothd[1660]: src/device.c:set_wake_allowed_complete() Set device flags return status: Invalid Parameters
Apr 29 18:22:07 ark kernel: Bluetooth: hci0: Bad flag given (0x1) vs supported (0x0)
Apr 29 18:22:17 ark systemd[1]: snapd.service: start operation timed out. Terminating.
Apr 29 18:22:17 ark systemd[1]: systemd-hostnamed.service: start operation timed out. Terminating.
Apr 29 18:22:27 ark systemd[1]: snapd.service: State 'stop-sigterm' timed out. Killing.
Apr 29 18:22:27 ark systemd[1]: systemd-hostnamed.service: State 'stop-sigterm' timed out. Killing.
Apr 29 18:22:37 ark systemd[1]: snapd.service: Processes still around after SIGKILL. Ignoring.
Apr 29 18:22:37 ark systemd[1]: systemd-hostnamed.service: Processes still around after SIGKILL. Ignoring.
Apr 29 18:22:41 ark kernel: EXT4-fs (dm-2): mounting with "discard" option, but the device does not support discard
Apr 29 18:22:47 ark systemd[1]: snapd.service: State 'final-sigterm' timed out. Killing.
Apr 29 18:22:47 ark systemd[1]: systemd-hostnamed.service: State 'final-sigterm' timed out. Killing.
Apr 29 18:22:57 ark systemd[1]: snapd.service: Processes still around after final SIGKILL. Entering failed mode.
Apr 29 18:22:57 ark systemd[1]: snapd.service: Failed with result 'timeout'.
Apr 29 18:22:57 ark systemd[1]: Failed to start Snap Daemon.
Apr 29 18:22:57 ark systemd[1]: systemd-hostnamed.service: Processes still around after final SIGKILL. Entering failed mode.
Apr 29 18:22:57 ark systemd[1]: systemd-hostnamed.service: Failed with result 'timeout'.
Apr 29 18:22:57 ark systemd[1]: Failed to start Hostname Service.
Apr 29 18:22:57 ark systemd[1]: snapd.service: Found left-over process 1661 (snapd) in control group while starting unit. Ignoring.
Apr 29 18:22:57 ark systemd[1]: snapd.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

Can provide more logs if required.

I tested dis/enabling the UEFI network-stack but this doesn’t makes a difference.

I’m out of ideas now and this gets also very frustrating and time consuming.

I really hope some of you can give a hint where to look further, I really want this solved.

Thanks in advance