NetworkManager crashing after recent update of curl package

Hi,

after a recent update of some packages the NetworkManager (v1.48.10) on my machine keeps crashing with something like below in the logs.

Crashlog (journalctl)
Okt 10 20:28:08 tp-fs kernel: NetworkManager[363920]: segfault at 28 ip 00005e47a0655a3b sp 00007ffcbee33d80 error 4 in NetworkManager[142a3b,5e47a052d000+2a3000] likely on CPU 8 (core 4, socket 0)
Okt 10 20:28:08 tp-fs kernel: Code: 0f 1e fa 55 48 8d 15 f9 66 18 00 be 03 00 00 00 48 89 e5 e8 07 f9 ff ff 31 c0 5d c3 0f 1f 00 55 48 89 e5 41 56 41 55 41 54 53 <48> 8b 5f 28 48 85 db 74 4c 4c 8d 77 28 49 89 fc 45 31 ed 49 39 de
Okt 10 20:28:08 tp-fs systemd-coredump[363947]: Process 363920 (NetworkManager) of user 0 terminated abnormally with signal 11/SEGV, processing...
Okt 10 20:28:08 tp-fs systemd[1]: Started Process Core Dump (PID 363947/UID 0).
Okt 10 20:28:09 tp-fs systemd-coredump[363952]: [🡕] Process 363920 (NetworkManager) of user 0 dumped core.
                                                
                                                Stack trace of thread 363920:
                                                #0  0x00005e47a0655a3b n/a (NetworkManager + 0x142a3b)
                                                #1  0x00005e47a0657a48 n/a (NetworkManager + 0x144a48)
                                                #2  0x000077c8e25ef299 n/a (libglib-2.0.so.0 + 0x5d299)
                                                #3  0x000077c8e2651ec7 n/a (libglib-2.0.so.0 + 0xbfec7)
                                                #4  0x000077c8e25effa7 g_main_loop_run (libglib-2.0.so.0 + 0x5dfa7)
                                                #5  0x00005e47a0540dab n/a (NetworkManager + 0x2ddab)
                                                #6  0x000077c8e1f3ae08 n/a (libc.so.6 + 0x25e08)
                                                #7  0x000077c8e1f3aecc __libc_start_main (libc.so.6 + 0x25ecc)
                                                #8  0x00005e47a05415c5 n/a (NetworkManager + 0x2e5c5)
                                                
                                                Stack trace of thread 363921:
                                                #0  0x000077c8e202063d __poll (libc.so.6 + 0x10b63d)
                                                #1  0x000077c8e2651e0d n/a (libglib-2.0.so.0 + 0xbfe0d)
                                                #2  0x000077c8e25ee795 g_main_context_iteration (libglib-2.0.so.0 + 0x5c795)
                                                #3  0x000077c8e25ee7f2 n/a (libglib-2.0.so.0 + 0x5c7f2)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                
                                                Stack trace of thread 363923:
                                                #0  0x000077c8e202063d __poll (libc.so.6 + 0x10b63d)
                                                #1  0x000077c8e2651e0d n/a (libglib-2.0.so.0 + 0xbfe0d)
                                                #2  0x000077c8e25effa7 g_main_loop_run (libglib-2.0.so.0 + 0x5dfa7)
                                                #3  0x000077c8e27f3d04 n/a (libgio-2.0.so.0 + 0x112d04)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                
                                                Stack trace of thread 363930:
                                                #0  0x000077c8e202c1fd syscall (libc.so.6 + 0x1171fd)
                                                #1  0x000077c8e264b807 g_cond_wait_until (libglib-2.0.so.0 + 0xb9807)
                                                #2  0x000077c8e25b7925 n/a (libglib-2.0.so.0 + 0x25925)
                                                #3  0x000077c8e26253cb n/a (libglib-2.0.so.0 + 0x933cb)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                
                                                Stack trace of thread 363924:
                                                #0  0x000077c8e202c1fd syscall (libc.so.6 + 0x1171fd)
                                                #1  0x000077c8e264b807 g_cond_wait_until (libglib-2.0.so.0 + 0xb9807)
                                                #2  0x000077c8e25b7925 n/a (libglib-2.0.so.0 + 0x25925)
                                                #3  0x000077c8e26253cb n/a (libglib-2.0.so.0 + 0x933cb)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                
                                                Stack trace of thread 363922:
                                                #0  0x000077c8e202c1fd syscall (libc.so.6 + 0x1171fd)
                                                #1  0x000077c8e264aeb0 g_cond_wait (libglib-2.0.so.0 + 0xb8eb0)
                                                #2  0x000077c8e25b795c n/a (libglib-2.0.so.0 + 0x2595c)
                                                #3  0x000077c8e26247f7 n/a (libglib-2.0.so.0 + 0x927f7)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                
                                                Stack trace of thread 363931:
                                                #0  0x000077c8e202c1fd syscall (libc.so.6 + 0x1171fd)
                                                #1  0x000077c8e264b807 g_cond_wait_until (libglib-2.0.so.0 + 0xb9807)
                                                #2  0x000077c8e25b7925 n/a (libglib-2.0.so.0 + 0x25925)
                                                #3  0x000077c8e26253cb n/a (libglib-2.0.so.0 + 0x933cb)
                                                #4  0x000077c8e261f1b6 n/a (libglib-2.0.so.0 + 0x8d1b6)
                                                #5  0x000077c8e1fa939d n/a (libc.so.6 + 0x9439d)
                                                #6  0x000077c8e202e49c n/a (libc.so.6 + 0x11949c)
                                                ELF object binary architecture: AMD x86-64
Okt 10 20:28:09 tp-fs systemd[1]: NetworkManager.service: Main process exited, code=dumped, status=11/SEGV
Okt 10 20:28:09 tp-fs systemd[1]: NetworkManager.service: Failed with result 'core-dump'.
Okt 10 20:28:09 tp-fs systemd[1]: systemd-coredump@57-363947-0.service: Deactivated successfully.

I tried a few things for debugging and this is what I found out:

  1. The issue occurs after the upgrade of the curl package from 8.10.0-1 to 8.10.1-1. Downgrading curl makes the issue disappear.
  2. The issue doesn’t happen if I compile the same NetworkManager version (1.48.10) from the upstream source and run it.

Any suggestions on how to resolve this or how to continue with debugging? Is there a way to get the NM package with debugging symbols?

Thanks!

Edit: Here is another report of this issue [Stable Update] 2024-10-10 - Kernels, Pacman 7.0, KDE Frameworks 6.6, Virtualbox 7.1.2, Mesa - #19 by iCEVooDoo

3 Likes

Thanks for sharing @fri.sch! You saved me from much frustration!

1 Like

Please check if 1.48.12-1 fixes the issue for you.

Unfortunately, no. It doesn’t. But thanks anyway.

I’m in the same situation Phil. For now, I postponed the upgrade after yesterday’s failed one, which I managed to escape because I had the daily backups done.

Please tell us how to proceed if this is the system we work on daily. Should we wait or downgrade? I understand that the downgrade is not preferable.

Thank you for your efforts.

If absolutely required you can downgrade curl to 8.10.0-1

You can accomplish this with the manjaro-downgrade package and

sudo downgrade curl

Or even directly with

sudo pacman -U /var/cache/pacman/pkg/curl-8.10.0-1-x86_64.pkg.tar.zst
1 Like

Thanks for the information. I know this, but my question was whether to do this now or wait a couple of days to see if it gets fixed for good.

Appreciate your response. :+1:

Why not install all the latest updates, then downgrade curl and wait for a proper fix. When the fix is available, then just upgrade curl again. That seems most convenient to me and that’s what I’m currently doing.

Ok. I will try later because now I’m working… :rofl:
I’ll keep you posted.

Done the upgrade and downgraded the curl. Now the Network Manager is working properly.
Do I need to put in pacman.conf to ignore this curl package till gets fixed?

Same for me. NetworkManager 1.48.12-1 still crashes with curl 8.10.1-1 but not with curl 8.10.0-1

@fri.sch Hmm, I compiled the newer version of networkmananger of the same series against the newer curl using the PKGBUILD. so how did you compile yours? I wonder if you all have some special setting or common case which makes this crash happen. I might need to check what changed in curl and might trigger that. It had about 50 changes. @fri.sch would it be possible for you to do a git-bisect on curl to find that one commit which might create this mess?

https://git-scm.com/docs/git-bisect

You can get the PKGBUILD of curl and instead of tag=curl-${pkgver//./_}?signed you can write commit=commit-hash-of-git-bisect-suggestions in line 28 and compile curl as many times needed and test it against your system with a package which creates that issue. Then check with Daniel Stenberg if he can figure out why that is.

I just noticed that other packages got also recompiled against curl. Might have something to do with it …

I did once a git bisect on a kernel. So you can check with my commits on how I did that using a PKGBUILD: GitHub - philmmanjaro/linux41: Manjaro Kernel 4.1

Depending on how easy you can verify the bug the process might take a while …


Side Note: did all updated your systems with the latest adwaita-legacy-icon-theme?

On my system, I have this version (adwaita-icon-theme-legacy not adwaita-legacy-icon-theme maybe you mistyped):

pamac search adwaita-icon-theme-legacy                                                                                                   ✔ 
adwaita-icon-theme-legacy  46.2-3 [Installed]                                                                                                      extra
    GNOME fallback icons for legacy apps

I didn’t use the PKGBUILD, I just compiled from the upstream repo and ran the binary. I recognize that this might not be comparable due to different build configuration, etc.

I wonder if you all have some special setting or common case which makes this crash happen.

I bet so.

@fri.sch would it be possible for you to do a git-bisect on curl to find that one commit which might create this mess?

Yes, I just ran the bisect with the PKGBUILD (stripped it down a bit to be more efficient) and it points me to this change: multi: check that the multi handle is valid in curl_multi_assign · curl/curl@48f61e7 · GitHub

I double-checked by checking out the non-working 8.10.1 tag and reverted the change above on top of it. The resulting package doesn’t cause crashes.

Side Note: did all updated your systems with the latest adwaita-legacy-icon-theme?

Yes, I did.

I’ve never done any bisects with PKGBUILDS but the easiest way to me seems to just use makepkg -o to fetch the source repo, then cd into the fetched repo and run git bisect there. Then run makepkg -ef after each bisect step to rebuild the package from the changed sources. At least that worked quite well for me in this case.

As I’m not sure who is to blame exactly, I now opened reports for both, NetworkManager and curl here:

@philm It looks like there is a fix queued for curl which will be released early in November (see here). I guess we could just wait for it to land and then upgrade.

1 Like

If that fixes the issue, it could be applied as a patch temporarily.

1 Like

That would be nice, yes.

@fri.sch I’ll push the fixed version to our branches so others have the fix as well. Could you write a small tutorial on how you did the git-bisect, so we could reference to that guide to other users? I think that might help for similar issues and improve our documentation …

1 Like

curl-8.10.1-1.0 and its split packages are now in all branches. Please test and confirm. Thx.

3 Likes