Raspberry pi 3 is stuck with task hung

[osteoblast22]

Hi,
I’ve installed Manjaro arm with the xfce environment, but as soon as the desktop boots, the wifi doesn’t work and the board is kinda stuck where running any new command causes the error “task $something blocked for more than 120 seconds.”

to reproduce all i have to do is boot up the board, open a terminal and run a command like “ip link” or similar.

The following error is spitted up in the kernel log, after that the board is stuck i can’t even run “sudo reboot” to reboot the machine

[  242.654560] INFO: task kworker/0:1:35 blocked for more than 120 seconds.
[  242.654586]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.654594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.654605] task:kworker/0:1     state:D stack:    0 pid:   35 ppid:     2 flags:0x00000028
[  242.654803] Workqueue: events_power_efficient reg_check_chans_work [cfg80211]
[  242.654823] Call trace:
[  242.654842]  __switch_to+0x114/0x170
[  242.654857]  __schedule+0x320/0x900
[  242.654868]  schedule+0x50/0x10c
[  242.654881]  schedule_preempt_disabled+0x24/0x3c
[  242.654891]  __mutex_lock.constprop.0+0x184/0x510
[  242.654902]  __mutex_lock_slowpath+0x1c/0x30
[  242.654911]  mutex_lock+0x54/0x60
[  242.654922]  rtnl_lock+0x24/0x30
[  242.655038]  reg_check_chans_work+0x34/0x410 [cfg80211]
[  242.655050]  process_one_work+0x1dc/0x4bc
[  242.655069]  worker_thread+0x148/0x47c
[  242.655079]  kthread+0x14c/0x160
[  242.655091]  ret_from_fork+0x10/0x38
[  242.655151] INFO: task kworker/u8:3:196 blocked for more than 120 seconds.
[  242.655159]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.655167] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.655177] task:kworker/u8:3    state:D stack:    0 pid:  196 ppid:     2 flags:0x00000028
[  242.655201] Workqueue: netns cleanup_net
[  242.655215] Call trace:
[  242.655225]  __switch_to+0x114/0x170
[  242.655237]  __schedule+0x320/0x900
[  242.655247]  schedule+0x50/0x10c
[  242.655258]  schedule_preempt_disabled+0x24/0x3c
[  242.655267]  __mutex_lock.constprop.0+0x184/0x510
[  242.655276]  __mutex_lock_slowpath+0x1c/0x30
[  242.655285]  mutex_lock+0x54/0x60
[  242.655298]  rtnl_lock+0x24/0x30
[  242.655390]  cfg80211_pernet_exit+0x28/0xa4 [cfg80211]
[  242.655403]  ops_exit_list+0x4c/0x80
[  242.655415]  cleanup_net+0x22c/0x380
[  242.655424]  process_one_work+0x1dc/0x4bc
[  242.655433]  worker_thread+0x148/0x47c
[  242.655443]  kthread+0x14c/0x160
[  242.655453]  ret_from_fork+0x10/0x38
[  242.655479] INFO: task NetworkManager:326 blocked for more than 120 seconds.
[  242.655489]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.655496] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.655506] task:NetworkManager  state:D stack:    0 pid:  326 ppid:     1 flags:0x00000208
[  242.655521] Call trace:
[  242.655533]  __switch_to+0x114/0x170
[  242.655544]  __schedule+0x320/0x900
[  242.655554]  schedule+0x50/0x10c
[  242.655574]  rpm_resume+0x168/0x704
[  242.655584]  rpm_resume+0x278/0x704
[  242.655594]  __pm_runtime_resume+0x48/0x9c
[  242.655607]  usb_autopm_get_interface+0x2c/0x70
[  242.655618]  usbnet_read_cmd+0x4c/0xb0
[  242.655628]  __smsc95xx_read_reg+0x74/0xdc
[  242.655637]  __smsc95xx_phy_wait_not_busy+0x68/0xd0
[  242.655646]  __smsc95xx_mdio_read+0x5c/0x160
[  242.655656]  smsc95xx_mdiobus_read+0x24/0x30
[  242.655670]  __mdiobus_read+0x50/0x174
[  242.655681]  mdiobus_read+0x48/0x6c
[  242.655692]  smsc_phy_reset+0x30/0x80
[  242.655702]  phy_init_hw+0x40/0xd0
[  242.655711]  smsc95xx_resume+0x6c/0x140
[  242.655722]  usb_resume_interface.part.0.isra.0+0x98/0xe0
[  242.655731]  usb_resume_both+0x7c/0x160
[  242.655743]  usb_runtime_resume+0x24/0x30
[  242.655755]  __rpm_callback+0x98/0x160
[  242.655765]  rpm_callback+0x64/0x90
[  242.655776]  rpm_resume+0x46c/0x704
[  242.655786]  rpm_resume+0x278/0x704
[  242.655796]  __pm_runtime_resume+0x48/0x9c
[  242.655806]  usb_autopm_get_interface+0x2c/0x70
[  242.655816]  usbnet_open+0x34/0x2b4
[  242.655829]  __dev_open+0x134/0x204
[  242.655838]  __dev_change_flags+0x19c/0x220
[  242.655848]  dev_change_flags+0x30/0x70
[  242.655861]  do_setlink+0x214/0xd54
[  242.655874]  __rtnl_newlink+0x504/0x820
[  242.655884]  rtnl_newlink+0x5c/0x90
[  242.655894]  rtnetlink_rcv_msg+0x128/0x344
[  242.655907]  netlink_rcv_skb+0x64/0x124
[  242.655918]  rtnetlink_rcv+0x24/0x30
[  242.655928]  netlink_unicast+0x268/0x32c
[  242.655937]  netlink_sendmsg+0x1c4/0x3d0
[  242.655948]  sock_sendmsg+0x60/0x6c
[  242.655958]  ____sys_sendmsg+0x218/0x234
[  242.655968]  ___sys_sendmsg+0x88/0xd0
[  242.655977]  __sys_sendmsg+0x74/0xcc
[  242.655987]  __arm64_sys_sendmsg+0x30/0x40
[  242.655998]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.656008]  do_el0_svc+0x30/0xa0
[  242.656018]  el0_svc+0x20/0x30
[  242.656029]  el0_sync_handler+0x1a4/0x1b0
[  242.656039]  el0_sync+0x180/0x1c0
[  242.656053] INFO: task python:388 blocked for more than 120 seconds.
[  242.656067]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.656074] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.656084] task:python          state:D stack:    0 pid:  388 ppid:     1 flags:0x00000200
[  242.656099] Call trace:
[  242.656111]  __switch_to+0x114/0x170
[  242.656122]  __schedule+0x320/0x900
[  242.656132]  schedule+0x50/0x10c
[  242.656142]  schedule_preempt_disabled+0x24/0x3c
[  242.656151]  __mutex_lock.constprop.0+0x184/0x510
[  242.656161]  __mutex_lock_slowpath+0x1c/0x30
[  242.656170]  mutex_lock+0x54/0x60
[  242.656180]  __netlink_dump_start+0xd8/0x310
[  242.656190]  rtnetlink_rcv_msg+0x254/0x344
[  242.656200]  netlink_rcv_skb+0x64/0x124
[  242.656210]  rtnetlink_rcv+0x24/0x30
[  242.656219]  netlink_unicast+0x268/0x32c
[  242.656229]  netlink_sendmsg+0x1c4/0x3d0
[  242.656239]  sock_sendmsg+0x60/0x6c
[  242.656248]  __sys_sendto+0xdc/0x144
[  242.656259]  __arm64_sys_sendto+0x34/0x4c
[  242.656269]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.656280]  do_el0_svc+0x30/0xa0
[  242.656290]  el0_svc+0x20/0x30
[  242.656300]  el0_sync_handler+0x1a4/0x1b0
[  242.656309]  el0_sync+0x180/0x1c0
[  242.656326] INFO: task vncserver-x11-c:402 blocked for more than 120 seconds.
[  242.656334]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.656341] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.656350] task:vncserver-x11-c state:D stack:    0 pid:  402 ppid:   392 flags:0x00000200
[  242.656366] Call trace:
[  242.656377]  __switch_to+0x114/0x170
[  242.656388]  __schedule+0x320/0x900
[  242.656398]  schedule+0x50/0x10c
[  242.656409]  schedule_preempt_disabled+0x24/0x3c
[  242.656418]  __mutex_lock.constprop.0+0x184/0x510
[  242.656427]  __mutex_lock_slowpath+0x1c/0x30
[  242.656437]  mutex_lock+0x54/0x60
[  242.656446]  __netlink_dump_start+0xd8/0x310
[  242.656457]  rtnetlink_rcv_msg+0x254/0x344
[  242.656466]  netlink_rcv_skb+0x64/0x124
[  242.656477]  rtnetlink_rcv+0x24/0x30
[  242.656486]  netlink_unicast+0x268/0x32c
[  242.656496]  netlink_sendmsg+0x1c4/0x3d0
[  242.656505]  sock_sendmsg+0x60/0x6c
[  242.656515]  __sys_sendto+0xdc/0x144
[  242.656526]  __arm64_sys_sendto+0x34/0x4c
[  242.656536]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.656546]  do_el0_svc+0x30/0xa0
[  242.656563]  el0_svc+0x20/0x30
[  242.656575]  el0_sync_handler+0x1a4/0x1b0
[  242.656585]  el0_sync+0x180/0x1c0
[  242.656629] INFO: task (t-daemon):511 blocked for more than 120 seconds.
[  242.656638]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.656645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.656653] task:(t-daemon)      state:D stack:    0 pid:  511 ppid:     1 flags:0x00000209
[  242.656669] Call trace:
[  242.656679]  __switch_to+0x114/0x170
[  242.656690]  __schedule+0x320/0x900
[  242.656700]  schedule+0x50/0x10c
[  242.656711]  schedule_preempt_disabled+0x24/0x3c
[  242.656720]  __mutex_lock.constprop.0+0x184/0x510
[  242.656729]  __mutex_lock_slowpath+0x1c/0x30
[  242.656739]  mutex_lock+0x54/0x60
[  242.656748]  rtnl_lock+0x24/0x30
[  242.656834]  fib6_rules_net_exit+0x24/0x40 [ipv6]
[  242.656845]  ops_exit_list+0x4c/0x80
[  242.656855]  setup_net+0x158/0x210
[  242.656866]  copy_net_ns+0xe4/0x50c
[  242.656877]  create_new_namespaces+0x11c/0x320
[  242.656888]  unshare_nsproxy_namespaces+0x70/0xbc
[  242.656902]  ksys_unshare+0x1b0/0x304
[  242.656912]  __arm64_sys_unshare+0x20/0x30
[  242.656923]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.656963]  do_el0_svc+0x30/0xa0
[  242.656975]  el0_svc+0x20/0x30
[  242.656985]  el0_sync_handler+0x1a4/0x1b0
[  242.656994]  el0_sync+0x180/0x1c0
[  242.657022] INFO: task pool-matray:649 blocked for more than 120 seconds.
[  242.657030]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.657037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.657046] task:pool-matray     state:D stack:    0 pid:  649 ppid:   430 flags:0x00000200
[  242.657063] Call trace:
[  242.657074]  __switch_to+0x114/0x170
[  242.657085]  __schedule+0x320/0x900
[  242.657095]  schedule+0x50/0x10c
[  242.657105]  schedule_preempt_disabled+0x24/0x3c
[  242.657115]  __mutex_lock.constprop.0+0x184/0x510
[  242.657124]  __mutex_lock_slowpath+0x1c/0x30
[  242.657133]  mutex_lock+0x54/0x60
[  242.657143]  __netlink_dump_start+0xd8/0x310
[  242.657153]  rtnetlink_rcv_msg+0x254/0x344
[  242.657163]  netlink_rcv_skb+0x64/0x124
[  242.657173]  rtnetlink_rcv+0x24/0x30
[  242.657183]  netlink_unicast+0x268/0x32c
[  242.657193]  netlink_sendmsg+0x1c4/0x3d0
[  242.657202]  sock_sendmsg+0x60/0x6c
[  242.657211]  __sys_sendto+0xdc/0x144
[  242.657222]  __arm64_sys_sendto+0x34/0x4c
[  242.657232]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.657242]  do_el0_svc+0x30/0xa0
[  242.657252]  el0_svc+0x20/0x30
[  242.657262]  el0_sync_handler+0x1a4/0x1b0
[  242.657271]  el0_sync+0x180/0x1c0
[  242.657284] INFO: task sudo:549 blocked for more than 120 seconds.
[  242.657292]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.657299] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.657313] task:sudo            state:D stack:    0 pid:  549 ppid:   430 flags:0x00000200
[  242.657329] Call trace:
[  242.657340]  __switch_to+0x114/0x170
[  242.657351]  __schedule+0x320/0x900
[  242.657361]  schedule+0x50/0x10c
[  242.657372]  schedule_preempt_disabled+0x24/0x3c
[  242.657381]  __mutex_lock.constprop.0+0x184/0x510
[  242.657390]  __mutex_lock_slowpath+0x1c/0x30
[  242.657400]  mutex_lock+0x54/0x60
[  242.657409]  __netlink_dump_start+0xd8/0x310
[  242.657419]  rtnetlink_rcv_msg+0x254/0x344
[  242.657430]  netlink_rcv_skb+0x64/0x124
[  242.657440]  rtnetlink_rcv+0x24/0x30
[  242.657449]  netlink_unicast+0x268/0x32c
[  242.657459]  netlink_sendmsg+0x1c4/0x3d0
[  242.657468]  sock_sendmsg+0x60/0x6c
[  242.657478]  __sys_sendto+0xdc/0x144
[  242.657489]  __arm64_sys_sendto+0x34/0x4c
[  242.657499]  el0_svc_common.constprop.0+0x84/0x1e0
[  242.657510]  do_el0_svc+0x30/0xa0
[  242.657519]  el0_svc+0x20/0x30
[  242.657530]  el0_sync_handler+0x1a4/0x1b0
[  242.657538]  el0_sync+0x180/0x1c0

[Darksky]

Try installing crda and set the regdomain code for your country then reboot.

https://wiki.archlinux.org/title/Network_configuration/Wireless#Respecting_the_regulatory_domain

[osteoblast22]

I uncommented the US code in “/etc/conf.d/wireless-regdom” and rebooted but the problem persist.

note that using raspbian( 32bit or 64 bit system) works just fine, the problem is only with manjaro

[Darksky]

@osteoblast22

I uncommented the US code in “/etc/conf.d/wireless-regdom” and rebooted but the problem persist.

note that using raspbian( 32bit or 64 bit system) works just fine, the problem is only with manjaro

I live in the US also (Arkansas). I just happened to download the latest xfce image yesterday just to test if all was good with the latest image. I just plugged the sdcard in my pi3b and booted up and all is good with the wifi here. I did not have to install anything. It still has it’s original kernel.

I’m guessing it has something to do with your router’s relationship. Really the kernel is the same as Pi OS with some more extra modules enabled.

[kerry_s]

make sure your running the latest release.

Releases · manjaro-arm/rpi4-images

it says rpi4, but it’s for the 3 to.

[osteoblast22]

Well i don’t know what to tell you, but i installed manjaro around 5 days ago and updated it so i’m pretty sure i’m using the latest version.
Also, i suspect that the problem is not the kernel why? because i copied the manjaro kernel along with it’s modules to the raspbian root file system and booted them(i.e i booted using manjaro kernel and raspbian os root file system) and it worked fine, so maybe the problem is somewhere in the userland.

[osteoblast22]

yes i’m running the latest release

[osteoblast22]

Sorry didn’t read that last line, you are assuming it’s only because of the wifi module, but I’ve not even configured the network on the system, so it’s not even trying to connect to a network, even a simple sudo systemctl status something command causes the system to hang

[Darksky]

I can only use the info you posted in your OP to try to make an educated guess regarding what others have had issues in the past.

wifi does not work
 
[  242.654586]       Tainted: G         C        5.10.46-1-MANJARO-ARM #1
[  242.654594] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.654605] task:kworker/0:1     state:D stack:    0 pid:   35 ppid:     2 flags:0x00000028
[  242.654803] Workqueue: events_power_efficient reg_check_chans_work [cfg80211]
[  242.654823] Call trace:
and so on....
 
Board gets stuck trying to run a command

What pops in my head from the info given being the wifi does not work and you have a hung task with a taint which most systems will eventually lock up. Network issues can do weird things to your systems; even to the point of not booting up all of the way.

Some here has had similar wifi issues because the generic default International regdomain code will not work for them until they install crda and set their country code.

Since I do not have any issues with the latest image on my pi3 here and others here are using the same image I can only assume the issue is on your end most likely with the router at this point.

[osteoblast22]

Ok, i downloaded the last manjaro image (Manjaro-ARM-xfce-rpi4-21.07.img) but it seems the image doesn’t contain an installer since the image boots directly to the xfce desktop(using the oem username) and doesn’t ask for username creation or locale selection.
Maybe the image was generated incorrectly since the image doesn’t even contain the manjaro-arm-installer

Regards

[Darksky]

It is doing what it is supposed to. When it reaches the Desktop with the Calamares Setup window start clicking “Next” until you go all through the Setup:

I know the latest xfce image is good as I installed it 2 days ago. The above pic is from me testing the Gnome image.

@osteoblast22

yes i’m running the latest release

Guessing now you were not.

[osteoblast22]

OK I wiped the card and installed the latest image from scratch and setup the regdom but the problem persist.

the board boots fine but as soon as i run ip addr or sudo the board hangs with the same error

[Darksky]

@osteoblast22

the board boots fine but as soon as i run ip addr or sudo the board hangs with the same error

Hard to say what is going on there when I have none of those issues here.

[osteoblast22]

Well i have the same board revision as you and a slightly newer kernel

[osteoblast22]

Ok after a lot of testing it seems the problem only happens when using NetworkManager or dhcpcd to configure the network.

I setup wpa_suppliacnt and dhclient to connect to the network and give me an ip (disabling NetworkManager of course) and the problem is gone
I can run ip addr or any network related commands with no problem, sudo also works now and i don’t see any hung task in dmesg.
I went back and forth between the two setups and it always gave the same results so i’m pretty sure that NetworkManager is somehow causing the problem.

It definitely needs more debugging to see how to fix the issue and return back to NetworkManager

[Darksky]

It looks like for some reason your router is not providing needed channel info:

[  242.654803] Workqueue: events_power_efficient reg_check_chans_work [cfg80211]
[  242.654823] Call trace:

Check and see if there is a firmware upgrade available for your router.

Another thing to try is setting a specific channel (like Channel 6) and band in your router and in Network Manager.

[osteoblast22]

well the NetworkManager doesn’t have a connection setup for my router(i.e it doesn’t have a profile setup with the ssid and the password of my router) so it isn’t even interacting with my router to cause the problem.

creating a config for my router is a bit hard since the system gets stuck if i start NetworkManager, nevertheless i was able to create a connection profile to connect to another router(with fixed channel) just to be sure that my router is not the problem and it didn’t solve the problem.

Note that it’s very unlikely that my router is the problem for those reasons:
1- NetworkManager was not connecting to my router when the problem occurred since it wasn’t setup to connect to my router.
2- My other system (x86) which is running Manjaro with the latest updates and also using NetworkManager is connecting just fine to my router.
3- Running Raspbian on the same raspberry pi is working just fine and connecting to the router(note that raspbian doesn’t use NetworkManager for networking, but even after installing and setting up NetworkManager in Raspbian and configuring it, it can connect with no problem)