System freeze | RT Throttling activated

[maurice-m]

Hi there,

since a few weeks (maybe even a few months) I get random freezes of my UI (using Wayland and sway x64). The rest of the system is running (e.g. music is still playing in the background and even video calls are continuing - the others see a frozen picture of me but can hear me and I can hear them). But I cannot switch to console (ctrl+alt+[1-9]) so I need to press the power button to restart my PC.
Now, since a few days, the freezes are occurring multiple times a day. I tried to tackle the triggers down and in journalctl always one line appears when the UI crashes:

kernel: sched: RT throttling activated

The last time it crashed I waited some time before rebooting. Then I checked with journalctl and this is literally the only entry at this time. Here are the logs with timestamps. The GUI froze at 17:37.

Feb 01 17:35:01 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:35:01 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:35:07 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:35:07 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:37:35 [PCNAME]> kernel: sched: RT throttling activated
Feb 01 17:38:00 [PCNAME]> systemd[1]: Started Refresh existing keys of archlinux-keyring.
Feb 01 17:38:00 [PCNAME]> archlinux-keyring-wkd-sync[7611]: Skipping key 91C7BE39F7D158E8E518C7D957247034E31EC72A with UID pacman@localhost...
[keyring process continues...]
Feb 01 17:38:15 [PCNAME]> archlinux-keyring-wkd-sync[7611]: Error refreshing key 48C3B1F30DDD0FE67E516D16396E3E25BAB142C1 with UID keenerd@archlinux.org.
Feb 01 17:38:15 [PCNAME]> systemd[1]: archlinux-keyring-wkd-sync.service: Main process exited, code=exited, status=1/FAILURE
Feb 01 17:38:15 [PCNAME]> systemd[1]: archlinux-keyring-wkd-sync.service: Failed with result 'exit-code'.
Feb 01 17:38:15 [PCNAME]> systemd[1]: archlinux-keyring-wkd-sync.service: Consumed 1.314s CPU time.
Feb 01 17:38:16 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:38:16 [PCNAME]> rtkit-daemon[1111]: Supervising 9 threads of 6 processes of 1 users.
Feb 01 17:38:21 [PCNAME]> systemd-logind[808]: Power key pressed short.
Feb 01 17:38:21 [PCNAME]> systemd-logind[808]: Powering off...
Feb 01 17:38:21 [PCNAME]> systemd-logind[808]: System is powering down.
Feb 01 17:38:21 [PCNAME]> polkitd[1279]: Unregistered Authentication Agent for unix-session:3 (system bus name :1.42, object path /org/gnome/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Feb 01 17:38:21 [PCNAME]> greetd[7863]: config: Config { file: ConfigFile { terminal: ConfigTerminal { vt: None, switch: false }, general: ConfigGeneral { source_profile: true, runfile: "/run/greetd.run" }, default_session: ConfigSession { command: "", user: "" }, initial_session: None }, internal: ConfigInternal { session_worker: 11 } }
Feb 01 17:38:21 [PCNAME]> systemd[1]: Stopping Session 3 of User [USERNAME]...
Feb 01 17:38:21 [PCNAME]> systemd[1]: Removed slice Slice /system/getty.
Feb 01 17:38:21 [PCNAME]> kernel: traps: 1password[4194] trap int3 ip:55c232cc6a14 sp:7ffe83aa24a0 error:0 in 1password[55c22f939000+71c3000]
Feb 01 17:38:21 [PCNAME]> systemd[1]: Removed slice Slice /system/modprobe.
[shutdown process continues... until next snippet with wayland errors and sourrounding logs]
Feb 01 17:38:21 [PCNAME]> systemd[1]: Stopped CUPS Scheduler.
Feb 01 17:38:21 [PCNAME]> systemd[1]: greetd.service: Deactivated successfully.
Feb 01 17:38:21 [PCNAME]> systemd[1]: Stopped Greeter daemon.
Feb 01 17:38:21 [PCNAME]> systemd[1]: ntpd.service: Deactivated successfully.
Feb 01 17:38:21 [PCNAME]> mkinitcpio[7953]: ==> Build complete.
Feb 01 17:38:21 [PCNAME]> foot[2008]:  err: wayland.c:1405: failed to read events from the Wayland socket: Broken pipe
Feb 01 17:38:21 [PCNAME]> foot[2008]:  err: wayland.c:1942: failed to flush wayland socket: Broken pipe
Feb 01 17:38:21 [PCNAME]> foot[2008]: wayland: failed to read events from the Wayland socket: Broken pipe
Feb 01 17:38:21 [PCNAME]> foot[2008]: wayland: failed to flush wayland socket: Broken pipe
Feb 01 17:38:21 [PCNAME]> systemd[1189]: foot-server.service: Main process exited, code=exited, status=230/PERSONALITY
Feb 01 17:38:21 [PCNAME]> systemd[1189]: foot-server.service: Failed with result 'exit-code'.
Feb 01 17:38:21 [PCNAME]> bluetoothd[806]: Endpoint unregistered: sender=:1.52 path=/MediaEndpoint/A2DPSource/ldac
Feb 01 17:38:21 [PCNAME]> bluetoothd[806]: Endpoint unregistered: sender=:1.52 path=/MediaEndpoint/A2DPSink/aptx_hd
[shutdown process continues...until next snippet with bluetooth error and sourrounding logs]
Feb 01 17:38:21 [PCNAME]> systemd[1]: session-3.scope: Deactivated successfully.
Feb 01 17:38:21 [PCNAME]> systemd[1]: Stopped Session 3 of User [USERNAME].
Feb 01 17:38:21 [PCNAME]> systemd[1]: session-3.scope: Consumed 10min 49.830s CPU time.
Feb 01 17:38:21 [PCNAME]> systemd-logind[808]: Session 3 logged out. Waiting for processes to exit.
Feb 01 17:38:21 [PCNAME]> dbus-broker-launch[800]: Activation request for 'org.bluez' failed.
Feb 01 17:38:21 [PCNAME]> systemd-logind[808]: Removed session 3.
Feb 01 17:38:22 [PCNAME]> systemd[1]: Removed slice Slice /system/syncthing.
Feb 01 17:38:22 [PCNAME]> systemd[1]: system-syncthing.slice: Consumed 7.109s CPU time.
Feb 01 17:38:22 [PCNAME]> systemd[1]: Stopped target Network is Online.
[shutdown process continues until end]

A few more information:

uname -r
6.1.71-1-MANJARO

pamac checkupdates    
Your system is up to date.

cat /proc/cpuinfo  
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 25
model		: 80
model name	: AMD Ryzen 7 5700G with Radeon Graphics
stepping	: 0
microcode	: 0xa50000f
cpu MHz		: 1400.000
cache size	: 512 KB

So I have no idea how to continue from here. Since at the other crashes the line kernel: sched: RT throttling activated always appeared this is currently my best bet. As far as I understand, some processes might declare themselves as real-time processes. I have never even tempered with RT things (and until now I didn’t know you can seemingly mix them with a non-RT system).
So I hope that someone can point me in a direction to go.

[maurice-m]

I had another freeze. This time I just left the system. When I came back (30min later) it was in ‘Sleep’ mode. Pressing a key woke it up and it is running normally (no freeze). Checking journalctl (the freeze was at 13:15):

Feb 02 13:12:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:12:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:13:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:13:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:14:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:14:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:15:06 [PCNAME]> kernel: sched: RT throttling activated
Feb 02 13:15:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:15:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:15:41 [PCNAME]> pcscd[7671]: 99999999 winscard.c:281:SCardConnect() Error Reader Exclusive
Feb 02 13:15:41 [PCNAME]> pcscd[7671]: 00034608 winscard.c:281:SCardConnect() Error Reader Exclusive
Feb 02 13:16:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:16:30 [PCNAME]> rtkit-daemon[1083]: Supervising 11 threads of 8 processes of 1 users.
Feb 02 13:16:41 [PCNAME]> pcscd[7671]: 60275489 winscard.c:281:SCardConnect() Error Reader Exclusive
Feb 02 13:16:41 [PCNAME]> pcscd[7671]: 00029586 winscard.c:281:SCardConnect() Error Reader Exclusive
[this goes on and on - even waking it up did not show other logs]

[Teo]

Well it seems to me that some program is abusing realtime priviliges. Maybe until you narrow it down, you can try to stop and maybe mask the rtkit-daemon.service?

[maurice-m]

Thanks for the reply. Yes, that are my assumptions, too. But I have no idea how to find out what process that might be. Do you have any suggestions?

I don’t know the rtkit-daemon.service. It seems to be responsible for getting RT privileges. Can I really just stop it? I use this machine for work, so I’m not really keen on changing things I don’t understand (but will do, if this is a reasonable way to tackle this problem further).

[Teo]

Sorry, cannot help any further. But the log says how many processes are monitored, so there should be a way to list them.
Maybe you can just stop the service on every boot and see if it have negative effects and only after you are sure mask it. But it is a temporary workaround. Realtime is sometimes useful. I guess it will make itself noticeable with sound hickckups under load at some point. But better so instead of a complete freeze.

[nikgnomic]

Messages from rtkit-daemon: Supervising [X] threads of [Y] processes of [Z] users
is common and minor problem

To stop rtkit-daemon reporting debug messages, edit the systemd service:

sudo systemctl edit rtkit-daemon.service

Add this to reduce journal loglevel

[Service]
LogLevelMax=warning

And restart the service

systemctl restart rtkit-daemon

log spam from rtkit-daemon · Issue #22 · heftig/rtkit · GitHub

But I do not expect this to resolve the main kernel error sched: RT throttling activated

[Fabby]

@maurice-m

so I need to press the power button to restart my PC.

Please read this:

[HowTo] reboot / turn off your frozen computer: REISUB/REISUO

Difficulty: ★★☆☆☆ This tutorial was created because people are using a hard power off during freezes, so this will teach you how to turn off your computer when everything is frozen using softer means. So whenever you see someone saying they had to “Power Off” or “Hard Reset” their system without doing a REISUB, please send them here! If you’re reading this in response to a question, please click the green link above this text to bring you to the full unabridged text of the tutorial…

[cscs]

This is not an RT (realtime) kernel?

Also … for whatever its worth see this

3.12. Real Time Throttling Red Hat Enterprise Linux for Real...

[maurice-m]

Thanks for your reply. Yes, this would probably not stop the actual error. And the monitoring logs do not really bother me.

[maurice-m]

Thanks for your reply. Yes, this is not a RT kernel. After a quick google search, processes can request RT privileges. I didn’t know that until this error

[maurice-m]

Thank you all for taking your time to have a look at this issue. After meddling around more I finally decided to upgrade the kernel. Since a few days I’m now on 6.6.10-1-MANJARO without system freezes so far (which is a huge uplift compared to 1 - 3 freezes per day).
So, I still have no idea what the actual problem was but the kernel upgrade seemed to fix it.