System is unresponsive for approximate a minute after screen saver / no user input

Don’t know where to put this, but my whole system (GUI, login screen after screen saver, KDE,…) becomes completely unresponsive for about a minute after there hasn’t been any user input for some time (screen lock triggered or not).

I have a feeling that it is a bit better with 5.9 kernel than 5.11 and using less external monitors also helps.

What can I do to solve this issue?