Kworker cpu lag spikes when connecting gamepad

When I connect a gamepad to my PC, some kworker processes appear and start to consume 2% of cpu every 21 seconds that lags all my system for a second. It happens in kernel 4.19, and not in 5.4, but I can´t use 5.4 because I have some different problems with it.
The processes are
The problem remains after disconnecting the gamepad and it persist until I restart the entire PC.