System hard freezes when screencasting with ffmpeg

If I record my screen for a couple minutes with ffmpeg my system will hard freeze (image on screen freezes, audio freezes, keyboard commands are not responded to (including sysrq) and after a couple seconds the keyboard will completely stop working (when it’s just frozen, if I press FN the LEDs for the keys it afects will light up, but after a few seconds that too will stop working indicating the keyboard is as frozen as the rest of the system)

I need to figure out what’s causing this, but all I know about how to troubleshoot it is running journalctl -b-1 but it doesn’t really give me much, the only thing I can find is this error:

kioslave5[21921]: Could not open input file:  "/home/rabcor/Videos/240822-201704.mp4"

It gets spammed over and over again from the moment I start recording till the crash (the file indicated is the file that the recording is being saved to by ffmpeg.

Everything besides this message is normal according to the journal, the only other messages within the timeframe are just reports on how ananicy is managing nice levels.

This is the full ffmpeg command:

ffmpeg -f pulse -ac 2 -i easyeffects_sink -c:a aac -video_size 1920x1080 -framerate 60 -f x11grab -i :0.0 -c:v h264_nvenc -tier high -spatial-aq 1 -multipass fullres -tune:v hq -qmin 20 -rc:v vbr_minqp -cq:v 29 -preset:v p7 -pix_fmt yuv444p -color_range tv -colorspace bt2020_ncl -color_primaries bt709 -color_trc gamma22 -vf hue=h=-3:s=0.98, eq=gamma=0.98:contrast=0.99 /home/rabcor/Videos/240822-201704.mp4

As you can see I am using nvenc, this could actually be related

I highly suspect the issue is related to the nvidia drivers somehow, they’ve been known to be problematic on this PC when I’m on linux, notably in a lot of games my system will freeze in just about the exact same way if vsync is turned off, and mpv freezes sometimes when I resize the window on vulkan renderer unless the swapchain depth is set to 1 (there’s a git issue for this, some other people had this problem); both issues can be prevented by running the xorg server off the nvidia gpu instead of the igpu (for instance via nvidia-xrun or optimus-manager).

The problem is I have no idea how to troubleshoot this beyond just the command to see the journal from your last boot.