That looks you have the limit of memory size. What size is your RAM? Did you enable Swap?
ffmpeg uses the lib libsvtav1
from the package svt-av1
that does not use Nvidia hardware acceleration, but your CPU and RAM.
Your Nvidia 3000 series does not have the ability AV1 encode.
See the wiki