OBS NVENC encoding overloaded

I am trying to record my screen with OBS using the NVENC encoder. If I do any big motions (eg. dragging a window around the screen), I get a ton of “skipped frames due to encoding lag”. When recording my desktop, about 25-50% of the frames are skipped, but if I try to record a game, then 99% of frames are skipped.

I think it’s worth noting that I have a 4k 144hz monitor and I’m recording a 4k 60hz video. I know recording 4k 60hz is rather intensive, but NVENC shouldn’t have a problem with it. I am also recording to a PCIe4 SSD, so disk IO should not be a bottleneck. Everything on my system is up to date with the stable branch and I restarted after updating. My BIOS is also up to date.

What I’ve tried

  • Disabling G-SYNC
  • Different sources in OBS
    • Window Capture (Xcomposite)
    • Screen Capture (XSHM)
  • Using the “Indistinguishable Quality, Large File Size” recording quality
  • Setting “Output (Scaled) Resolution” to 1080p while keeping the “Base (Canvas) Resolution” at 4k
  • Different OBS versions/packages
    • obs-studio-27.2.4-1
    • obs-studio-git-27.2.0.177-1
    • obs-studio snap package
      • This one actually caused my screen to go black and then return me to the login screen every couple of minutes. Uninstalling it fixed it.

I previously had single GPU VM passthrough set up, so I figured that could be causing this issue. I reset my BIOS settings to default (other than my memory’s XMP profile). When I setup the VM, I had updated the grub config (/etc/default/grub) to use:

GRUB_CMDLINE_LINUX_DEFAULT="efi=runtime quiet apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=pt video:vesafb=off video=efifb:on"

I tried changing this to the following, but it didn’t help:

GRUB_CMDLINE_LINUX_DEFAULT="efi=runtime quiet apparmor=1 security=apparmor udev.log_priority=3"

I also tried renaming the config file and then reinstalling grub with pamac reinstall grub to get the default grub config. However that didn’t fix it either.

GRUB_CMDLINE_LINUX_DEFAULT="quiet udev.log_priority=3"

I ran sudo grub-mkconfig -o /boot/grub/grub.cfg and restarted after every edit.

Specs

  • OS: Manjaro Linux x86_64
  • Kernel: 5.15.49-1-MANJARO
  • DE: Plasma 5.24.5
  • WM: KWin
  • CPU: AMD Ryzen 7 3800X (16) @ 3.9 GHz
  • GPU: NVIDIA RTX 3090
  • Memory: 32gb dual-channel DDR4-3600

Graphics driver

$ inxi -G
Graphics:
  Device-1: NVIDIA GA102 [GeForce RTX 3090] driver: nvidia v: 515.48.07
  Display: x11 server: X.Org v: 21.1.3 with: Xwayland v: 22.1.2 driver: X:
    loaded: nvidia gpu: nvidia resolution: 3840x2160
  OpenGL: renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2
    v: 4.6.0 NVIDIA 515.48.07

I can’t post screenshots since I just made my account, but here is what I see in nvidia-settings:

  • System Information
    • Operating System: Linux-x86_64
    • NVIDIA Driver Version: 515.48.07
    • NVML Version: 11.515.48.07
  • X Server Information
    • Display Name: elias-desktop-manjaro:0
    • Server Version Number: 11.0
    • Server Vendor String: The X.Org Foundation
    • Server Vendor Version: 1.21.1.3 (12101003)
    • NV-CONTROL Version: 1.29
    • Screens: 1
  • X Server Display Configuration
    • Selection: GBT AORUS FI32U (DP-2 on GPU-0)
    • Configuration: X screen 0
    • Resolution: 3840x2160 144 Hz
    • Mode Name: 3840x2160_144
    • Orientation: No Rotation, No Reflection
    • ViewPortIn: 3840x2160
    • ViewPortOut: 3840x2160+0+0
    • Panning: 3840x2160
    • Force Composition Pipeline: No
    • Force Full Composition Pipeline: No
    • Allows G-SYNC on monitor not validated as G-SYNC Compatible: Yes

I exported my nvidia-settings config to a temporary place (so it doesn’t overwrite the normal config):

#
# /home/elias/Documents/.nvidia-settings-rc
#
# Configuration file for nvidia-settings - the NVIDIA Settings utility
# Generated on Sat Jul  2 14:08:53 2022
#

# ConfigProperties:

RcFileLocale = C
DisplayStatusBar = Yes
SliderTextEntries = Yes
IncludeDisplayNameInConfigFile = No
UpdateRulesOnProfileNameChange = Yes
Timer = PowerMizer_Monitor_(GPU_0),Yes,1000
Timer = Thermal_Monitor_(GPU_0),Yes,1000
Timer = Memory_Used_(GPU_0),Yes,3000

# Attributes:

0/SyncToVBlank=1
0/LogAniso=0
0/FSAA=0
0/TextureClamping=1
0/FXAA=0
0/AllowFlipping=1
0/FSAAAppControlled=1
0/LogAnisoAppControlled=1
0/OpenGLImageSettings=1
0/FSAAAppEnhanced=0
0/AllowGSYNC=1
0/AllowVRR=1
0/ShowGSYNCVisualIndicator=0
0/ShowVRRVisualIndicator=0
0/ShowGraphicsVisualIndicator=0
[DPY:DP-0]/Dithering=0
[DPY:DP-0]/DitheringMode=0
[DPY:DP-0]/DitheringDepth=0
[DPY:DP-0]/ColorSpace=0
[DPY:DP-0]/ColorRange=0
[DPY:DP-0]/SynchronousPaletteUpdates=0
[DPY:DP-1]/Dithering=0
[DPY:DP-1]/DitheringMode=0
[DPY:DP-1]/DitheringDepth=0
[DPY:DP-1]/ColorSpace=0
[DPY:DP-1]/ColorRange=0
[DPY:DP-1]/SynchronousPaletteUpdates=0
[DPY:HDMI-0]/Dithering=0
[DPY:HDMI-0]/DitheringMode=0
[DPY:HDMI-0]/DitheringDepth=0
[DPY:HDMI-0]/ColorSpace=0
[DPY:HDMI-0]/ColorRange=0
[DPY:HDMI-0]/SynchronousPaletteUpdates=0
[DPY:DP-2]/RedBrightness=0.000000
[DPY:DP-2]/GreenBrightness=0.000000
[DPY:DP-2]/BlueBrightness=0.000000
[DPY:DP-2]/RedContrast=0.000000
[DPY:DP-2]/GreenContrast=0.000000
[DPY:DP-2]/BlueContrast=0.000000
[DPY:DP-2]/RedGamma=1.000000
[DPY:DP-2]/GreenGamma=1.000000
[DPY:DP-2]/BlueGamma=1.000000
[DPY:DP-2]/Dithering=0
[DPY:DP-2]/DitheringMode=0
[DPY:DP-2]/DitheringDepth=0
[DPY:DP-2]/DigitalVibrance=0
[DPY:DP-2]/ColorSpace=0
[DPY:DP-2]/ColorRange=0
[DPY:DP-2]/SynchronousPaletteUpdates=0
[DPY:DP-3]/Dithering=0
[DPY:DP-3]/DitheringMode=0
[DPY:DP-3]/DitheringDepth=0
[DPY:DP-3]/ColorSpace=0
[DPY:DP-3]/ColorRange=0
[DPY:DP-3]/SynchronousPaletteUpdates=0
[DPY:HDMI-1]/Dithering=0
[DPY:HDMI-1]/DitheringMode=0
[DPY:HDMI-1]/DitheringDepth=0
[DPY:HDMI-1]/ColorSpace=0
[DPY:HDMI-1]/ColorRange=0
[DPY:HDMI-1]/SynchronousPaletteUpdates=0
[DPY:DP-4]/Dithering=0
[DPY:DP-4]/DitheringMode=0
[DPY:DP-4]/DitheringDepth=0
[DPY:DP-4]/ColorSpace=0
[DPY:DP-4]/ColorRange=0
[DPY:DP-4]/SynchronousPaletteUpdates=0
[DPY:DP-5]/Dithering=0
[DPY:DP-5]/DitheringMode=0
[DPY:DP-5]/DitheringDepth=0
[DPY:DP-5]/ColorSpace=0
[DPY:DP-5]/ColorRange=0
[DPY:DP-5]/SynchronousPaletteUpdates=0

Swap

In my search to fix this issue, I saw one person mention that using a swap file instead of a swap partition can cause issues with OBS. I haven’t yet tried this because I don’t really want to resize my partitions, but I am willing to try if anyone here thinks it’s worth a shot. I do not use hibernation, so I thought that using a swapfile would be fine.

$ swapon --show
NAME      TYPE SIZE USED PRIO
/swapfile file  64G   0B   -2

OBS

I made a new profile to reset the settings to default. The only things I changed are the recording path and “Output (Scaled) Resolution” from 1080p to 4k.

Version: 27.2.4-1 (64 bit)

Settings:

  • Output
    • Output Mode: Simple
    • Streaming
      • Video Bitrate: 2500 Kbps
      • Encoder: Hardware (NVENC)
      • Audio Bitrate: 160
      • Enable Advanced Encoder Settings: No
    • Recording
      • Recording Path: /home/elias/Videos
      • Generate File Name without Space: No
      • Recording Quality: High Quality, Medium File Size
      • Recording Format: mkv
      • Encoder: Hardware (NVENC)
      • Custom Muxer Settings: (blank)
      • Enable Replay Buffer: No
  • Video
    • Base (Canvas) Resolution: 3840x2160
    • Output (Canvas) Resolution: 3840x2160
    • Downscale Filter: Bicubic (Sharpened scaling, 16 samples)
    • Common FPS Values: 60

Scene:

  • Screen Capture (XSHM)
    • Screen: Screen DP-2 (3840x2160 @ 0,0)
    • Capture Cursor: Yes
    • Advanced Settings: No

If you are streaming while recording then the NVENC is used twice in the same time. Only use CBR for streaming, not for recording.

Anyway, this is a OBS specific topic and you have more chances asking questions about OBS settings in their forum. If you do not experience any other system and drivers issues with your Manjaro install, the scope of this forum might not suit your help request.

Edit: This might be helpful or worth to try.