OBS Studio Black Screen at recording games with obs-vkcapture

Support Software & Applications
1

Hello there,

I have Nvidia RTX3070 as my only gpu and I want to record games.

Problem is that when I record using obs-vkcapture only the audio is recorded but not the video itself.

I tried to use Flatpak and regular repo OBS versions and it is the same result.

I tried to check the log and this is what I get when I launch Flatpak and regular OBS trough terminal.

Qt: Session management error: Could not open network socket
debug: Found portal inhibitor
debug: Attempted path: share/obs/obs-studio/locale/en-US.ini
debug: Attempted path: /app/share/obs/obs-studio/locale/en-US.ini
debug: Attempted path: share/obs/obs-studio/locale.ini
debug: Attempted path: /app/share/obs/obs-studio/locale.ini
debug: Attempted path: share/obs/obs-studio/themes/Yami.qss
debug: Attempted path: /app/share/obs/obs-studio/themes/Yami.qss
warning: [Safe Mode] Unclean shutdown detected!
warning: [Safe Mode] User elected to launch normally.
info: Using EGL/X11
info: CPU Name: 11th Gen Intel(R) Core(TM) i7-11700F @ 2.50GHz
info: CPU Speed: 900.011MHz
info: Physical Cores: 8, Logical Cores: 16
info: Physical Memory: 15713MB Total, 1950MB Free
info: Kernel Version: Linux 6.5.9-1-MANJARO
info: Flatpak Branch: stable
info: Flatpak Arch: x86_64
info: Flatpak Runtime: runtime/org.kde.Platform/x86_64/6.5
info: App Extensions:
info:  - com.obsproject.Studio.Plugin.OBSVkCapture
info:  - com.obsproject.Studio.Locale
info: Runtime Extensions:
info:  - org.freedesktop.Platform.GL.default
info:  - org.freedesktop.Platform.GL.nvidia-535-129-03
info:  - org.freedesktop.Platform.openh264
info:  - org.kde.KStyle.Adwaita
info:  - org.kde.Platform.Locale
info:  - org.freedesktop.Platform.GL.default
info: Flatpak Framework Version: 1.15.4
info: Desktop Environment: KDE (KDE)
info: Session Type: x11
info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.21.1
info: Qt Version: 6.5.3 (runtime), 6.5.3 (compiled)
info: Portable mode: false
info: OBS 30.0.0 (linux)
info: ---------------------------------
info: ---------------------------------
info: audio settings reset:
        samples per sec: 48000
        speakers:        2
        max buffering:   960 milliseconds
        buffering type:  dynamically increasing
info: ---------------------------------
info: Initializing OpenGL...
info: Loading up OpenGL on adapter NVIDIA Corporation NVIDIA GeForce RTX 3070/PCIe/SSE2
info: OpenGL loaded successfully, version 3.3.0 NVIDIA 535.129.03, shading language 3.30 NVIDIA via Cg compiler
info: ---------------------------------
info: video settings reset:
        base resolution:   1920x1080
        output resolution: 1920x1080
        downscale filter:  Bicubic
        fps:               60/1
        format:            NV12
        YUV mode:          Rec. 709/Partial
info: NV12 texture support not available
info: P010 texture support not available
info: Audio monitoring device:
        name: CORSAIR VOID ELITE Wireless Gaming Dongle Analog Stereo
        id: alsa_output.usb-Corsair_CORSAIR_VOID_ELITE_Wireless_Gaming_Dongle-00.analog-stereo
info: ---------------------------------
warning: No AJA devices found, skipping loading AJA UI plugin
warning: Failed to initialize module 'aja-output-ui.so'
warning: No AJA devices found, skipping loading AJA plugin
warning: Failed to initialize module 'aja.so'
warning: Failed to load 'en-US' text for module: 'decklink-captions.so'
warning: Failed to load 'en-US' text for module: 'decklink-output-ui.so'
libDeckLinkAPI.so: cannot open shared object file: No such file or directory
warning: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed
warning: Failed to initialize module 'decklink.so'
info: [pipewire] Available captures:
info: [pipewire]     - Desktop capture
info: [pipewire]     - Window capture
warning: v4l2loopback not installed, virtual camera disabled
info: [obs-browser]: Version 2.22.1
info: [obs-browser]: CEF Version 103.0.5060.134 (runtime), 103.0.0-5060-shared-textures.2594+g17f8588+chromium-103.0.5060.134 (compiled)
info: NVENC supported
error: VAAPI: Failed to initialize display in vaapi_device_h264_supported
info: FFmpeg VAAPI H264 encoding not supported
error: VAAPI: Failed to initialize display in vaapi_device_hevc_supported
info: FFmpeg VAAPI HEVC encoding not supported
info: [obs-websocket] [obs_module_load] you can haz websockets (Version: 5.3.3 | RPC Version: 1)
info: [obs-websocket] [obs_module_load] Qt version (compile-time): 6.5.3 | Qt version (run-time): 6.5.3
info: [obs-websocket] [obs_module_load] Linked ASIO Version: 102800
info: [obs-websocket] [obs_module_load] Module loaded.
info: [linux-vkcapture] plugin loaded successfully (version 1.4.5)
info: ---------------------------------
info:   Loaded Modules:
info:     linux-vkcapture.so
info:     text-freetype2.so
info:     rtmp-services.so
info:     obs-x264.so
info:     obs-websocket.so
info:     obs-webrtc.so
info:     obs-vst.so
info:     obs-transitions.so
info:     obs-qsv11.so
info:     obs-outputs.so
info:     obs-libfdk.so
info:     obs-filters.so
info:     obs-ffmpeg.so
info:     obs-browser.so
info:     linux-v4l2.so
info:     linux-pulseaudio.so
info:     linux-pipewire.so
info:     linux-jack.so
info:     linux-capture.so
info:     image-source.so
info:     frontend-tools.so
info:     decklink-output-ui.so
info:     decklink-captions.so
info: ---------------------------------
info: ==== Startup complete ===============================================
info: All scene data cleared
info: ------------------------------------------------
info: pulse-input: Server name: 'PulseAudio (on PipeWire 0.3.83) 15.0.0'
info: pulse-input: Audio format: s16le, 48000 Hz, 2 channels
info: pulse-input: Started recording from 'alsa_output.usb-Corsair_CORSAIR_VOID_ELITE_Wireless_Gaming_Dongle-00.analog-stereo.monitor'
info: [Loaded global audio device]: 'Desktop Audio'
info: pulse-input: Server name: 'PulseAudio (on PipeWire 0.3.83) 15.0.0'
info: pulse-input: Audio format: s16le, 48000 Hz, 1 channels
info: pulse-input: Started recording from 'alsa_input.usb-Corsair_CORSAIR_VOID_ELITE_Wireless_Gaming_Dongle-00.mono-fallback'
info: [Loaded global audio device]: 'Mic/Aux'
info: Switched to scene 'Scene'
info: ------------------------------------------------
info: Loaded scenes:
info: - scene 'Scene':
info:     - source: 'Game Capture' (vkcapture-source)
info: ------------------------------------------------
info: adding 21 milliseconds of audio buffering, total audio buffering is now 21 milliseconds (source: Desktop Audio)

I tried all kinds of video encoders but no luck but I do see errors from the log regarding FFmpeg VAAPI H264 and FFmpeg VAAPI HEVC

Any help would be appreciated, I’ve been struggling with this for over a day now.

I also installed manjaro-pipewire repo not sure if this has any importance.

2

You might have better success with the recommended version of OBS-Studio; the one in the official Manjaro repos; instead of the Flatpak version.

Remove Flatpack OBS-Studio:

pamac remove obs-studio

Install Repo OBS-Studio:

sudo pacman -S obs-studio

Apart from that, it seems like you need to check your drive(s) for consistency (man fsck):

That’s all I have to add, at this time.

When was the last time you performed a system update?

3

Thanks for the response.

I ran out of ideas so as the last thing I did was to check for updates.

I tried to use for a long time OBS official repo and eventuality I switched to Flatpak as my last resort but I have the same problems with it too. Like I don’t know if I am missing some sort of dependency or something is wrong with my drivers.

4

There have been recent issues related to xdg-desktop-portal and flatpak, which is why I suggested the repo version; albeit probably the better choice in any case.

error: VAAPI: Failed to initialize display in vaapi_device_h264_supported
info: FFmpeg VAAPI H264 encoding not supported
error: VAAPI: Failed to initialize display in vaapi_device_hevc_supported
info: FFmpeg VAAPI HEVC encoding not supported

Is ffmpeg installed?

5

Yeah it is installed.

I have been researching so what did was I installed manjaro-pipewire
Then I installed libva-nvidia-driver just in case.

I tried to use OBS official repo and build from AUR obs-vkcapture also obs-vkcapture-git. It does see the plugin when I launch the app.
I tried to set in grub nvidia-drm.modeset=1 but with this I couldn’t even launch the games so reversed back.

Later I put into the Obs .desktop launch parameter OBS_USE_EGL=1 obs in Exec section.

Then after all the tries I installed Flatpak version of OBS at the end same results from both of them.

I have Manjaro KDE plasma x11. Not sure if OBS has issue with KDE generally.

I noticed on both OBS versions that pipewire in Screen and Window sources does not see any screens or windows running.

Like do I have just a bad luck or is this normal? :smiley:

Sometimes my Firefox or Brave web browsers hard freeze the system and I just wonder if any of this has something to do with all my troubles manjaro. Games do run fine though.

Generally My Manjaro is about 3 days old after fresh install so I don’t think I messed up anything.

I have also noticed that since the source is Game Capture and it still records (system) sounds when I don’t even run vulkan translated windows game. Is this normal?

6

This is what I forgot to mention. libva-nvidia-driver has been updated, to fix some issues that might be related; but I’m uncertain whether that version has reached Stable yet. Here’s a Post that mentions it.

So, I don’t know if the version you installed is the updated driver, or not, however I’m sure someone will jump in to either confirm of deny it.

7

Thanks for informing regarding the libva-nvidia-driver. I installed from the Add/Remove Software as official repo so not sure how things should be.

Just recently I tried to put /usr/bin/sh -c "QT_QPA_PLATFORM=xcb obs" in the exec of OBS .desktop icon and still nothing.

I have put in one of my games obs-gamecapture #command# launch options so I can record the game but nothing only a black screen and just game sounds or maybe whole system sounds since I am not certain if its even trying to capture Vulkan running game API at this point.

9

Similar issues a few months ago OBS-Studio not working after last update (back in January); several people with remarkably similar logs.

10

Thanks for the link. I did everything according what people were trying to do. Like Removing nvidia driver, reboot, install video linux reboot and vice versa, nothing. Tried to reinstall libva-nvidia-driver. Nothing.

At this point I feel like I am out of options. I just wonder what else to do besides jumping to another distro at this point.

I wonder should I just install Wayland Session at this point.

Thank you for your time, I really like manjaro and just arch generally but it seems OBS got the better of me. :smiley:

11

You could install Wayland. You could also try reverting to the 6.1 LTS kernel, or maybe try the newly released 6.6 kernel. I can’t say whether these options will improve the situation, but they might.

I have [the repo version of] OBS installed (KDE, Wayland, amdgpu, 6.6 kernel) with no apparent issues. The obvious common denominator between your issue and those on the thread previously linked, is the use of Nvidia graphics. Some highlights:

This user has a less than desirable solution, but it worked:

Remember, these issues were from back in January; the cause in your situation might be entirely different; although it seems to tick all the boxes.

12

I completely forgot the kernel change in my mind thanks for giving me some fresh new ideas.

But yeah I do believe its my RTX3070 is at fault here. My pc does have hard freezes when browsing with Firefox and Brave until I had to put in GRUB boot parameter pci=nommconf to stop that. I checked the logs and the error was PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID).

The same freezes were also on Linux Mint Ubuntu.

But yeah I will jump to 6.6 kernel and if nothing I will jump to 6.1LTS, and if that won’t work then will try to use Wayland session. :sweat_smile:

13

Another popular resolution (again, in the other thread) was to remove the libva-vdpau-driver completely. That might also be a valid option if there is an updated version forthcoming.

I mentioned reverting to the nvidia-linux driver earlier. This post (in the other thread) was also well received, with several claiming it solved their issue.

14

I did remove libva-vdpau-driver and nothing helped after that I installed libva-nvidia-driver also and still nothing.

I tried to change kernal to 6.6 and later to 6.1LTS and nothing helped. Just in case I also reinstalled nvidia driver to for each kernel.

About reverting to the nvidia-linux also did not helped my situation.

I tried to install plasma-wayland-session and I couldn’t even log in to Wayland session so back to x11.

Like the only thing I could think of is to do BIOS update.

15

What was preventing you logging in?

Did you remember to select ‘Wayland’ (bottom left corner of SDDM)?

16

To be frankly honest I don’t know. When I installed Wayland, the system required a reboot. Once the system was booting it was stuck at Linux BIOS/bootting(sorry I can’t tell what it is) screen where is says bunch of OK OK OK for each line. In there the system couldn’t boot. I had to do a hard reset for the system and it booted normally as usually.

In the Login screen I selected Wayland session and all I see is a black screen for about 4 seconds and then returns back to the Login screen.

18

Disturbing, in itself. :slight_smile:

You might have better luck installing it outside of the GUI. This tutorial explains how to perform an update in that way; it’s the same principle:

1 Like
19

I also tried this to check regarding my ffmpeg with:

ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda \                                                                                    ✔ 
      -f lavfi -i testsrc -t 50 \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 4M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 1M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 8M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 6M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 5M -f null - \
      -vf hwupload -c:a copy -c:v h264_nvenc -b:v 7M -f null -

And this is what I got:

ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.1 (GCC) 20230801
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librav1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
-vsync is deprecated. Use -fps_mode
Passing a number to -vsync is deprecated, use a string argument as described in the manual.
Input #0, lavfi, from 'testsrc':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, rgb24, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #1:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #2:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #3:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #4:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
  Stream #0:0 -> #5:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #0:0: Video: h264 (Main), cuda(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 4000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/4000000 buffer size: 8000000 vbv_delay: N/A
Output #1, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #1:0: Video: h264 (Main), cuda(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 1000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 2000000 vbv_delay: N/A
Output #2, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #2:0: Video: h264 (Main), cuda(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 8000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/8000000 buffer size: 16000000 vbv_delay: N/A
Output #3, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #3:0: Video: h264 (Main), cuda(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 6000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/6000000 buffer size: 12000000 vbv_delay: N/A
Output #4, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Stream #4:0: Video: h264 (Main), cuda(pc, gbr/unknown/unknown, progressive), 320x240 [SAR 1:1 DAR 4:3], q=2-31, 5000 kb/s, 25 fps, 25 tbn
    Metadata:
      encoder         : Lavc60.3.100 h264_nvenc
    Side data:
      cpb: bitrate max/min/avg: 0/0/5000000 buffer size: 10000000 vbv_delay: N/A
[h264_nvenc @ 0x5624e69380c0] OpenEncodeSessionEx failed: out of memory (10): (no details)
zsh: segmentation fault (core dumped)  ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -f lavfi -i  -t

I have no idea what this means but it seems I have an issue with ffmpeg

I will check about the updates as well thank you for the help :smiley:

20

Out of interest, what’s the resolution of whatever you’re trying to encode?

21

I am trying to record at 1920x1080 60fps.

So far I found about this https://github.com/keylase/nvidia-patch and trying to figure out if my NVIDIA has some sort of limitations placed in.

22

I found this similar issue with nvenc on StackOverflow. A workaround suggested was to use libx264 instead of h.264. You could also try encoding at a lower resolution, for the sake of testing, to see if it completes without:

[h264_nvenc @ 0x5624e69380c0] OpenEncodeSessionEx failed: out of memory (10): (no details)
zsh: segmentation fault (core dumped)  ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -f lavfi -i  -t

Also, can you post the output of:

inxi --admin --verbosity=7 --filter --no-host --width

Update:-

This appears to be a known issue: OBS studio crashes upon clicking Start Recording #9764. No indication of a fix, however.

“A workaround: setting software encoder instead of NVENC HEVC/H.264 (both cause crash).”

So, it’s not just you; that’s some consolation, at least. :slight_smile: