Support for mesa with Vulkan update in Raspberry Pi 4

:sigh: Today I am having failed to allocate from CMA type issues with x11. It is causing plasmashell to crash. Seems to still be fine for wayland. Gremlins must have visited over the weekend.

Stopping the compositor stops the crashes. Now waiting on KDE 5.21 in hopes of a fix.

Application: ksmserver-logout-greeter (ksmserver-logout-greeter), signal: Segmentation fault

[KCrash Handler]
#4  0x0000ffff96fa19a4 in ?? () from /usr/lib/dri/vc4_dri.so
#5  0x0000ffff9d22acb0 in ?? () from /usr/lib/libEGL_mesa.so.0
#6  0x0000ffff9d22b640 in ?? () from /usr/lib/libEGL_mesa.so.0
#7  0x0000ffff9d2209bc in ?? () from /usr/lib/libEGL_mesa.so.0
#8  0x0000ffff9d212d4c in ?? () from /usr/lib/libEGL_mesa.so.0
#9  0x0000ffff9e7e0aec in ?? () from /usr/lib/qt/plugins/wayland-graphics-integration-client/libqt-plugin-wayland-egl.so
#10 0x0000ffffa5de3064 in ?? () from /usr/lib/libQt5Quick.so.5
#11 0x0000ffffa5de39bc in ?? () from /usr/lib/libQt5Quick.so.5
#12 0x0000ffffa4224888 in ?? () from /usr/lib/libQt5Core.so.5
#13 0x0000ffffa2c25f44 in start_thread () from /usr/lib/libpthread.so.0
#14 0x0000ffffa3eb795c in thread_start () from /usr/lib/libc.so.6

Thread 3 (Thread 0xffff9cd91d70 (LWP 1197) "QQmlThread"):
#1  0x0000ffffa201f0e0 in ?? () from /usr/lib/libglib-2.0.so.0
#2  0x0000ffffa201f264 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#3  0x0000ffffa44b3a24 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#4  0x0000ffffa44441d4 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#5  0x0000ffffa4222f84 in QThread::exec() () from /usr/lib/libQt5Core.so.5
#6  0x0000ffffa5a1b19c in ?? () from /usr/lib/libQt5Qml.so.5
#7  0x0000ffffa4224888 in ?? () from /usr/lib/libQt5Core.so.5
#8  0x0000ffffa2c25f44 in start_thread () from /usr/lib/libpthread.so.0
#9  0x0000ffffa3eb795c in thread_start () from /usr/lib/libc.so.6

Thread 2 (Thread 0xffff9e4c1d70 (LWP 1196) "QDBusConnection"):
#1  0x0000ffffa207f59c in g_mutex_lock () from /usr/lib/libglib-2.0.so.0
#2  0x0000ffffa201e848 in g_main_context_check () from /usr/lib/libglib-2.0.so.0
#3  0x0000ffffa201efcc in ?? () from /usr/lib/libglib-2.0.so.0
#4  0x0000ffffa201f264 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#5  0x0000ffffa44b3a24 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#6  0x0000ffffa44441d4 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#7  0x0000ffffa4222f84 in QThread::exec() () from /usr/lib/libQt5Core.so.5
#8  0x0000ffffa555a5ac in ?? () from /usr/lib/libQt5DBus.so.5
#9  0x0000ffffa4224888 in ?? () from /usr/lib/libQt5Core.so.5
#10 0x0000ffffa2c25f44 in start_thread () from /usr/lib/libpthread.so.0
#11 0x0000ffffa3eb795c in thread_start () from /usr/lib/libc.so.6

Thread 1 (Thread 0xffff9eb58a50 (LWP 1195) "ksmserver-logou"):
#1  0x0000ffffa422d92c in QWaitCondition::wait(QMutex*, QDeadlineTimer) () from /usr/lib/libQt5Core.so.5
#2  0x0000ffffa5de54a4 in ?? () from /usr/lib/libQt5Quick.so.5
#3  0x0000ffffa5de68f4 in ?? () from /usr/lib/libQt5Quick.so.5
#4  0x0000ffffa4862fb0 in QWindow::event(QEvent*) () from /usr/lib/libQt5Gui.so.5
#5  0x0000ffffa4f815ec in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/libQt5Widgets.so.5
#6  0x0000ffffa444607c in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/libQt5Core.so.5
#7  0x0000ffffa4855e18 in QGuiApplicationPrivate::processExposeEvent(QWindowSystemInterfacePrivate::ExposeEvent*) () from /usr/lib/libQt5Gui.so.5
#8  0x0000ffffa4824bec in QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Gui.so.5
#9  0x0000ffffa4824f2c in QWindowSystemInterface::flushWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Gui.so.5
#10 0x0000ffffa447e95c in QObject::event(QEvent*) () from /usr/lib/libQt5Core.so.5
#11 0x0000ffffa4f815ec in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /usr/lib/libQt5Widgets.so.5
#12 0x0000ffffa444607c in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /usr/lib/libQt5Core.so.5
#13 0x0000ffffa4449570 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /usr/lib/libQt5Core.so.5
#14 0x0000ffffa44b4780 in ?? () from /usr/lib/libQt5Core.so.5
#15 0x0000ffffa201eb80 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#16 0x0000ffffa201f160 in ?? () from /usr/lib/libglib-2.0.so.0
#17 0x0000ffffa201f264 in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
#18 0x0000ffffa44b3a08 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#19 0x0000ffffa44441d4 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /usr/lib/libQt5Core.so.5
#20 0x0000ffffa444e900 in QCoreApplication::exec() () from /usr/lib/libQt5Core.so.5
#21 0x0000aaaadb296a40 in ?? ()
#22 0x0000ffffa3e08538 in __libc_start_main () from /usr/lib/libc.so.6
#23 0x0000aaaadb296ba4 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further
[Inferior 1 (process 1195) detached]

But I still have the above. I think I will roll back to stable.

Edit: I am not sure what happened. What started off being very promising on Friday, somehow went south on me. I rolled back to my arm-stable installation and it seems to be better. No crashing at least.

Edit 2: And so of course, I have two other RPi4 on unstable and they have no issues with the latest updates.

So I rolled back and reapplied the arm-testing updates to an arm-stable install on the RPi4 that had the errors above. When I edit the /etc/environment and either comment out the QT_QUICK_BACKEND=software or change it to =vulkanrhi… splat, errors with the VC4.

Does this require a particular eeprom version? This one is still using the version dated Sept 3, 2020.

Edit: When I run rpi-eeprom-update it shows the Sept 3 installed and no update available. I know there is a December version, and I thought all that was required was to change the “FIRMWARE_RELEASE_STATUS=” from “critical” to “stable” in /etc/default/rpi-eeprom-update.

Sept 3, 2020 is their current production image and it should be in all branches. I do not know if it has anything to do with your issue but If I remember right on one of your pi’s at least you installed rpi-eeprom-git 20201214-1.

[ray@pi4 ~]$ pacman -Ss pi-eeprom
community/rpi-eeprom 2020.09.03-2
    Raspberry Pi4 boot EEPROM updater
community/rpi-eeprom-git 20201214-1 [installed]
    Raspberry Pi4 boot EEPROM updater - git version
[ray@pi4 ~]$  
[ray@pi4 ~]$ sudo rpi-eeprom-update 
BCM2711 detected
VL805 firmware in bootloader EEPROM
BOOTLOADER: up-to-date
CURRENT: Fri Dec 11 11:15:17 AM UTC 2020 (1607685317)
 LATEST: Fri Dec 11 11:15:17 AM UTC 2020 (1607685317)
 FW DIR: /lib/firmware/raspberrypi/bootloader/stable
VL805: up-to-date
CURRENT: 000138a1
 LATEST: 000138a1

ADDED:

Another thing to consider is the 2 raspberrypi-bootloader package versions?

$ sudo rpi-eeprom-update

BCM2711 detected
Dedicated VL805 EEPROM detected
BOOTLOADER: up-to-date
CURRENT: Thu Sep  3 12:11:43 PM UTC 2020 (1599135103)
 LATEST: Thu Sep  3 12:11:43 PM UTC 2020 (1599135103)
 FW DIR: /lib/firmware/raspberrypi/bootloader/stable
VL805: up-to-date
CURRENT: 000138a1
 LATEST: 000138a1

Something so basic, and yet, I am missing something.

rpi-eeprom-git? I don’t recall installing that before. But it would appear that is the what I am missing.

Edit: Unfortunately, rpi4-eeprom updates do not seem to work via network booting. So I updated via SD card to the December version.

Edit 2: And even more unfortunately for me, the eeprom update does not resolve the video driver issue.
Still have the following errors occurring:

kernel: [drm:vc4_bo_create [vc4]] *ERROR* Failed to allocate from CMA:
kernel: vc4-drm gpu: [drm]                           dumb: 243136kb BOs   (15)

The RPi4 that work for me are 8GB. The one that is giving me issues is a 4GB.

Edit 3: If I change to dtoverlay=vc4-fkms-v3d no errors on the 4GB. The performance is not as good as with kms but better than crashing and the vulkan support does work. glxgears @ ~380 fps on a 4K monitor.

Edit 4: Tested with another 8GB and it too has the errors. So I guess it has something to do with network booting, I will test more. Also, using the fkms is not enough, I have to add cma-384. A lower value might work, unsure.

dtoverlay=vc4-fkms-v3d,cma-384

rpi-eeprom got update in PiOS today.

Built and pushed to the unstable branch.

https://forum.manjaro.org/t/new-raspberry-pi-kernels-related-packages/4721/281

Well, try as I may, I can not recreate this issue on my setup at home. I searched the interwebs and I lean toward this being a 4K video issue, requiring a greater cma amount. I peeked at the kernel config, the default is a mere 5MB.

CONFIG_CMA_SIZE_MBYTES=5

@Darksky are you sure that is not a typo? 5? Kind of a weird number. I found where it was set to 32MB at one point in the past.

We have changed cma from 64 to 256 on 'Linuxandlinux-vim` package.

So it must be a typo for sure.

Ok, the kms works too, and at 4K with:

gpu_mem=128
dtoverlay=vc4-kms-v3d-pi4,cma=512

$ cat /proc/meminfo

CmaTotal:         524288 kB
CmaFree:          260048 kB

So judging from that, 384 should work and 256 might work. The CmaFree moves around as you run apps, so setting a little extra is probably not a bad thing.

The default vc4* overlays are configured for 256M. What you have done above is the proper way to increase the cma on the pi’s. 384M could work if not running any full screen high bitrate video’s.

[ray@pi4 ~]$ dmesg |grep cma
[ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[ 0.000000] Memory: 7725840K/8244224K available (11584K kernel code, 2374K rwdata, 3984K rodata, 3648K init, 1364K bss, 256240K reserved, 262144K cma-reserved)

The memory requirement is a bit lower for wayland:

$ cat /proc/meminfo

CmaTotal:         524288 kB
CmaFree:          333708 kB

So for my use with Plasma and 4K, 256 looks to be the minimum setting, but I will leave it at 384.
For reference glxgears with Plasma @ 4K: ~980 FPS (no overclocking)

Edit: I just tested with cma-256 @ 4k on xorg and it is a no-go. So 256 is not enough for my needs, 384 is my minimum.

For reference and comparison to 4K above:
Plasma on wayland @1920x1080 with kms and no cma setting
gxlgears ~1600 fps

$ cat /proc/meminfo

CmaTotal:         262144 kB
CmaFree:          181392 kB

This RPi4 is moderately overclocked:

over_voltage=6
arm_freq=1800
v3d_freq=600
#isp_freq=600
#h264_freq=600
hdmi_enable_4kp60=1

does plasma sddm start from kms, my test fail and halt.

Yes, it does for me. But my previous testing revealed that a hdmi 2.0 cable (and likely a compatible monitor) or proper hdmi_mode and hdmi_group were required. Without the settings or the cable, a black screen. For the hdmi settings, see this.

Edit: But if fkms works for you, use it. While there is some difference in fps, the user experience is about the same. You can still get hardware acceleration with vulkan support (if you are on the unstable branch).

Pi guy@6by9,

3D rendering (via OpenGL or Vulkan) also goes through DRM/KMS, but for Pi4 it uses the v3d driver instead of vc4 as the hardware had changed sufficiently to warrant a new driver.
3D uses Mesa as well as the kernel.

@0n0w1c is v3d_freq how one should overclock the GPU? I have been using gpu_freq.

# Overclock the CPU because Argon Neo keeps the Pi cool.
over_voltage=6
arm_freq=2100
gpu_freq=600

It is, according to the Raspberry Pi overclocking docs, in the section entitled Specific to Pi 4B.

@0n0w1c Yes, thanks. I noticed that after. I had earlier never read post gpu_freq from the doc. :disappointed:

I noticed that the vulkan-broadcom package breaks vulkan support with retroarch and uninstalling vulkan-broadcom and switching back to mesa-git will make it work again. That’s a shame because vblank_mode=0 glxgears reported a huge improvement with the current mesa package over mesa-git.

Hello!

Is the vulkan-broadcom driver (20.3.3-1) expected to work in the stable branch right now?

After reading this thread, I attempted to get it working, but ran into the blank screen at LightDM login page issue discussed above.

At this point, my config.txt is:

~]$ cat /boot/config.txt

initramfs initramfs-linux.img followkernel
kernel=kernel8.img
arm_64bit=1
enable_gic=1
disable_overscan=1

#enable sound
dtparam=audio=on
hdmi_drive=2

gpu_mem=128 # Set GPU memory to 128M
#enable vc4
dtoverlay=vc4-fkms-v3d,cma=512
max_framebuffers=2

#Monitor Configuration
#hdmi_enable_4kp60=1 # Enable 4K60 display

#overclock (ETA Prime @ How to Overclock Raspberry Pi 4 To 2.0Ghz! All 4 Cores - With Benchmarks - YouTube )
over_voltage=6 # Overvolt CPU
arm_freq=2000 # Overclock to 2 GHz from 1.5 GHz
v3d_freq=600
gpu_freq=600

glxgears correctly reports version 4.2 of the v3d drivers. It gives me 60fps with vysnc enabled, which is what I would expect on my 49" 4K 60fps TV operating in 1080p mode.

inxi -Fazy reports the modesetting driver with fbdev alternate (fbdev is not installed–I checked).

vulkaninfo gives me a ton of information, but at the very top:
WARNING: v3dv is neither a complete nor a conformant Vulkan implementation. Testing use only."