Rk3399 hwdec - followed directions from blog post with poor results

ARM Technical Assistance
1

@spikerguy

Furkan, from the team, has been hard at work lately trying to get hardware accelerated video playback working on our devices. We can now announce that we have packages available in the repository for rockchip devices to enable this functionality. But there are some caveats though. It only works with local media, it needs a special ffmpeg package based on version 4.4, only works in wayland it seems, and only works with special mediaplayers compiled with the support, which we have provided as mpv-rk and kodi-rk. So to enable this functionality on your rockchip based device, install ffmpeg-rk, kernel-api-header-60 and mpv-rk or kodi-rk.

So this was from late december 2022. I went to pull these packages and the kernel-api-header-60 doesnt seem to exist, mpv-rk also didnt exist. I found out mpv-hwdec was where this was now. I pulled it and it pulled ffmpeg-rk along. I tried these out and it does say itā€™s trying to use drm to produce the video but I dont think itā€™s working right. I tried h264 and h265 1080p and 4k videos and it couldnt manage any of them well, from inside a desktop or launched from tty

I know the hardware is capable of smooth 4k60 because Iā€™ve seen it on Libreelec. I had a good look around their kernel patches and ffmpeg patches, but excising that from the buildchain they have looked a pain. I was just looking for that kind of performance with a full linux install and not something Elec based. It looked like they were using v4l2_m2m and wrapping it in something else to escape the m2m part resulting in zero-copy output to drm. The old rkmmp stuff would also produce smooth 4k60 but who wants kernel 4.x these days.

Am I doing something wrong here?

2

For this to work you need the kernel to have that support.

Currently only linux-opi kernel package have it.

I donā€™t know which device youā€™re using so I cannot confirm if this kernel will work for you or not.

You can give it a try but do it as your own risk.

All the kernel patches are from the LE team itself so it should work exactly the same as LE on mpv and kodi only.

It will not be able to do vdec on browsers.

Btw where did you get that quoted text from?

3

https://blog.manjaro.org/december-2022-in-manjaro-arm/

4

Currently only linux-opi kernel package have it.

thanks

Im using the rockpro64. I foudn this in the gitlab/repos, but am unable to find mpv-hwdec there to look through

#  Apply LE VPU Patches
#  apply_patches 5

5000-drm-rockchip.patch                                                                                                                                                                                    
5000-v4l2-wip-rkvdec-hevc.patch                                                                                                                                                                           
5001-v4l2-rockchip.patch                                                                                                                                                                                  
5001-v4l2-wip-iep-driver.patch                                                                                                                                                                            
5002-for-libreelec.patch                                                                                                                                                                                     
5002-rockchip-from-list.patch                                                                                                                                                                             
5011-v4l2-from-list.patch                                                                                                                                                                                 
5020-drm-from-list.patch

These 5s looks like they were merged into the 4 section at some point in the version on the repos

It plays video and it looks like itā€™s still loading the cpu by about 20% on all cores. On libreelec the cpu jumps about 3-5% under 4k60 10bit h265

I tried it with h264/h265 , 1080p/4k60, 8 bit and 10 bit color

I found a 10 year old sharp aqious video with no hdr infomation in it at all and it played better, but still loaded the cpu. All of these test formats on my usb played silky smooth with no issue on LE. Something is still off

I also went hunting for libreelecā€™s kernel patches and couldnt find them, all i could find was ffmpeg patches

I am willing to keep testing if you need a tester

1 Like
5

Comparing Leā€™s cpu usage to De is not completely valid cause LE just runs kodi over gbm.

You can try the same with manjaro by disabling Display manager and only keeping kodi gbm on boot.

I have faced similar issue with 10bit but was not able to pinpoint the issue.

Hopefully we should solve it once I migrate le patched to ffmpeg-5

Thanks for testing.

6

well, I was doing a lot of calling mpv and ffmpeg | ffplay from tty, with sddm killed. It was still pretty bad playback. Even with 1080p there were giant squares in the rendering and massive screen tearing, and I could tell it was well below 60 or even 30fpsā€¦ but they did say they were hardware decoding through drm.

Hopefully we should solve it once I migrate le patched to ffmpeg-5

It does look like thats what they are patching against

PKG_NAME="ffmpeg"
PKG_VERSION="5.1.2"

and kernel

PKG_VERSION="6.1.18"
PKG_PATCH_DIRS="default"
https://github.com/LibreELEC/LibreELEC.tv/tree/master/packages/linux/patches/default

I still cant find any kernel patches from them pertaining to vpus or video. I think all the kernel things might be commited and in use upstream

7

No you need to look into linux pkg patches.

8

I did, thereā€™s nothing there about vpus

Just gamepad stuff, realtek wireless stuff, and a few randoms

9

Here

It is the same I am using.

1 Like
10

I wonder if these patches in different dirs stack on top of one another because

what is here
projects/rockchip/ffmpeg
doesnt match what is here
packages/ffmpeg

I guess it would make a lot of sense to have a global set of patches, and then device specific stuff stacked on

11

it is upto them on how they want to maintain the patches.

I will work on ffmpeg5 in the coming week.

1 Like
12

I did some messing around with this today, and thought I would leave some investigatory stuff.

I made a pkgbuild for ffmpeg 5.1.2. I applied the patches in order of least specific to most specific : pkg patches, project, and device specific patches. So ffmpeg & rockchip & 3399

Then I configured with LEā€™s configure options for my device minus the cross compile stuff. Everything built fine, and it runs.

I can ffmpeg -hwaccel drm -f null - -benchmark at 24fps on 4k60 video. I havnt quite figured out how to pipe it into ffplay. Every time I have managed to produce a picture with ffmpeg into ffplay Its about .05 fps. the hwaccel option is drm, but manually setting output formats for ffplay to recieve is a pain.

mpv-hwdec probably needs to be rebuilt against it. When I try it in itā€™s current build, it throws lots of errors about v4l2 requests and blackscreens. I went looking for the src for that one but couldnt find it on the gitlab

So my system state currently is linux-opi, ffmpeg-rockpro64(what I built), mpv-hwdec, libdrm

I think something is still not quite right or optimized. It may be their libdrm build. I saw in the manjaro repos there were libdrm and libdrm-pinebook-pro. I went looking for the src for these to compare but couldnt find it on the gitlab.

I
Their configs for libdrm seem to be:
PKG_MESON_OPTS_TARGET=ā€œ-Dnouveau=disabled
-Domap=disabled
-Dexynos=disabled
-Dtegra=disabled
-Dcairo-tests=disabled
-Dman-pages=disabled
-Dvalgrind=disabled
-Dfreedreno-kgsl=false
-Dinstall-test-programs=true
-Dudev=falseā€

Its also possible they have panfrost/gallium tweaks, as I was followinga thread through the code

if listcontains ā€œ${GRAPHIC_DRIVERS}ā€ ā€œpanfrostā€; then
GALLIUM_DRIVERS+=" kmsro panfrost"
VULKAN_DRIVERS_MESA+=" panfrost"
V4L2_SUPPORT=ā€œyesā€

I havnt a clue what gallium or kmsro is

I am learning quite a bit about how all this stuff fits together going down this rabbithole, but have to have a break because of headache lol

13

You must have missed linux-api-headers

14

Best I can tell is that v4l2-m2m engages with drm directly to make and proccess video, using zero-copy renderer patches for speed. From there it should just output to the framebuffer by way of drm. What LE calls drm(prime)

What looks like is happening in majaro is v4l2m2mā€™s output, is being pushed through panfrost and opengl and out the gpu video out instead of v4l2-m2m -->DRM ā†’ screen. Im guessing this is why 10bit color doesnt work the same way it does in gdm/drm in LE and the performance is drastically reduced across the board.

I am in proccess of building up a meta-package with the needed pieces and packages to do a full drm/v4l2-m2m pipeline on manjaro to see if it behaves the same. I dont think the opengl or panfrost peices are needed for vpu. I think those are gpu related, for speeding up renders of things like kodi menus on gbm or desktops in the case of manjaro. But who knows, Im kind of feeling my way through the darkness here

edit:

At this point I think I have basically retraced your steps. I got it all up and working and patched mpv to enable wayland support. I can play gpu drm_prime video inside of wayland context using sway. I can also play it from tty with very few dropped frames, but the screen goes black. I think im most of the way there.

It appears the second part of this is where I am hitting a wall. The special sauce is this second part, which is what I was picking up on thinking opengl and panfrost wereā€™nt neccessary to make this work.

Note that DRM PRIME acceleration consists of two parts:

  1. DRM PRIME HW decoding

  2. DRM PRIME zero-copy rendering

First one is absolutely necessary to have it enabled, otherwise VPU wonā€™t be used and instead video will be SW decoded. Second one is still very important - it bypasses GPU for rendering and instead it uses DRM planes which is much more efficient. This is important for SoCs

kodi has basically written their own code to go straight from drm_prime to screen, bypassing the gpu/wayland/opengl/panfrost. This is where the differences in performance are coming in. To do this in manjaro arm we would have to have some kind of context-switching or tty switching or otherwise explicitly avoid the gpu path. I can launch kodi-gbm from tty and do h.265 4k60. Still have issues with 10 bit color that LE doesn thave though. All the performance issues come in when the gpu comes in

15

well it looks like I am behind the learning curve. dmabuf-wayland since mpv v35(november 2022) has allowed gpu bypass and is essentially the same method *ELEC/kodi uses, inside a wayland compositor

mpv --vo=dmabuf-wayland --hwdec file

cool, goal complete. My brain hurts, but i got it to work. I think i mainly retraced steps you had already gone through, but I now understand for the most part how graphics on linux work

Something is still off with the 10 bit color accel, but this is on par with libreelec in terms of performance.

1 Like
16

Yes most arm vpu only work with drm to plane.

I patched it kodi-rk and mpv-hwdec to make it use drm.

I didnt get time to dig into 10bit yet but thatā€™s not even my priority atm.

I will work on new ffmpeg so we can have working chromium when ffmpeg-rk is installed.

Thanks for going through it in detail and sharing it for everyone to go through too.

1 Like
17

thanks this was enlightening

18

Hi @spikerguy,

mpv --vo=dmabuf-wayland --hwdec file

Does the above command also works for ā€œAmlogic Deviceā€ like GT King Pro?

Edit: Tried it on Manjaro-Arm-KDE-linux-aml-6.1.19-1 with ffmpeg-m2m and mpv-hwdec.

mpv --vo=dmabuf-wayland --hwdec=v4l2m2m-copy https://www.youtube.com/watch?v=wZnVQT_iEYo

Using hardware decoding (v4l2m2m-copy).
[hwupload] no support for this hw format
[hwupload] hardware format not supported
[hwupload] no support for this hw format
[hwupload] hardware format not supported
[autoconvert] Failed to create HW uploader for format nv12
[autoconvert] can't find video conversion for nv12
Cannot convert decoder/filter output to any format supported by the output.
Falling back to software decoding.

[autoconvert] Failed to create HW uploader for format yuv420p
[autoconvert] can't find video conversion for yuv420p
Cannot convert decoder/filter output to any format supported by the output.
Could not initialize video chain.
Video: no video

Only ā€œAudioā€ played NO Video output. How to set to ā€œdrmā€ output?

19

I think hwdec is drm or drm_prime

All the pieces ended up being a patched Linux kernel, a patched ffmpeg and a patched mpv. It looks like u might need an aml ffmpeg build

I would also use local h265 or h264 and encoded material, thereā€™s no telling what format this is pulling from YouTube

I traced the libreelec build files to find the proper options for my rockpro64 for these.

You could put libreelec on an SD card play a video with it and hit ā€˜oā€™ and it will show u what hwdec is using

20

Thanks for the info. The ffmpeg-m2m and mpv-hwdec does work on Amlogic S922X but limited.

mpv --gpu-context=wayland --hwdec=v4l2m2m-copy video-file it does work reasonably well.

[jfl@jfl-mnjro ~]$ mpv --gpu-context=wayland --hwdec=v4l2m2m-copy https://www.youtube.com/watch?v=wZnVQT_iEYo

AO: [pulse] 48000Hz stereo 2ch float
Using hardware decoding (v4l2m2m-copy).
VO: [gpu] 1920x1080 nv12
AV: 00:04:45 / 00:04:45 (100%) A-V:  0.000 Dropped: 40 Cache: 0.0s

Exiting... (End of file)
[jfl@jfl-mnjro ~]$

Just wanted try out ā€œā€“vo=dmabuf-waylandā€ but unfortunately it did not work well.