The hardware video decoder contained within most modern Amlogic SoC’s is quite powerful but it cannot be fully or properly exploited using the current open source drivers. Today Christian Hewitt (chewitt), a prominent ARM Linux developer, gave a detailed rundown of the current state of vdec support in reply to a mpv github ticket:
"The incorrect assumption(s) are that the current (staging) vdec driver is good, the handling of 8-bit/10-bit output in the DRM layer is good, the firmware blobs from Amlogic are good, and support for v4l2m2m in ffmpeg is both also good and aligned with the current vdec. All of the variables have some question marks against them.
The vdec was developed by Maxime Jourdan in 2018/19 before the stateful UAPI started to mature and so v4l2m2m support in ffmpeg was also immature, but in combination H264 worked well and other codecs were being added - based on an 8-bit output pipeline that did not support Amlogic framebuffer correction used in Mali Utgard SoCs or ARM framebuffer correction (different, but similar) used in Mali Midgard and Bifrost SoCs. AFBC(s) are crucial for supporting 4K output else RAM speeds (esp. in older/cheaper devices) can’t sustain the throughput the video pipeline needs.
Once the initial stateful UAPI conformance tests were published in late summer 2019, some work to adapt the vdec to the tests was done by Maxime to meet them. This work focussed on the vdec itself and the conformance test-app, so didn’t result in matching changes for userspace apps like ffmpeg and gstreamer. Around this time mesa has started to support AFBC in Panfrost, and the dw-hdmi DRM bridge layer has evolved support for 10-bit output (GXBB/GXL are 10-bit internally but output 8-bit, GXM and newer are 10-bit through the full pipeline). Maxime sadly had to stop work in late 2019 for personal reasons and has never resumed. Neil Armstrong (kernel maintainer and colleague to Maxime) was able to cleanup the remaining scraps of code Maxime had lying around in test branches to finish the kernel side of conformance work, and those went upstreaam in early 2020.
Earlier this year some minor work to revive HEVC support was done and this works well (if you have the right ffmpeg source) on devices with 8-bit ouput. However 10-bit output (essential for 4K) results in hard crashes. One of Maximes original goals was to support all hardware in a single driver. We have learnt over time that HEVC/VP9 are handled quite differently in older and newer SoCs (different firmwares, different approaches to memory management, use of IOMMU, etc.) and to progress support the current vdec should be split-up to handle these differences. One example of bad is, the vdec currently needs a huge CMA reservation (896MB!) for 4K which doesn’t play nice with older devices equipped with 1GB RAM.
In the wider V4L2 world there has been great progress on the stateless UAPI (v4l2-request) but stateful is some way behind, and I perceive this is due to dependencies on closed-source firmwares (less attractive to the open-source community) and other devices that use the stateful UAPI, e.g. Exynos, iMX6, often being in the late stages of their lifecycle or with a much higher overall dependency on BSP code so there is less interest or less need for a mainline solution (Exynos is quite broken, iMX6 works okay but its world exists behind NXPs NDA-wall).
The notable stateful exception is the H264 IP in the Raspberry Pi which is under current and very active development from the Pi Foundation developers. RPI4 is also unusual since the HEVC IP is stateless, so you have use of both v4l2-request and v4l2m2m in the same chip. In their ffmpeg sources v4l2m2m is now working very (very) well, but “it takes two to tango” and it works well due to an alignment of ffmpeg changes, vdec changes, and UAPI tweaks. At some point there will be a serious attempt to upstream everything, but for now we’re still at the “make it all work” stage. “We” is the Pi devs working with Kodi (a fancy wrapper on ffmpeg) and LibreELEC (which I contribute to).
Up to a specific git-commit point, the improvements Pi devs made in ffmpeg also benefitted the Amlogic vdec and H264 playback is generally great, and with some of the nip/tuck work done on HEVC this now plays (but not seeks) reliably on the 8-bit output devices. However, once Pi devs started to tweak the Pi vdec in combination with ffmpeg we started to see a large regression. From this point the Amlogic vdec needs matching changes … and that leads into the redesign work previously mentioned.
Why put all this in a long reply? … because I’ve been looking for developer talent to work on the vdec since 2019 without much success. The gene-pool of people who can write kernel vdec code is small. The gene-pool of people who also understand ffmpeg/gstreamer etc. is smaller again. The main names on the list are all professional developeers who rightly expect to be paid for their work. It’s not impossible for “community” developers to take on the task, but it’s not a small task and thus needs a small group with a personal interest; else the commitment is too large. If anyone is interested … RSVP!"
I have been fooling around the video playback on GT King Pro and found out “Bluetooth” Enabled affected mpv --gpu-context=wayland --hwdec=v4l2m2m-copy videofile60fps with much higher frame drops on Gnome Wayland.
And also on Ubuntu-Hirsute with kernel 5.13.y (Wayland Session) mpv with HW Accl works significantly better than with kernel 5.11.0-y (Ubuntu Kernel).