if the only hardware decoding is h264 firefox won’t work miracles because it uses local ffmpeg
Tried the firefox option settings above and use enhanced-h264ify to select h264 video only but unfortunately firefox_v137.0.1 does not seems to use the ffmpeg-v4l2-request 7.1.1-2, so no vpu hw acceleration even for h264 youtube.
Then use yt-dlp/mpv. Personally, with this RPI5 I can play 2K (vp9 av1) sdr youtube videos in software. And h265 hdr (with tone-mapping) 4k in hardware.
On vendors bsp kernels 6.1.75-joshua-git or 6.1.99-armbian-git, vpu hw acceleration is available on mpv and chromium-mpp using libv4l-rkmpp-git, ffmpep-mpp-git and mpp-git up to 4K/60 which support vp9, h264 and av1 (not sure whether av1 4K/60 is supported).
On mainline kernel-6.14.y vpu hw acceleration support is limited currently.