VA-API only works on the newest of nvidia and if the package libva-nvidia-driver is present.
With nvidia having no vaapi capability until recently … I dont know if its actually suggestable or not.
(no nvidia here 
)
Archwiki on hardware acceleration in general:
https://wiki.archlinux.org/title/Hardware_video_acceleration
Also … you might take my advice and ditch vlc and instead use mpv with a frontend like smplayer.
Its superior in a number of ways … but also its ‘automatic’ options dont fail half the time. 
Also might be worth mentioning the ‘Advanced’ option that may be useful for some
(again you usually wouldnt want vaapi on nvidia)