I run Gnome on wayland and when trying to recreate the situation you describe I can record video with the same tool. Why it does not let you do the same I cannot tell from the info supplied. I can choose video where your mouse is in the screenshot. Please read this the thread below and supply system information and if possible the relevant parts of the journal.
Edit: Pressing v switches between screenshot / screencast for you, does this work for you?