Headset only available for one account at a time

Thank you for the clarification. I was going in the wrong direction to disable headset for one user

I’ve noticed that my primary user (mho) is in the audio group (and some others), none of the other users (test) are. I don’t think that’s correct

I agree that is not correct (and this might be related to the problem with switching users)

PulseAudio - Configuration Files - ArchWiki
There is usually no need to add your user to the audio group, as PulseAudio uses [udev] and logind to give access dynamically to the currently “active” user. Exceptions would include running the machine headless so that there is no currently “active” user.