I got this problem when i siwtched to my new amd system and used digital audio connection. Ignored it for years, but some months ago decided to take a look and the fix was quite simple, disable the power saving feature of PA. I don’t remember exact feature name nor where i saw it mentioned, but should be quite easy to find spending some time with google.
edit: it was the first one here How to disable power savings in Linux | Audio Science Review (ASR) Forum