I would first switch to the unstable branch and install the latest:
linux-rpi5 (Version 6.6.18-1) (linux-rpi4 for the rpi4).
rpi-overlays
raspberrypi-bootloader
raspberrypi-bootloader-x
If the issue still exists with the latest packages then create an issue with the RPi kernel gitlab. I have not seen a similar issue reported here or on their gitlab.