Maybe kernel 6.1 and newer needs configuration changes

Heya, For reference I do not notice any difference in performance with the 6.1 kernel on hardware and virtuals (I use libvirt/qemu with virtmanager)

The extra memory usage would not bother me, anything in ram is available faster then from disk so that is more of a plus I’d say.

Post some hard data / times so other users can compare with your specific situation.

An option is to build your own and see what works for you.