A pipewire developer just posted a way to disable MIDI support (if not needed) and this stops the high CPU usage of pipewire 1.4.1 with kernel 6.1 for me. See here: https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/4621#note_2828157
It sounds like they’re working on a more complete fix.