Tried a bunch of different protocols and all of them ended up with more or less the same speed. I suspect even with a class 10 SD card with the RPi there are limitations that just cannot be overcome with just software alone.
Could not get KDE connect to work between laptop and RPi but Dolphin + fish:// protocol I was using previously is the option I’d go with. Those on other DE’s can try wrapinator (available as flatpak) for file transfer.
Not a real answer to getting KDE connect working as stated in the subject line, but hopefully the answers above help others in a similar situation.