It’s difficult to have an opinion on and while I have not tried the Xfce persistent image - I have tried several USB installations.
When you have ruled out bad stick quality and the system’s general responsiveness it is an issue with the installation itself.
For example - what filesystem is used? ext4 is fine but for an USB installation I have found f2fs to perform significantly better.
Other things is how the files are scattered in the image - I mean - if files are fragmented then the system must jump between locations which can cause a noticeable delay when reading files.
My test system is a cheap Yepo laptop and I have used cheap Kingston 32GB USB3 devices with a handcrafted LXDE and this config runs almost as from the system SSD.
But as noted - it is difficult to say.
I will link to my notes - in a moment - when I have found them 
(this one is with LUKS encryption) but it is fairly easy to do without.