It's not USB (unless it's USB2), it's the drive. They all suck, as do sdcards. I never see much higher than 10MB/s with them, and certainly not for long. What you want is an SSD.
I would think the best option is a small/smallish USB3 SSD, there are a few out there. Possibly the smallest you'll find after USB keys and sdcards.
There's the M.2 to USB3 pendrives/keys, they tend to be cases and you put an M.2 in them. Should still fit in most pockets. Linux Tech Tips has a video on one, they compare different options.
Then there's the larger 2.5" external USB3 SSDs, and of course internal 2.5" SSDs and a USB3 case. Cheap cases may have a lower upper limit (mine maxes out at 160MB/s).
There's also thunderbolt NVME drives, with much higher advertised speeds. Probably about the same size as the M.2 cases.