Windows and MacOS are the only ones who do fractional scaling properly, maybe Sway (Wayland) does as well.
I use a Retina Macbook Pro (2880x1800, which is too small, 2x scaling is too big) and I don’t bother using fractional scaling because it’s nothing but problems. I just set my screen to 1920x1200 (i.e. one of the supported built-in modes) and it looks great. So maybe just pick a different available resolution and let the hardware do what it does