The main culprit was the HDD.
Game for Linux was installed on my regular SATA HDD, should have been on M2 SSD.
Unlike what people said on reddit, ESO greatly benefits of being installed on an SSD.
If not, the game will have stutterings. And that’s why FPS remained high despite this effect - it wasn’t GPU related.