I didn’t have the problem in the beginning and the problem hasn’t appeared since I switched to kernel 5.11. With kernel 5.10 I switched the SSD into a different machine and had the same exact issue so for now I guess it was something about kernel 5.10, which is bad because it is the LTS kernel. Here is a list with the other issues I have with this machine.
EDIT: With kernel 5.11 the problem seems less frequent but I also had it happen.