This is done by xdg-user-dirs I think.
You can try disabling the service with:
systemctl disable --now --user xdg-user-dirs-update
Might need sudo, but I don’t think it does, since it’s a user service.
You can also edit .config/user-dirs.dirs to remove the Downloads folder one.
You probably need to logout/login after making these changes.