If xorg is used, gdm is running an extra instance of xorg, gnome-shell and all the gnome-settings-daemon components. They don't actually do anything during the desktop session, they are just waiting for you to return to the login screen. The workaround is a postlogin script that kills the gdm xorg and gnome-shell after you log in, so you are using resources only on the session you are actually using. When you log out, gdm components are automatically restarted by the gdm service. The only adverse effect is seeing your bootsplash or console output briefly when you log in or out.
If you want a smoother experience, then you might want to try out wayland session. It also uses a bit more resources than the xorg version, but at least it is not wasted on running something that you cannot use.