Okay, so precisely what happened in this particular instance and how can we learn from this?
The main issue with this update was removing the depricated GLX symlinks at the beginning of the update process, basically breaking OpenGL for X, breaking X11, but still allowing the pacman update process to complete on most systems. Completion of the update replaced the depricated symlinks with the updated libglvnd GLX symlinks, so after reboot everything fine.
If the update was stopped for any reason, either manually or through pacman file conflicts, no applications could be launched as the GLX symlinks were already removed. Rebooting did not help as without GLX symlinks X11 session could not be established, thus black screen.
For a majority of users that attempted to update through Octopi and Pamac the update did not complete, but the depricated GLX symlinks were removed. This lead to broken OpenGL for X, broken X11, unable to launch apps, and black screen on reboot. Remedy for this was to continue the install using pacman, either through tty, or within a Manjaro Live chroot session. Most users who experienced black screen reboots were able to recover this way.
On a small number of systems removing the depricated GLX symlinks seemed to crash the current user X session, crashing the update part way through, leaving the system in an unstable state, and in a few cases in an unrecoverable state. Not good.
There were a handful of users who for some reason manually stopped their update part way through, very bad idea. The severity of the issue they experienced was dependent on how far through the update process when they killed it. Some experienced black screen reboot issues, resolved by tty or chroot pacman remedy, others left their system is more a more serious state. These are self inflicted wounds, hopefully these users have learned their lesson and won’t do it again.
Finally, there are a small number of systems that the libglvnd simply does not seem to work on. These are probably edge case hardware configurations that were not available in the unstable and testing environments, thus were not tested. An example of this is @huladaddy and his dual Nvidia Macbook Pro setup that just doesn’t seem to work with libglnvd.
Knowing what we know now, how could have these issues been avoided?
- Update recommendation to apply from tty with
=> Potentially avoid removing GLX symlinks at the beginning of the process and instructions to install mesa, lib32-mesa and any driver packages using --force first. Then apply rest of update. Maybe.
Final question, how can we test intended stable batch updates as a whole, before applying in stable? Updates trickle into unstable and testing, thus large batch updates are not adequately tested before being made available in Stable.