Why do Manjaro Stable updates break the system? Is it bad testing?


#25

Perhaps our tools need a “Oh Hell No” mode, where the installers are signaled somehow to show the Forum Posting, release notes, etc, and then drop into a terminal, display scary warnings, launch a shell script, and just refuse to run in graphical mode.

(I say this even though for me Pamac did everything correctly.)

There are very few updates that need this treatment. So it doesn’t have to be pretty.


#26

No need when simply reading the announcement thread will recommend applying an update using pacman if appropriate. Personally I don’t really like the idea of notifying a user of an available system update, then not allowing them to install it.

If an alteration to tools is to be made, potentially an auto popup window with important update instructions and recommendations would be useful, as @Chrysostomus also pointed out.

If a user chooses to ignore these… well, can’t really legislate against arrogance or stupidity.

Hence many users’ apathetic attitude to updating their system, absentmindedly applying all updates either in Octopi or Pamac without the slightest clue to what is being updated. Why should they bother, they’ve been running Manjaro for XX months and haven’t had an issue. How cool is this?

It is not until an update leads to a non booting system that their attention is well and truly captured, especially when they realize it could have been avoided by reading the announcement thread and following simple instructions.

Bit like falling asleep at the wheel and waking up with the sound of long grass flapping against the front bumper as you are ploughing through a field.

Some users will learn from this, others will blame Manjaro and throw all their toys out of the cot, some will just re-install and continue as normal. Bottom line is users who want to learn will, but there is no real cure for apathy or laziness, regardless of the how much information or number of warnings are presented.


#27

:raising_hand: I think if ‘manjaro-hello’ contained a big bold message about reading the announcements thread before updating, and detailing best practices for update we could reduce the number of new(ish) users having problems with updates. Yes many will ignore but it will help some.

With the last few stable updates any advice has been posted some time after the announcement notice. (About post #80/6-8 hours after the update went ‘live’ this time around.) A placeholder disclaimer warning inexperienced users to wait until @philm has had a chance to come back and provide any needed advice could also be useful.

Manjaro is still the best! :heart_eyes_cat:


#28

People could also read the testing/unstable update announcments to get an idea of what issues there were with the update.


#29

If the update dot goes a different color just check the update announcements, problem solved for most.
It ain’t hard, folks just got to get used to reading before pressing the dot.
All folks I know that use arch always read about updates before actually updating, it’s just a sane practice that folks do and get used to.
It is just a normal practice on any rolling release, Manjaro shouldn’t be any different.
The only thing I can think of to babysit the masses is if they click on the dot to update it first takes them to the announcement page or put the info up first with the option to continue at the end of the message.
If they do something silly by bypassing the info first then they have no one to blame but themselves.
Things break on any system, but rolling releases should always be read first before any update, it’s just more saner.


#30

Please read all the story first before you assume we didn’t test it.

So we worked since the 12th of March on this issue with a lot of people involved. We knew that we can’t fix it for everybody, however for most of you. And if you can’t run your system a new ISO was there.

So please tell me what more do you expect from us?


#31

March. 12th of March. Not May. May hasn’t happened yet. :slight_smile:


#32

I’m back from the future my friend …


#33

This is crazy, you are not seriously suggesting the OS only works if you read forum?

I think, we are trying to pound a squared peg in a round hole, which of course doesn’t work.
arch based systems are not nanny systems, will never be.
To achieve more nannying, we would have to build all packages ourselves.
Its why I don’t even aim at newbie users with openrc and lxqt edition.


#34

Hi,

when I started with linux in 1994 such problems were “your daily business”.
In the first years I often had to reinstall my systems from backup to get them working again.
Years later I used LVM (or was it EVMS?) to solve the problem of having an destroyed system after updates.
That way I was able to create a snapshot and rollback in case of problems.
To safely use snapshots one has to use a partition scheme that splits root from home such that it is possible to roll back only the system part.
This system worked perfectly for years. Maybe an installation setup like this would be beneficial for all of us.

Another point I would like to mention is IMHO the main reason for that many failed updates (I have 3 systems - all three failed while following the instructions exactly).
I saw that the manjaro-system package removed symlinks to GL in a first step but only 2 packages were installed in that round (manjaro-keyring and manjaro-system).
This leads to a unusable (graphical) system for sure.
IMHO the better solution would have been to not delete the symlinks beforehand but force replace them with the updates. This way all binaries can resolve their libraries all the time. And no system outage is to be expected.
I don’t know the pacman package architecture well enough but I assume that would be possible. right?

Bernhard


#35

I very rarely read update announcements. I think I should have when I primarily used arch, but didn’t know about arch announcements back then. But usually pacman output is very sufficient, even with the “problem updates”. But a beginner does not have the confidence to act on that information, so announcements are helpful. And they save troubleshooting time also for more experienced users.

Do we have somewhere a dedicated page for announcements requiring manual intervention? That could be easier to script for than forum topics.


#36

I think its a worng go with forum announcements. No amount of threads will change wrongly established expectations and perceptions. People are too lazy to read these days, its useless, and you only reach a small percentage of users anyway.

The tribal wisdom of the Dakota Indians, passed on from one generation to the next, says that when you discover that you are riding a dead horse, the best strategy is to dismount.

But in modern business, because heavy investment factors are taken into consideration, other strategies are often tried with dead horses, including the following:

  1. Buying a stronger whip.
  2. Changing riders.
  3. Threatening the horse with termination.
  4. Appointing a committee to study the horse.
  5. Arranging to visit other sites to see how they ride dead horses.
  6. Lowering the standards so that dead horses can be included.
  7. Reclassifying the dead horse as “living-impaired”. .
  8. Hiring outside contractors to ride the dead horse. .
  9. Harnessing several dead horses together to increase speed.
  10. Providing additional funding and/or training to increase the dead horse’s performance.
  11. Doing a productivity study to see if lighter riders would improve the dead horse’s performance.
  12. Declaring that the dead horse carries lower overhead and therefore contributes more to the bottom line than some other horses.
  13. Rewriting the expected performance requirements for all horses.
  14. Promoting the dead horse to a supervisory position.

Thats so true, and I am with the Dakota Indians on this one, they are sane, the modern world is not.


#37

You may have missed the message where I was saying I’m not pointing finger at anyone, and certainly not at the people who tested the update since unstable and warned early of the incoming troubles.

Now, I really, really appreciate all the work, dedication and love that’s poured into Manjaro to make it an awesome distribution for everyone.

I also vaguely understand the technical difficulties. I haven’t had the time to properly dive into the whole framework, but I use linux since kernel 1.2 (slackware), and it’s my main desktop since 1998. I have used since about all the big names (RH, Mandrake, Debian, Fedora, and even FreeBSD for a while…), and even built once a LFS. So I imagine easily the challenges. Really, I do. (Full disclosure : while my friends would probably describe me as a massive nerd, I have no background in computer science, or engineering ; I studied law science, and now I work in real estate. I use linux because I believe security, ownership of one’s own data and long-term ability to access your documents are paramount parameters of my job).

What I was saying isn’t technical at all. It’s “marketing” or “psychology” or whatever you like, and it won’t be solved by a technical solution. There needs to be a warning system for potentially catastrophic updates, they can’t appear like ordinary benign and simple upgrades. You can’t have on the one hand “user-friendly Linux at its best” on the top of the website main page, and explain to people whom computers have been ruined that they should have subscribed to all the traffic of the unstable forum. Conflicting messages is what doomed HAL in 2001 Space Odyssey. Don’t make Manjaro be HAL.

Cheers,

M.


#38

Okay, so precisely what happened in this particular instance and how can we learn from this?

The main issue with this update was removing the depricated GLX symlinks at the beginning of the update process, basically breaking OpenGL for X, breaking X11, but still allowing the pacman update process to complete on most systems. Completion of the update replaced the depricated symlinks with the updated libglvnd GLX symlinks, so after reboot everything fine.

If the update was stopped for any reason, either manually or through pacman file conflicts, no applications could be launched as the GLX symlinks were already removed. Rebooting did not help as without GLX symlinks X11 session could not be established, thus black screen.

For a majority of users that attempted to update through Octopi and Pamac the update did not complete, but the depricated GLX symlinks were removed. This lead to broken OpenGL for X, broken X11, unable to launch apps, and black screen on reboot. Remedy for this was to continue the install using pacman, either through tty, or within a Manjaro Live chroot session. Most users who experienced black screen reboots were able to recover this way.

On a small number of systems removing the depricated GLX symlinks seemed to crash the current user X session, crashing the update part way through, leaving the system in an unstable state, and in a few cases in an unrecoverable state. Not good.

There were a handful of users who for some reason manually stopped their update part way through, very bad idea. The severity of the issue they experienced was dependent on how far through the update process when they killed it. Some experienced black screen reboot issues, resolved by tty or chroot pacman remedy, others left their system is more a more serious state. These are self inflicted wounds, hopefully these users have learned their lesson and won’t do it again.

Finally, there are a small number of systems that the libglvnd simply does not seem to work on. These are probably edge case hardware configurations that were not available in the unstable and testing environments, thus were not tested. An example of this is @huladaddy and his dual Nvidia Macbook Pro setup that just doesn’t seem to work with libglnvd.

Knowing what we know now, how could have these issues been avoided?

  1. Update recommendation to apply from tty with pacman -Syyu.

What else?

=> Potentially avoid removing GLX symlinks at the beginning of the process and instructions to install mesa, lib32-mesa and any driver packages using --force first. Then apply rest of update. Maybe.

Final question, how can we test intended stable batch updates as a whole, before applying in stable? Updates trickle into unstable and testing, thus large batch updates are not adequately tested before being made available in Stable.


#39

The main problem with announcements on forum:

  • thread gets quickly simply messy, and stuff gets buried quickly
  • announcements require an attention span beyond 160 characters junk info style

here we are, its a social problem first of all, and we sadly can’t fix this just in manjaro. It needs to be fixed more broadly.

On top of that, using pacman in terminal, always, is still the best way to handle pacman.


#40

If a user is too lazy to memorize all the pacman commands (like I am), there are terminal programs like
pacui


                      .: PacUI - Package manager :.                     
 ┌─────────────────────────────────────────────────────────────────────┐
 │    1   Update System                 2   Clean System               │
 │    3   Install Packages              4   Remove Packages + Deps     │
 │---------------------------------------------------------------------│
 │    5   Package Information           6   List Package Files         │
 │    7   Dependency Tree               8   Reverse Dependency Tree    │
 └─────────────────────────────────────────────────────────────────────┘
      9   Pacman Log                   10   Reverse File Search         
     11   Roll Back System             12   Fix Pacman Errors           
     13   Configure System                                              
     15   Force Update System          16   Empty Package Cache        
     17   Force Install Packages       18   Force Remove Packages      
 ┌─────────────────────────────────────────────────────────────────────┐
 │   19   Force Update AUR             20   List Installed from AUR    │
 └─────────────────────────────────────────────────────────────────────┘

  Enter number or marked letter(s)   -   00   Help   -   0   Quit PacUI 

or pacli

                      ::Pacli - Package manager::
 ┌────────────────────────────────────────────────────────────────────┐
 │    1   Update System                2   Clean System               │
 │    3   Install Package              4   Remove Package + Deps      │
 │    5   Package Information          6   List Local Package Files   │
 │    7   Dependency Tree              8   Reverse Dependency Tree    │
 └────────────────────────────────────────────────────────────────────┘
 ┌────────────────────────────────────────────────────────────────────┐
 │    9   Defragment Database         10   Help                       │
 │   11   Downgrade Packages          12   Pacman Log                 │
 │   13   Fix Errors                  14   Configure Pacman           │
 │   15   Force Install Package       16   Force Update System        │
 │   17   Force Remove Package        18   Empty Package Cache        │
 └────────────────────────────────────────────────────────────────────┘

 ┌────────────────────────────────────────────────────────────────────┐
 │   19   Update AUR                  20   Force Update AUR           │
 │   21   Search+Install from AUR     22   Install from AUR           │
 │   23   List Installed from AUR     24   Configure Yaourt           │
 └────────────────────────────────────────────────────────────────────┘


     Enter a number between 0..24 and press [Enter] - 0 Exit Pacli

#41

I would advise not to use wrappers, no matter terminal or otherwise, such as gui.

I want to see pacman output when it runs. :wink:


#42

I don’t see it like this. An update process can fail for many reasons and ideally the system should still be able to boot (even if it is in rescue mode).
In an ideal world the system would rollback to an older snapshot and the user can restart the update process again from that point. I think users should always having the feel of being completely safe - especially new users.

One solution to this problem would be another tier - like the fast ring in Windows 10. I really don’t like that term but it is a good way to selectively update users from one version to another.
Also the concept of update waves as used by Android rollouts is a good one I think.
Unfortunately I don’t have a practical idea how to solve that with the current mirror system.

Not only terminal - it had to be text terminal (tty) in this case. My first installation went wrong because I used a graphical terminal inside X (even though I knew that GLX updated - I know somehow stupid).


#43

The terminal output is visible with both, you also get the chance to answer y/n when needed. The only hidden part is the full command, but the script is readable.


#44

Hi everyone. I wan’t to share my experience using the os and try to give some perspective for linux converts such as me. I’ve been using manjaro for a month now and I get stressed when there’s a new update notification. To start with, the first boot after I installed manjaro i3, I had these problems:

  1. no sound
  2. the touchpad won’t tap to click
  3. suspend/closing the laptop lid would actually hang the computer once opened and then (1) again
  4. sound would just randomly break (like I was literally just watching a youtube video and the sound is gone); and
  5. sound would break if I remove the charger while the comptuer boots (after grub but before login screen; happened twice)

I manage to fix (1) and (2) but (4) is still pestering me, and honestly, for (3), I won’t even try now as that just borks more stuff. I’m now treating my laptop like a baby by making sure I don’t do other stuff while its doing some terminal stuff, and this “solved” my problem with (5).

Now comes some updates. If the package has any dependency with pulse or alsa I try to hold it off untill I check the forums if people has issues with it. Or at least that’s what makes sense to me (again, I’m not very much familiar with linux and its updates). Sometimes it doesn’t matter as sound will still be broken when I reboot after the update and would either come back if I seek the solution or just wait until it returns (seriously, I have no sound for five days and it just returned without me trying to fix it). For the latest “stable” update, I would actually call it my worst experience yet, because it forced me to use Windows 10 for two days. Fortunately, there was a solution for it.

I’m not a linux veteran. This is my first distro that I managed to use for 1 whole month. When everything works (including sound) it’s actually quite nice, even if I can’t suspend like I used to in Windows 10. I don’t know exactly the reason why I choose to install manjaro. I saw it on distrowatch and got curious because of how they advertise the os as “user-friendly” and “stable” but I wished they had made it more clear who the “user” is (or maybe I wish I read some reviews before diving in). I get that they are a small team and they can’t test every system there is. I only wish they had disclaimers for these statements on their website:

Professional and user-friendly Linux at its best.

Manjaro is a user-friendly Linux distribution based on the independently developed Arch operating system.

…with a focus on user-friendliness and accessibility

Manjaro is suitable for newcomers

Manjaro Linux targets beginners and advanced users at the same time…

…Its own dedicated software repositories to ensure delivery of fully tested and stable software packages

Some people in the forum think that users are at fault because they aren’t doing it the arch way. I guess that’s true and I don’t know why I keep using it while still expecting it to be stable. Windows updater never told me to check microsoft.com to see if the following update will cause problems (and rarely does it break something, at least for me). It’s just now that it made me think that rolling distros aren’t for me because I’m in a period of adjustment. Windows gave me some convience that I didn’t have to think about before and I got used to it.

I’m still using manjaro but I’ll start trying other distros (My first installation is actually xubuntu, but it didn’t work well with my system. Manjaro was my second attempt to try a linux distro).

Sorry for rambling, I’m so stressed this past few days and I finally got it all working same as it was before the update. Still, I appreciate that there’s a community forum for when troubles like this happen.

EDIT: I’m using Solus now.