Bizarre Pamac & Pacman behavior: used 62GiB memory, took 10min to refresh DBs

Support
,
1

Earlier this afternoon (Fri Jan 5, 2024), my Pamac and Pacman were broken in a really bizarre way.

When I attempted to check for updates via Pamac, Pamac filled my RAM (62GiB) with garbage then started using M2 storage as virtual RAM, and took about 10 minutes to update databases. Then on attempting to actually update 4 AUR packages, Pamac crashed.

So I tried the same thing with Pacman, but it also failed, in an interesting way:
%sudo pacman -S -yy -u --needed
Job start time: 01:26:34 PM
[sudo] password for aragorn:
:: Synchronizing package databases…
core 144.7 KiB 348 KiB/s 00:00 [######################################] 100%
extra 8.6 MiB 33.3 MiB/s 00:00 [######################################] 100%
community 29.0 B 674 B/s 00:00 [######################################] 100%
multilib 143.5 KiB 2.98 MiB/s 00:00 [######################################] 100%
:: Starting full system upgrade…
error: could not read db ‘extra’ (Damaged tar archive)
error: could not read db ‘extra’ (Damaged tar archive)
error: could not read db ‘extra’ (Damaged tar archive)
error: could not read db ‘extra’ (Damaged tar archive)
error: could not read db ‘extra’ (Damaged tar archive)
continuing with hundreds more lines reading “error: could not read db ‘extra’ (Damaged tar archive)”, and slowly filling up all 62GiB of my RAM with garbage. Normally this would only use about 1 or 2 GiB, right? 62GiB seems outlandish. And something tells me that the “extra” db isn’t actually damaged, Pacman just thinks it’s damaged

So I tried this and it apparently worked:
First I ran “sudo pacman -S -yy” as a separate transaction.
Then I ran “sudo pacman -S -u” as a separate transaction.
Those both succeeded, quickly, with the net effect of updating my Brave browser only. So clearly nothing was actually wrong with the “extra” db; Pacman and Pamac were just getting confused somehow.

Afterward, however, I noted these two issues:

  1. The 4 AUR programs that Pamac had been attempting to update are no-longer flagged as needing update. I don’t remember exactly what they were. Unknown if these actually got updated or not, or whether they’re still on my system, or whether they still work.

  2. My “Pamac Plasma System Tray Icon” (which was a AUR, and, ironically, was one of the 4 AUR programs which Pamac was trying to upgrade which caused this whole fiasco) was missing (broken? corrupted?). So I uninstalled it and installed the official-repository version instead.

So the problem seems to have been fixed when I did “sudo pacman -S -yy” followed by “sudo pacman -S -u”, as separate transactions. Weird.

Has anyone else been having similar problems with Pamac and/or Pacman? Or was this just a bizarre one-off glitch?

2

Why the --needed flag?
Thats useful for install strings to skip already installed packages.
For sync+upgrade I could only guess it would fail or be ignored.

Also … why the forced resync?

Unless explicitly needing something else … the normal upgrade command with pacman is

sudo pacman -Syu

This repo is also deprecated.
Which is of course a problem for you in general …
But it also probably denotes neglected pacnews.
To print all of them:

pacdiff -o

Anyhoo back to your original problem … you likely force-resynced (+ weird needed flag) on a bad mirror. The deprecated repo would not help in this regard. And who knows what else with pacnews.
You certainly need to handle the one for /etc/pacman.conf, afterwards you should sort mirrors.

sudo pacman-mirrors -f

Or … if you have a dated mirror pool that needs a reset

sudo pacman-mirrors -c all

Afterwards make sure to sync+upgrade

sudo pacman -Syu
3

To avoid things not needed.

Because I was already suspecting a database problem. And, indeed, when I did “sudo pacman -S -yy”, then “sudo pacman -S -u”, in that order, as two separate transactions, it appears to have cured whatever this bizarre problem was.

I know; however, I don’t know how to get rid of it (or even if I should get rid of it). At 29 bytes, it can’t have much in it anyway.

Ok, that gives:
/etc/bash.bashrc.pacnew
/etc/profile.pacnew
/etc/shells.pacnew
/etc/pam.d/system-auth.pacnew
/etc/login.defs.pacnew

Those appear to be alternate versions of 5 existing settings files. They have contents varying either slightly or drastically from the real files, and would cripple my system if I took those suggestions. No. Those files are already fine the way they are; the versions suggested by “pacnew” (whatever that is) would not work. So lets get rid of those…

%mv /etc/bash.bashrc.pacnew /home/aragorn/Data/Kled/Cut/Excalibur/etc/
%mv /etc/profile.pacnew /home/aragorn/Data/Kled/Cut/Excalibur/etc/
%mv /etc/shells.pacnew /home/aragorn/Data/Kled/Cut/Excalibur/etc/
%mv /etc/pam.d/system-auth.pacnew /home/aragorn/Data/Kled/Cut/Excalibur/etc/pam.d/
%mv /etc/login.defs.pacnew /home/aragorn/Data/Kled/Cut/Excalibur/etc/

There. “pacdiff -o” now prints nothing.

However, I don’t see any connection between those files and the weird pamac/pacman glitch I experienced earlier today, or with how to get rid of the useless, empty “community” repo.

“One” what? pacnew? There was no pacnew for pacman.conf. Let me see what’s in that file…

Ah, it does have the following lines, which I can see may cause problems:

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

Should I just erase that section? Na, I’ll comment it out:

#[community]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

That should deactivate those lines. As for the database file for the “community” repo, I have no idea where that is, or whether I should erase it or what.

I suspect that was probably it.

Well, let me see what’s in my mirrorlist… yikes, mirrors from all over, including Turkey, Ukraine, China, etc. Since I’m in USA, I think it makes more sense to use US mirrors, so I did “sudo pacman-mirrors -c United_States”. My mirrorlist now contains just 20 mirrors, all in USA.

All of which fails to definitively explain the “pacman fills 62GiB of memory with garbage” glitch, though I think your “bad mirror” guess seems like the most-likely option. But unless this weird glitch recurs, we may never know for sure.

4

  • So … there was no reason for the needed flag. During a sync+upgrade all packages are needed or none are. As already pointed out above.

  • Thats not how to handle pacnews. Please use the search function to learn how to compare, merge, etc. Its not something you can simply ignore.
    (what do you expect to happen when you keep a configuration file using old syntax for software that has upgraded to different terms and variables?)

  • -Syyu (or whatever else) is no different than separating the flags. Doing -Syy and then following up with -Su is likewise no different than -Syyu besides the short time difference between the sync and upgrade. Usually avoided for the miniscule possibility of partial upgrades.

  • There are reasons using the force resync is not recommended. As pointed out elsewheres doing such a forced resync on a bad mirror can actually create problems that would not otherwise exist.

  • We prefer -Syu also because 99.9% nothing else is needed, while it avoids the absolute thrashing the servers receive when loads of people are superfluously invoking yy

  • To get rid of the old db (and optionally any uninstalled package cache)

sudo pacman -Sc

PS.
Yes, I meant /etc/pacman.conf.pacnew (as being ‘the one’) … I suspect you may have similarly deleted it … as all systems old enough to have that repo would have gotten such a pacnew during updates. Or else if a system was installed from an ISO created after the change … the entry simply would not have been there.

5

To establish a fact

pamac != pacman

pacman is the native Arch Linux package manager - while pamac is a GTK GUI developer by a founding member of Manjaro.

 $ man pacman
[...]
BUGS
       Bugs? You must be kidding; there are no bugs in this
       software. But if we happen to be wrong, submit a bug report
       with as much detail as possible at the Arch Linux Bug Tracker
       in the Pacman section.
[...]

So your problem likely boils down to one or more the following

  • file-system error
  • disk error
  • connection error
  • configuration error - mirror list
1 Like
6

Ah, OK. Done. Thanks.

Well, perhaps not, but the last time I tried messing around with the official tools for comparing and merging pacnew files it was anything but straightforward. So I’ve tended to take a “laissez faire” approach when Manjaro generates a “pacnew” file suggesting changes to a file “X”: if something’s broken in a way that implicates file “X” then I’ll spend time comparing and merging. But otherwise, if X ain’t broken, I don’t mess with it, because it might make things work worse instead of better, and because my time, energy, patience, and attention are not free.

Yes, I should eventually teach myself to become fluent with comparing and merging pacnew files. But not tonight; I’m too sick, tired, and groggy; if I mess with system guts I’ll just break something; it’s a project for another day.

For now, everything works, ignored pacnew files notwithstanding. :slight_smile: And that really is amazing. Having an OS that works this well would have seemed like science fiction to me just 5 years ago.

7

I would like some kind of example of what sort of problem it would create.

Obviously it shouldn’t be done, but if you do a regular sync of the bad mirror, and you force a sync of the same bad mirror, to me it seems like you would end up with the same mess either way. This isn’t something I’ve tried and maybe I’ve overlooked it somehow, but I haven’t found any examples of such problems it would create.

8

It was maybe @linux-aarhus who illustrated an example.
If memory serves it was a scenario of a bad mirror … where if you -Syu it results in no sync, but -Syyu forces the sync resulting in a mangled db or similar.

9

That’s interesting. To me it seems if a normal -Syu knows to not to sync a bad mirror then it should also know when forcing a refresh -Syyu to also know not to sync the bad mirror.

Anyway thanks for the answer, I’m gonna play around with it to see for myself, as in the past a normal -Syu on an out-of-date mirror resulted in borking my system, that was over four years ago, maybe things changed?

10

Its the ‘force’ being operative.

Something like

“Mirror is weird” > Syu > No sync refresh
“Mirror is weird” > -Syyu > Forced sync refresh

I guess I’ve been using ‘sync’ … when really thats the -S part … refresh, as for the package database.

11

That must be a feature that is very new (within the last few years) else it would not have left my system in a partial upgrade that needed to be fixed.

12

Its true that in the last few years there was some smarts added.
(hence the change of no longer suggesting ‘-Syyu’ even after pacman-mirrors, where previously it was a requirement)
Dont ask me for an exact date. I just know how it was vs how it is.

13

And there’s still room for more improvement.

14

No - they are not broken

I do think I am being misrepresented here - I don’t recall exemplifying anything like this.

Since I have been tagged - I have to say that I have had various observations over the years regarding pacman and how it handles sync.

Because I didn’t dive into the pacman code - I was heavily occupied in coding pacman-mirrors and at the same time learning the Python language - my understanding may have been flawed.

As you may recognise - this learning process may have led me to pass inaccurate understanding of the sync process onto my fellow community members - please accept my sincere apology for such inaccurate understanding.

With the HTTP protocol you can intercept the answer

  • query the Last-Modified header
  • compare the value to the value of an existing file
  • if the comparison checks out you can abort the request
  • or continue downloading

When you use a single -y, pacman will compare the date of an existing metadata file and skip downloading if the dates are equal or newer.

How exactly the comparison is made will require a deep dive into the pacman and libalpm source code - I have not done so.

On the other hand, using a doubled -yy will instruct pacman to ignore Last-Modified and download anyway.

I have gathered some experience running a mirror and I know that file systems may be corrupted especially on SSD so I know from experience - this is possible.

Because mirrors sync at different intervals and because those syncs are done using rsync - mirrors may or may not use the --delay-updates option - there is a slight chance of short lived inconsistencies which may lead to 404 (Not Found) errors.

If a data corruption happens - the file may download but the content is garbage - thus the failure.

The only viable option for a member consuming that particular mirror will be to create a custom pool and exclude the mirror.

The short mirror story

I provided an official for some years 2017-2021 if memory serves and that ran without hick-ups on my Synology diskstation.

When I a new workstation I repurposed the previous to run the mirror i5 quad core with 32G ram on 500Mbit fibre. The device was equipped with a PCI card with room for 4 mSATA SSD disks.

This was setup using and md/raid10 spanning four disk using EXT4 - and just like before the mirror synced every 5m with the mother repo.

BUT within a few months the file system became corrupted with no warnings the raid went offline. This happened several times over the next year - I eventually took down the mirror - it had become a burden - not a pleasure.

I still mirror Manjaro unofficially - this time on a RPI4B 8GB and 500GB m2 SSD.

Index of /repo/

This nothing new - it has always been so - if you sync correct you will never end in a partial sync.

You cannot create a partial sync using either -Syu or -Syyu.

The double -yy only force a sync of the metadata - and this is recommended when you switch mirrors using pacman-mirrors as you cannot predict if new primary mirrors metadata is newer or older than the current.

It has been argued that pacman is capable of updating the metadata thus the -Syyu has been changed to -Syu on the wiki and that change will only create 404 (Not Found) errors - never partial sync.

But as most is using pamac and pamac has a history of metadata inconsistencies - the use of the doubled -yyu has been necessary.

The only way you can create a partial upgrade is by using and up-to-date metadata to install one or more package(s).

Never do pacman -Sy <pkgname> as this is the recipe to a partial sync state.

15

There is no way to characterize stack overflow filling 64GiB of RAM with garbage and slowing the system to near standstill due to then going into “paging” mode, and thrashing my m.2 modules in the process – which was what was happening – as being “not broken”. And this was happening with both Pamac and Pacman, on my main home computer only. So, yes, something is “broken” to the extent that this behavior was possible. I suspect the bug is in the common library between Pamac and Pacman. Somewhere in the code, an infinite loop with unlimited memory leak is possible under some specific circumstance.

Personally, I don’t find this surprising. Not all edge cases can be tested, because it’s impossible to imagine all possible scenarios. And Pamac / Pacman are not programs with billions of users. Maybe not even millions. How many people are using this software? 100,000? Something like that? The smaller the user base, the more “interesting” behavior can slip through testing.

The exact trigger of the infinite loop remains unproven, though the “bad mirror” hypothesis seems a good guess to me, especially seeing as how immediately after I refreshed mirrors to use “currently-up-to-date USA servers only” the problem want away, and seeing as how none of my other Manjaro computers (I use Manjaro both at home and at work) were experiencing this.

My takeaway from this is three things, really:

  1. Keep an eye on my memory-usage meter when doing Pamac or Pacman updates and force-quit if it gets above 10GiB, because it’s shown the ability to get into a loop and use unlimited memory (murder on m.2 SSD if it goes into paging).

  2. Frequently update mirror lists.

  3. Check pacdiff after all major updates and merge all pacnew files. (I’ve been practicing this. I think this will be easier if I change default diff-checker from “vi -d” to “meld”, as I’m not a vi person.)

16

I just saw another topic Pamac Update Manager causes the system to freeze and while your initial topic suggests pacman is at fault - which is very unlikely - it appears similar with respect to the memory filling up.

It could be that your pamac update nag was causing the initial clogging.

I suggest to disable pamac update nag - and use only the native Arch Linux pacman.

not an error of the package manager

%sudo pacman -S -yy -u --needed
Job start time: 01:26:34 PM
[sudo] password for aragorn:
:: Synchronizing package databases…
core 144.7 KiB 348 KiB/s 00:00 [######################################] 100%
extra 8.6 MiB 33.3 MiB/s 00:00 [######################################] 100%
community 29.0 B 674 B/s 00:00 [######################################] 100%
multilib 143.5 KiB 2.98 MiB/s 00:00 [######################################] 100%
:: Starting full system upgrade…
error: could not read db ‘extra’ (Damaged tar archive)

The db’s appears to be fetched correctly but when pacman is reading the files the content does not match what is expected - and is thrown away.

When pacman encounters an error it continues with the next mirror in the mirror list. When all mirrors has failed to provide the data - pacman errors out.

So either the filesystem on your local system produces an error which makes pacman discard the database or all mirrors provide the same error.

If all mirrors provide the same error - it is likely caused by your network connection - as it is next to impossible that all mirrors would have the same connection problem.

Nothing to do with pacman

The native Arch Linux package manager pacman does not handle custom AUR scripts

I think something is missing here.

If your system is seeing custom packages as updated after running

pacman -Syyu

Then you must have done something to rebuild your custom scripts prior to running pacman.

17

My issue started last night. Pamac was set to search for updates every 3 hours, this was the default setting from the last install - which I think was over a year ago but can’t say it has anything to do with any recent changes to calamares. And the swap file should be sufficient considering my memory is 16gb. When I realized it was the update part of pamac consuming my system memory and ultimately freezing the computer I turned off the auto update function. And restarted the computer - which now works normally.

However when initiating an update pamac again races to consume system memory - and that’s the bug I have. I have not let it run it’s course as usually this results in the computer freezing so I just kill the process in task manager.

18

I have this problem as of tonight, It’s not Pacman its pamac background check that is consuming all memory and then starting to consuming all swap file as well causing my computer to be completely unresponsive and resulting in me having to press and hold my power button down to turn off my computer.

Rebooting and getting system monitor loaded before pamac could start it’s background check is how i discovered that it’s the background checker as i watched it just eat memory at an alarming rate, killing it stop it for a while but it soon run the checker again and started eating all my memory and swap.

For now i have loaded pamac gui and set it not to check for Updates and AUR Updates and turned off pamac gnome extension, so far it has not tried to background check, but i have not rebooted the computer yet.

If after a reboot it happens again i will have to uninstall all pamac files, unless someone knows what the hell is going on and why, and of cause how to fix this problem?

19

The developer is aware of the situation - as a fact - it was the developer, whom pinpointed the issue to be the AUR part of the GUI causing the issue.

There is no need to remove pamac - simply disable AUR for the GUI.

You will still be able to use command line pamac for searching AUR and building custom scripts.

20

So far after a few reboots/powered off with Pamac GUI check updates and AUR check updates turned off it has so far not happened again, Oh and Gnome Pamac updates indicator extension turned off.

I only used pamac to install a few AUR Programs (E.G. CKB-Next), all normal updates i do with Pacman and always have, and only used the Gnome Pamac updates indicator extension to let me know when updates are ready, then drop to TTY and update with pacman.

So i just have to run Pacman from terminal every day to check for updates, but will need to check AUR website manually to see if AUR Updates, unless i can somehow check and update AUR Programs without pamac?