System becomes unresponsive for a while during and after update

[jezek]

Hello.

I’m experiencing something that didn’t happened before last update (or the update before? I don’t recall after which update, but a month ago everything was fine).

During updates and even after an update the system becomes unresponsive for a few minutes. Everything freezes, the cooling fans begin to spin and nothing happens. After ~5 minutes the system unfreezes and everything is working as expected. This happens during and after pacman and pamac updates.

I did some investigations, run htop during update and made a screenshot during the freeze and after a few minutes (after the system unfroze) this landed in my clipboard.

There are tons of pamac checkupdates -q --refresh-tmp-files-dbs --use-timestamp running commands which are causing the freezes.

Does anyone has a similar troubles or does anyone know why this happens to me and how to prevent it?

Thank you.

Note: I’m using fully updated manjaro gnome with btrfs and timeshift is setup to make a snapshot during updates.

Edit: pacmac → pamac typo

[megavolt]

@jezek

Does anyone has a similar troubles?

Never.

@jezek

Does anyone know why this happens to me and how to prevent it?

Probably you run a while/for loop and run a lot of checkupdates in parallel.

[Aragorn]

Your system only has 8 GiB of RAM. The last update was substantial in size, and those packages all need to be downloaded first (which requires a lot of caching) and decompressed (which happens in RAM). This all requires large amounts of memory. You were also running the update process while logged into a graphical user interface, which also uses RAM, and you were probably still using other applications as well, which also uses up RAM.

You were already hitting swap — see the screenshot — and under these conditions as mentioned above, swapping things in and out of memory is going to be very CPU- and I/O-intensive. The system was therefore not frozen, but your graphical user interface was, because it was being swapped out. I/O is a high-priority process, and also a slow one in comparison to direct RAM access.

The best advice I can give you when you’re going to do a full update is to completely log out of the GUI environment and run the update from within a tty session. That way, you will be freeing up more RAM, and none of the shared libraries that are being updated will be in use.

• First, close all running applications and log out of the GUI completely. You have to be looking at the login screen.

• Then, press Ctrl+Alt+F3. Log in at the tty with your own credentials. At the command prompt, type…

pamac update

• Let it finish. Do not interrupt the process and do not power down the computer, even if you think it’s frozen — doing so is guaranteed to leave your computer unbootable, because the kernels and initcpios are removed at the start of the update process and only replaced again at the end of the update process. Let the process take however long it needs.

• After the update process has finished, you will see the command prompt reappear. Then type the following command…

sudo systemctl reboot

[jezek]

@megavolt

Probably you run a while/for loop and run a lot of checkupdates in parallel.

No, I just run sudo pacman -Syu

[jezek]

@Aragorn

Your system only has 8 GiB of RAM. The last update was substantial in size, and those packages all need to be downloaded first (which requires a lot of caching) and decompressed (which happens in RAM). This all requires large amounts of memory. You were also running the update process while logged into a graphical user interface, which also uses RAM, and you were probably still using other applications as well, which also uses up RAM.

Didn’t I mention that the screenshot comes from an update with only 2 packages updating.

$ sudo pacman -Syu                                  [22:41:25]
:: Synchronizing package databases...
...
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (2) manjaro-gnome-extension-settings-20220527-2  manjaro-gnome-settings-20220527-3

Total Download Size:    0,30 MiB
Total Installed Size:  12,52 MiB
Net Upgrade Size:       0,00 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
...
(2/2) checking keys in keyring                                                                 100%
(2/2) checking package integrity                                                             100%
(2/2) loading package files                                                                  100%
(2/2) checking for file conflicts                                                            100%
(2/2) checking available disk space                                                          100%
:: Running pre-transaction hooks...
(1/1) Creating Timeshift snapshot before upgrade...
Using system disk as snapshot device for creating snapshots in BTRFS mode
...
done
:: Processing package changes...
(1/2) upgrading manjaro-gnome-settings                                                       100%
(2/2) upgrading manjaro-gnome-extension-settings                                             100%
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating system dconf databases...
(3/4) Compiling GSettings XML schema files...
(4/4) Collection of usefull scripts for Manjaro Gnome

==> Printing support installed                      [22:50:01] 

And before the update there were around 2 GB of RAM free for sure.

$ free -h                                           [22:41:08]
               total        used        free      shared  buff/cache   available
Mem:           7,5Gi       3,5Gi       2,0Gi       1,5Gi       1,9Gi       2,2Gi
Swap:           10Gi       1,2Gi       9,6Gi

I don’t think that an 12 MiB pacman update should take more than 2 GiB RAM.

And also why after a pacman command there are multiple pacmac update commands?

[jezek]

I restarted the system and all troubles were gone.

Thank you for your answers and sorry I didn’t try to restart in first place.

Have a nice day.

[jezek]

I’m here again.

After the restart I wrote in post above I’ve spotted only 2 of the pamac checkupdates -q --refresh-tmp-files-dbs --use-timestamp after launching the sudo pacman -Syu.

But after just one day, there are already 10 of them. See picture.

Note: I’m not shutting down my system, just putting it to sleep about 3 times a day.
Edit: pacmac → pamac typo

[megavolt]

@jezek

I’ve spotted only 2 of the pacmac checkupdates -q --refresh-tmp-files-dbs --use-timestamp

On my systems I never saw this command and cannot find such anywhere. So I assume it must be a custom command which was configured by yourself.

Also, if pamac would run this, then it would be run as root, but your commands are running as your own user which indicates that you run it.

[jezek]

@megavolt

On my systems I never saw this command and cannot find such anywhere. So I assume it must be a custom command which was configured by yourself.

I don’t recall to customize/configure pacman. But my memory can trick me. I’ll try to investigate.

@megavolt

Also, if pamac would run this, then it would be run as root, but your commands are running as your own user which indicates that you run it.

Ah, thank you for pointing that out. I missed this. That should mean, that it is not launched from the sudo pacman -Syu command. Maybe something that listens to some pacman singnal.

And now I realized, I misread the troublemaker command. Instead of pamac i always searched for pacmac on my system. That’s why I found nothing. I’ll try to search again for the cause.

Thanks for pointing me in the right direction.

[zbe]

If you only use CLI for package management you can ditch pamac altogether (just don’t tell anyone I told you).
Btw, nice nick, ježek.

[jezek]

So, I dug a little deeper and found a source code inside libpamac library, which launches the exact (troublesome) command as shown in htop. When I studied that code, I came to a conclusion, that the libpamac.check_updates (which launches the pamac checkupdates ... command) is launched automatically when local or sync db are changed. So you just need to create the libpamc.UpdatesChecker object and it automatically checks the db files and launches update command if dbs are changed. This could be troublesome, if more UpdateChecker objects are created or not properly destroyed, because if db file is changed all the UpdateChecker instances will try to check updates.

So I needed to know where the UpdatesChecker instance is used. After some time I found the usage in pamac tray icon code and pamac gnome-shell extension. I’m using the extension, so I tried to investigate the extension more.

I edited the extension code and added a log before the UpdateChecker creation to see, when it is created. Restarted gnome-shell, checked if the log worked. Yes it worked there was a log in journalctl:

jun 04 22:09:55 jeZenBook gnome-shell[133535]: UpdatesChecker creation

Then I periodically checked for packages with the sudo pacman -Syu command and monitored pamac processes in htop. Everything worked as expected for a day, even after sleep/wake up. After every updates checking with pacman only one pamac checkupdates ... command was launched.

Until now. I checked again some time ago and there were more pamac checkupdate ... processes. I checked journalctl and there it was.

jnn 05 21:42:25 jeZenBook gnome-shell[133535]: UpdatesChecker creation

Gnome-shell created UpdatesChecker again. Without restarting the shell or any other action.

So my theory is. Gnome-shell sometimes for some reasons restarts extensions. The previously created UpdatesChecker instance isn’t properly destroyed and new one is created. Both of the instances are monitoring db files and when the files are changed, both are running the pamac checkupdate ... command. I don’t know yet, if before the new instance creation, destroy is called on the one before, so I planted more logs into the code and if it is properly destroyed, I think the UpdateChecker instance destruction (somewhere here) should solve the problem.

I’ll try to debug more and if I find something more, I’ll report.

[jezek]

Hello, I’m back again with new observations.

After I added more logging to extension code and observed what is causing the UpdateChecker creations, I figured out that every time before the creation happens, the extension’s disable and enable functions are called. Basically for some reasons (memory, error, …) the gnome-shell disables and enables the extension sometimes (so my journalctl observations say).

This brought me to the idea, that now I know how to reproduce the erroneous behavior. Open the extension manager and disable/enable the Pamac updates indicator extension a couple of times. If you do this sufficient number of times (20±), then you should experience a temporary unresponsiveness of the system after sudo pacman -Syu command or Pamac updates.

Maybe I should file a bug report.

[andreas85]

@jezek

with btrfs and timeshift is setup to make a snapshot during updates.

You find good Information about Btrfs in the wiki

Btrfs

Did you check how full your btrfs is ?

Maybe you are near EOSPC ? Then btrfs will have a lot to do after an update

Btrfs

Tip: Check how close you are to “out of space”. Look at Device unallocated: (not at Free (estimated): ) (Could not boot due to brfs no-space condition. How should I keep using BTRFS? - #7 by andreas85)

as root: btrfs filesystem usage /

[guinux]

Yes please, this seems to be a legit bug to report.

[jezek]

I’m sure this is not btrfs low free space issue.
But thanks anyway for links and good reading material.

[jezek]

I’ll report soon.

Right now I’m testing a quick fix for the gnome extension.
The fix passed my extension disable/enable testing, now I’m waiting if it works when gnome-shell decides to disable/enable the extension by himself.

So there is great chance that the issue report comes with fix for gnome extension.

[jezek]

Issue report was filled.

Also, if you are bothered by this bug, the solution is to disable the Pamac Updates Indicator extension, or use this hotfix instead of /usr/share/gnome-shell/extensions/pamac-updates@manjaro.org/extension.js until it is fixed.