Pamac/pacman 'forget' their mirrors

Since this last update, which was applied without issues, pamac/pacman seem to somehow ‘forget’ their mirrors from time to time.

Happened three times for me thus far, pamac GUI complains that there are no mirrors configured when trying to install a package and, indeed, /etc/pacman.d/mirrorlist somehow got emptied out:

##
## Manjaro Linux custom mirrorlist
## Generated on 2025-07-31 11:02
##
## Please use 'pacman-mirrors -id' To reset custom mirrorlist
## Please use 'pacman-mirrors -c all' To reset custom mirrorlist
## To remove custom config run  'pacman-mirrors -c all'
##

Running pacman-mirrors fixes the issue, /etc/pacman.d/mirrorlist gets refilled.

Judging from journal entries, this seems to coincide with the system waking up from sleep/suspend:

Jul 31 11:02:00 host kernel: PM: suspend exit
...
Jul 31 11:02:00 host systemd[1]: Starting Generate mirrorlist...

I’d assume that Starting Generate mirrorlist... fails due to the network interface still being down this early after waking up the system. However, the service behind this (pamac-mirrorlist.service) should account for this (by depending on network-online.target). So… some sort of race condition perhaps?

Anyone else experiencing this?

As the maintainer of pacman-mirrors and primary architect, I can only think of one reason this would happen.

You are using a custom mirror pool and the mirror has been removed from the official mirror pool.

pamac installs a pamac-mirrorlist.timer and it is possible this timer will generate an empty mirrorlist if the custom pool has no valid mirrors.

This is not an error, but intended behavior when using a custom mirror pool.

You can see the output from service started by the timer

journalctl -u pamac-mirrorlist.service
2 Likes

Not meaning to hijack this thread, I had the same experience, the journal shows a.o:

aug 07 20:44:12 manja systemd[1]: Starting Generate mirrorlist...
aug 07 20:44:12 manja pacman-mirrors[67751]: ::INFO Internet connection appears to be down
aug 07 20:44:12 manja pacman-mirrors[67751]: ::INFO Mirror ranking is not available
aug 07 20:44:12 manja pacman-mirrors[67751]: ::INFO Mirror list is generated using random method
aug 07 20:44:12 manja pacman-mirrors[67751]: ::INFO Writing mirror list

I changed the trigger to update the mirror list to 1 minute after boot:

/usr/lib/systemd/system ❯ sudo nano pamac-mirrorlist.timer
[Unit]
Description=Generate mirrorlist on boot

[Timer]
OnBootSec=1min

[Install]
WantedBy=timers.target

/usr/lib/systemd/system/pamac-mirrorlist.timer

looks like this on my system:

cat /usr/lib/systemd/system/pamac-mirrorlist.timer

[Unit]
Description=Generate mirrorlist weekly

[Timer]
OnCalendar=Thu *-*-* 7:00:00
RandomizedDelaySec=15h
Persistent=true

[Install]
WantedBy=timers.target

There is no: OnBootSec in the file on my system :man_shrugging:

1 Like

@hans

Please see the

man pacman-mirrors

Generating the mirror list requires the mirrors.json or the custom-mirrors.json

Both files live in /var/lib/pacman-mirrors (yes - I forgot to update the man page, sorry about that).

The file custom-mirrors.json takes precedence over mirrors.json. The content is never changed by a normal run - it is only ever changed

  • if you change country in Pamac settings
  • if you run `pacman-mirrors´ to change your mirror pool

To remove the custom mirror pool (custom-mirrors.json) - follow the instructions in the mirror list file

1 Like

Thanks for all the information on the topic.

I have not specified my own mirrors anywhere if that is what you are referring to, but I did set a specific country for mirror selection - is that also considered a custom mirror pool?

I checked the journal for the service in question:

Jul 31 11:02:00 host systemd[1]: Starting Generate mirrorlist...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO Downloading mirrors from Manjaro
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO => Mirror pool: https://repo.manjaro.org/mirrors.json
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO => Mirror status: https://repo.manjaro.org/status.json
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='repo.manjaro.org', port=>
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO Using custom mirror file
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO Querying mirrors - This may take some time
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]:   ..... Country        : https://...
Jul 31 11:02:05 host pacman-mirrors[259440]: ::ERROR Connection: HTTPSConnectionPool(host='...', port=...>
Jul 31 11:02:05 host pacman-mirrors[259440]:
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO Writing mirror list
Jul 31 11:02:05 host pacman-mirrors[259440]: ::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
Jul 31 11:02:05 host pacman-mirrors[259553]:  MIRRORLIST ERROR
Jul 31 11:02:05 host systemd[1]: pamac-mirrorlist.service: Deactivated successfully.
Jul 31 11:02:05 host systemd[1]: Finished Generate mirrorlist.

So it would seem the service actively kicks out all of the mirrors. The question is why, though. My guess would still be an inactive network connection right after waking up from sleep.

When a mirror pool is not the default pool it is a custom pool.

Redacting the output makes it impossible to provide a meaningful comment.

The only thing I can see is a network errors.

And yes - those errors will exclude the mirror - and in the end - it results in an empty mirror list.

The result would have been the same if you had used the default mirror pool.

network-online does not imply an internet connection - I don’t recall there is any function in systemd which provide such condition

1 Like

The mirrors that get kicked out are exactly the same that get re-added after refreshing the mirrors. In other words, those are not mirrors that somehow became unavailable or inactive.

From the journal I can tell this happened three times thus far, always with the same messages, starting July 24th.

They are in the custom-mirrors.json thus it make sense.

1 Like

Should I reset those as per the instructions you gave @hans?

@fonic
Wouldn’t running

sudo pacman-mirrors -c all

once
be sufficient?

the file is gone after I just did that

And it is back / recreated with fresh values when I run:
sudo pacman-mirrors -c germany

1 Like

Not necessary… when pacman-mirrors creates the mirror list, it uses the flag set by the repo server - a flag that indicates if the mirror is up-to-date - and since the mirrors are included in the mirror list - they are up-to-date.

This can be verified by checking the repo status at repo.manjaro.org

I don’t know why pamac provides the timer - my guess is the intention is to ensure the mirror list is always up-to-date.

I am opposed to having such timer - but alas - it is how it is - and I am not into the update-nagging thing, my personal preference is to disable the timer, use only one mirror and update when I have time to deal with the potential fallout.

3 Likes

I think the question is why that timer runs right after resuming the system - which is bound to fail, as you pointed out, as there is no way for the timer to know whether the network connection has already been reestablished or not.

I’ll dig into the logs a bit more to check if I can pinpoint what might be causing that.

That one is easy - the timer expired while suspended - thus it will run at the first given opportunity.

And as your system suspended when network-online was true the service will run - but there is no internet connection - which makes pacman-mirrors fail, printing the network error messages.

But the possibility that pacman-mirrors can write an empty mirror list when it happens - that I don’t like - and I will look at the options to prevent it from happening.

3 Likes

Indeed. I checked the journal and all instances occurred on Thursdays, which matches the timer’s description:

[Unit]
Description=Generate mirrorlist weekly

[Timer]
OnCalendar=Thu *-*-* 7:00:00
RandomizedDelaySec=15h
Persistent=true

[Install]
WantedBy=timers.target

I’m no expert on systemd timers, is there a setting that would add a delay after resuming?

If you want to change the service invocation you can do so in the service definition by adding a sleep statement as ExecStartPre with the number of seconds to wait before ExecStart

sudo systemctl edit pamac-mirrorlist.service

Then add

ExecStartPre=sleep 30
2 Likes

Thanks, so that might be an option - or disabling the timer like you suggested. I mean, how often do mirrors actually change? Not that often, I’d presume.

That is what I do… you just need to find your own way :slight_smile:

1 Like

That I usually do, but I’m always looking for a way to stay as vanilla as possible when issues arise. Customizations and hacks have to be maintained over time, that’s why I try to avoid them whenever possible.

Disabling a timer is still vanilla :slight_smile:

Another option is to use the Static option in pacman-mirrors-conf.

## Static configuration
## Add a comma separated list of urls to use
## e.g. Static = https://mirror1.tld/manjaro/,https://mirror2.tld/manjaro/
Static = https://manjaro.kurdy.org/

Which is useful in enterprise environments and offline environments.

3 Likes