Pacman-mirrors 4.16.x

Pre release testing

@papajoke reminded me of the code I made for a --continent argument for pacman-mirrors.

Patrick also suggested a status function to show the status of the mirrors in your mirror list. He also provided a sample implementation - and personally I think the idea is awesome.

These two functions is available in a dev package.

I have rewritten a portion of the code and I would like your feedback if I - by accident - have broken other parts.

My own tests are green - but what about yours?

Available in unstable branch v4.16.1-2

sudo pacman -Syu pacman-mirrors

pacman-mirrors --status

and

sudo pacman-mirrors --continent
3 Likes
kde% pacman-mirrors --status
Local mirror status for unstable branch
Mirror # 1  --  00:17   Germany                    https://mirror.netzspielplatz.de/manjaro/packages/
Mirror # 2  OK  00:07   Germany                    https://mirror.alpix.eu/manjaro/
Mirror # 3  OK  00:07   Germany                    https://manjaro.moson.eu/
kde% sudo pacman-mirrors --continent
::INFO Downloading mirrors from repo.manjaro.org
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.232 Germany        : https://mirror.alpix.eu/manjaro/
  0.310 Germany        : https://manjaro.moson.eu/
::INFO Writing mirror list
::Germany         : https://mirror.alpix.eu/manjaro/unstable/$repo/$arch
::Germany         : https://manjaro.moson.eu/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'

It doesn't like the http version of my mirror :disappointed_relieved:

[moson@vbox-mo ~]$ pacman-mirrors --status
Local mirror status for unstable branch
 Mirror # 1 http://manjaro.moson.eu/ do not exist
kde% pacman-mirrors --status               
Local mirror status for unstable branch
Mirror # 1  OK  00:06   Germany          https://mirror.alpix.eu/manjaro/
Mirror # 2  OK  00:06   Germany          https://manjaro.moson.eu/
kde% sudo pacman-mirrors --continent            
[sudo] password for sgs: 
::INFO Downloading mirrors from repo.manjaro.org
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.208 Germany        : https://mirror.alpix.eu/manjaro/
  0.241 Germany        : https://mirror.netzspielplatz.de/manjaro/packages/
  0.281 Germany        : https://manjaro.moson.eu/
::INFO Writing mirror list
::Germany         : https://mirror.alpix.eu/manjaro/unstable/$repo/$arch
::Germany         : https://mirror.netzspielplatz.de/manjaro/packages/unstable/$repo/$arch
::Germany         : https://manjaro.moson.eu/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'
kde% pacman-mirrors -v         
Version 4.16

pacman-mirrors-dev 4.16dev1-4


Irgendwo stand doch es ist "nur" für https, jetzt finde ich es wieder mal nicht.
Somewhere it said "only" for https, now I can't find it again.

1 Like

I know why - when mirrors has more than one protocol only https is checked.

I will have to rewrite that piece :slight_smile: thank you for spotting it.

2 Likes

With the new package I pushed to unstable the mirror should be detected correct.

2 Likes

Just tried with 4.16 and 4.16dev1-5. Still have the http issue with both of them.
(although the typo got corrected now, do -> does ) :wink:

mirrors = [m for m in mirrors if m['url'] in mirrors_pacman]

here we not remove protocol so mirrors_list is not complete


not tested, we can create all url from protocols, and after make a simple filter

datas = json.loads(req)
mirrors = []
for m in datas:
    for p in m['protocols']:
        new = m.copy()
        new['url'] = p + "://" +strip_protocol(new['url'])
        mirrors.append(new) # add one mirror by protocol

# now filter is good, we have same urls in pacman and json
mirrors = [m for m in mirrors if m['url'] in mirrors_pacman]
# we not use strip_protocol() after in mirror = [ m for m in mirrors if m['url'] == url][0]
~ >>> pacman-mirrors --status                                                                                                       
Local mirror status for unstable branch
Mirror # 1  OK  00:06   Germany          https://mirror.alpix.eu/manjaro/
Mirror # 2  OK  00:06   Germany          https://manjaro.moson.eu/
~ >>> sudo pacman-mirrors --continent                                                                                               
[sudo] Passwort für sgs: 
::INFO Downloading mirrors from repo.manjaro.org
::INFO Using custom mirror file
::INFO Querying mirrors - This may take some time
  0.224 Germany        : https://mirror.alpix.eu/manjaro/
  0.278 Germany        : https://manjaro.moson.eu/
  0.253 Germany        : http://manjaro.moson.eu/
::INFO Writing mirror list
::Germany         : https://mirror.alpix.eu/manjaro/unstable/$repo/$arch
::Germany         : https://manjaro.moson.eu/unstable/$repo/$arch
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
::INFO To reset custom mirrorlist 'sudo pacman-mirrors -id'
::INFO To remove custom config run  'sudo pacman-mirrors -c all'

When I run the code - the strip_protocol returns the url like this

python code: return url.split("//")[1]

returns: mirrors.gigenet.com/manjaro/

python code: mirror = [m for m in mirrors if strip_protocol(m['url']) == strip_protocol(url)][0]

So I don't understand why this will fail - as the comparison is done without the protocol part (https:// or http:// or ftp://)

you use strip_protocol line 62 but not line 56 so mirrors is false if user use other protocol than url

with my soluce we have one mirror by protocol (before filter) OR ... not make this filter line 56 ???
with my soluce (duplicate entry by protocol) we test valid protocols - with no filter line 56 and strip_protocol() we not test if protocol is good ...

You are right - I have changed it - added your suggested loop

I will push a package shortly.

a question: how does the program manage protocols for same mirror : it tests all and adds only the fastest ?

It adds from protocol priority. If undefined in descending order starting with https.

It is not ideal to test them all but as https is proritized and sometimes certificates expire - such connection errors needs to be filtered from the final mirrorlist.

I know I read something :wink:
cat /etc/pacman-mirrors.conf
#If a mirror has more than one protocol defined only the first is written to the mirrorlist

1 Like

tips for format country length - here is const 16, calculate max len

len_country = max(len(m['country']) for m in mirrors)+1
print(f"Mirror #{i + 1:2}", color, f"{text}", C_NONE,
#   f"{mirror['last_sync']:7} {mirror['country']:16} {mirror['url']}")
    f"{mirror['last_sync']:7} {mirror['country']:{len_country}} {mirror['url']}")

:+1:

Implemented that too :slight_smile:

Just to see how it works

$ sudo pacman-mirrors -c all -m random --no-status
❯ pacman-mirrors
Pacman-mirrors version 4.16.1
Local mirror status for unstable branch
Mirror # 1  --  08:27   Hong_Kong       http://ftp.cuhk.edu.hk/pub/Linux/manjaro/
Mirror # 2  --  05:31   Netherlands     http://ftp.snt.utwente.nl/pub/linux/manjaro/
Mirror # 3  --  05:57   Norway          http://mirror.terrahost.no/linux/manjaro/
Mirror # 4  --  04:27   Indonesia       http://kartolo.sby.datautama.net.id/manjaro/
Mirror # 5  OK  00:32   United_Kingdom  http://mirror.catn.com/pub/manjaro/
Mirror # 6  OK  00:29   Brazil          https://www.caco.ic.unicamp.br/manjaro/
Mirror # 7  OK  00:05   Belgium         https://mirror.futureweb.be/manjaro/
Mirror # 8  OK  00:07   Germany         http://mirror.ragenetwork.de/manjaro/
Mirror # 9  --  24:05   Bulgaria        https://mirrors.netix.net/manjaro/
Mirror #10  --  03:57   Iceland         https://mirrors.opensource.is/manjaro/
Mirror #11  OK  00:20   Denmark         https://mirrors.dotsrc.org/manjaro/
Mirror #12  OK  00:06   Denmark         https://www.uex.dk/public/manjaro/
Mirror #13  OK  00:07   Germany         https://mirror.netzspielplatz.de/manjaro/packages/
Mirror #14  --  05:32   Turkey          http://ftp.linux.org.tr/manjaro/
Mirror #15  --  06:29   Australia       http://manjaro.melbourneitmirror.net/
Mirror #16  --  05:31   Italy           https://ct.mirror.garr.it/mirrors/manjaro/
Mirror #17  --  02:56   Costa_Rica      https://mirrors.ucr.ac.cr/manjaro/
Mirror #18  --  10:28   Thailand        https://mirror.kku.ac.th/manjaro/
Mirror #19  --  02:19   Belarus         http://mirror.datacenter.by/pub/mirrors/manjaro/
Mirror #20  --  23:28   United_States   http://distro.ibiblio.org/manjaro/
Mirror #21  --  08:27   Philippines     http://mirror.rise.ph/manjaro/
Mirror #22  --  45:35   Belgium         http://ftp.belnet.be/mirrors/manjaro/
Mirror #23  --  05:31   Poland          http://ftp.vectranet.pl/manjaro/
Mirror #24  OK  00:27   Japan           http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/
Mirror #25  --  05:29   Austria         http://mirror.inode.at/manjaro/
Mirror #26  --  23:47   Greece          https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
Mirror #27  --  02:21   Indonesia       https://mirror.deace.id/manjaro/
Mirror #28  OK  00:21   Poland          https://mirror.tuchola-dc.pl/manjaro/
Mirror #29  --  02:31   South_Africa    http://mirror.is.co.za/mirrors/manjaro.org/
Mirror #30  --  01:31   Germany         https://mirror.23media.com/manjaro/
Mirror #31  --  22:29   Bangladesh      http://mirror.xeonbd.com/manjaro/
Mirror #32  --  02:21   Russia          https://mirror.yandex.ru/mirrors/manjaro/
Mirror #33  --  05:01   France          http://ftp.free.org/mirrors/repo.manjaro.org/repos/
Mirror #34  OK  00:22   Ukraine         http://mirrors.colocall.net/manjaro/
Mirror #35  --  05:31   Italy           https://manjaro.mirror.garr.it/mirrors/manjaro/
Mirror #36  --  02:20   China           https://mirrors.sjtug.sjtu.edu.cn/manjaro/
Mirror #37  --  01:58   Sweden          https://mirror.zetup.net/manjaro/
Mirror #38  --  04:01   Netherlands     https://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
Mirror #39  OK  00:08   United_Kingdom  http://manjaro.mirrors.uk2.net/
Mirror #40  --  06:31   Netherlands     https://mirror.neostrada.nl/manjaro/
Mirror #41  --  05:31   Germany         https://ftp.halifax.rwth-aachen.de/manjaro/
Mirror #42  --  00:52   United_Kingdom  http://repo.manjaro.org.uk/
Mirror #43  --  27:08   United_States   https://mirrors.ocf.berkeley.edu/manjaro/
Mirror #44  OK  00:07   Germany         https://mirror.alpix.eu/manjaro/
Mirror #45  --  05:02   United_States   https://mirror.clarkson.edu/manjaro/
Mirror #46  --  03:58   United_States   https://mirrors.gigenet.com/manjaro/
Mirror #47  --  02:05   Australia       http://manjaro.mirror.serversaustralia.com.au/
Mirror #48  --  21:08   South_Africa    http://manjaro.mirror.ac.za/
Mirror #49  --  01:55   Brazil          https://manjaro.c3sl.ufpr.br/
Mirror #50  --  05:01   Germany         https://mirror.netcologne.de/manjaro/
Mirror #51  --  11:31   Japan           http://ftp.riken.jp/Linux/manjaro/
Mirror #52  --  00:37   France          http://kibo.remi.lu/
Mirror #53  --  03:57   Netherlands     https://mirror.koddos.net/manjaro/
Mirror #54  --  11:31   Iran            https://repo.iut.ac.ir/repo/manjaro/
Mirror #55  --  04:17   Germany         http://ftp.tu-chemnitz.de/pub/linux/manjaro/
Mirror #56  --  00:37   New_Zealand     http://manjaro.mirrors.theom.nz/
Mirror #57  OK  00:07   Hungary         https://quantum-mirror.hu/mirrors/pub/manjaro/
Mirror #58  --  01:58   Sweden          https://ftp.lysator.liu.se/pub/manjaro/
Mirror #59  --  06:07   Germany         http://ftp.rz.tu-bs.de/pub/mirror/manjaro.org/repos/
Mirror #60  --  03:18   Spain           https://ftp.caliu.cat/pub/distribucions/manjaro/
Mirror #61  OK  00:17   France          https://manjaro.ynh.ovh/
Mirror #62  OK  00:07   Germany         https://manjaro.moson.eu/
Mirror #63  --  05:30   Czech           https://mirror.dkm.cz/manjaro/
Mirror #64  --  14:26   China           https://mirrors.tuna.tsinghua.edu.cn/manjaro/
Mirror #65  --  28:07   Ecuador         https://mirror.cedia.org.ec/manjaro/
Mirror #66  --  01:39   Chile           https://mirror.ufro.cl/manjaro/
Mirror #67  --  05:31   France          https://mirror.oldsql.cc/manjaro/
Mirror #68  OK  02:19   China           https://mirrors.ustc.edu.cn/manjaro/
Mirror #69  --  05:31   Italy           https://ba.mirror.garr.it/mirrors/manjaro/
Mirror #70  --  05:31   Georgia         https://manjaro.grena.ge/
Mirror #71  --  05:31   Kenya           https://manjaro.mirror.liquidtelecom.com/
Mirror #72  --  00:58   United_States   https://mirror.math.princeton.edu/pub/manjaro/
Mirror #73  OK  00:28   United_States   https://repo.ialab.dsu.edu/manjaro/
Mirror #74  OK  00:05   Austria         http://mirror.easyname.at/manjaro/
Mirror #75  --  05:31   Portugal        http://ftp.dei.uc.pt/pub/linux/manjaro/
Mirror #76  --  00:47   Ecuador         http://mirror.espoch.edu.ec/manjaro/
Mirror #77  --  00:35   Bulgaria        https://manjaro.ipacct.com/manjaro/
Mirror #78  --  03:18   Taiwan          http://free.nchc.org.tw/manjaro/
Mirror #79  --  05:29   Brazil          http://linorg.usp.br/manjaro/
Mirror #80  OK  00:08   South_Korea     https://mirror.d-tl.com/manjaro/
Mirror #81  --  05:31   India           https://mirrors.piconets.webwerks.in/manjaro-mirror/
Mirror #82  --  04:27   Singapore       https://download.nus.edu.sg/mirror/manjaro/
Mirror #83  --  28:07   Iran            https://repo.sadjad.ac.ir/manjaro/
Mirror #84  OK  00:07   Hungary         https://nova.quantum-mirror.hu/mirrors/pub/manjaro/
Mirror #85  --  01:55   Bulgaria        http://manjaro.telecoms.bg/
Mirror #86  --  05:02   United_Kingdom  https://www.mirrorservice.org/sites/repo.manjaro.org/repos/
Mirror #87  --  02:21   Russia          http://mirror.truenetwork.ru/manjaro/
Mirror #88  --  02:59   Chile           http://manjaro.dcc.uchile.cl/

usefull "random" :laughing:


# 1 not fan if more then 9

f"#{i + 1:02d}

we have #01 ... #99

I have noted it - pure cosmetic :slight_smile:

Yes - very useful - and the --no-status argument - returns all mirrors from the pool.

print(f"Mirror #{str(i + 1):2}"
Mirror #1   --  04:57   Taiwan          http://free.nchc.org.tw/manjaro/
Mirror #2   --  01:45   Belgium         https://mirror.futureweb.be/manjaro/
...
Mirror #10  --  03:10   Germany         https://mirror.23media.com/manjaro/
Mirror #11  OK  00:06   Germany         https://mirror.alpix.eu/manjaro/

Forum kindly sponsored by