How to create a Custom mirrorlist - the NEW .json way


#1

Created on 21-March-2017

I’m using the Testing repo & did an upgrade this morning which overwrote my /etc/pacman-mirrors.conf So after the update without editing the new pacman-mirrors.conf I was set to use the Stable repo & the OnlyCountry= line was commented out “#” instead of saying Custom . So know that /etc/pacman-mirrors.conf needs editing after this update.

Anyway, for those of us that use allservers & for all of the other speedy reasons that we may like to use a much smaller Custom mirrorlist - following is what you now have to do to create a ne Custom mirrorlist (it automates the process that I’d previously had written here that I’d worked out for myself).

After you have updated have a look with your text editor (root access) at your “new” unedited /etc/pacman-mirrors.conf & change the repo line from stable to whatever you use if you need to. Here is the line:

## Branch Pacman should use (stable, testing, unstable) Branch = stable

Then look further down in the same file & find the following line & change it so that it looks like the one below:

[code]## Generation method

1) rank - rank mirrors depending on their access time

2) random - randomly generate the output mirrorlist

Method = rank[/code]

Then look further down in the same file & find the following line & change it so that it looks like the one below:

[code]## Specify to use only mirrors from specific a country.

Can add multiple countries separated by a comma (ex: Germany,France)

Empty means all

OnlyCountry = Custom[/code]

Then save the file.

Now go to the terminal & enter the following at the prompt & hit enter:

This will cause the process to begin in the terminal & then! Surprise, surprise, a small GUI requester will pop up that will list all of the ranked mirrors so you can select which ones will be used in your Custom mirrorlist. A mirrorlist that is located here:

NOTE:
I just ran the command that bought up the GUI, & due to my already having a custom-mirrors.json file, which has been previously ranked by the pacman-mirrors -g command, & so has therefore become my /etc/pacman.d/mirrorlist I only have my Custom list shown to me in the little GUI.

So I think that I’ve uncovered a bug in the GUI program. The GUI program needs to use the /var/lib/pacman-mirrors/mirrors.json file.


Suggestion: how to create/edit mirror list
[Testing Update] 2017-03-22 - Elogind, Plasma, Deepin, Manjaro-Architect
[Stable Update] 2017-04-02 - Mesa-Stack, Kernels, Plasma, Firefox
#2

@handy, I too use your allservers script, and, by trial and error found that if I edit the new pacman-mirrors.conf like the old version, (or, use the old version) for my custom mirror list, (w/ OnlyCountry = Custom)…then run “sudo pacman-mirrors -g --interactive” and choose my mirror/s of choice, the file /var/lib/pacman-mirrors/custom-mirrors.json was created for me. I didn’t have to manually edit/rename. I have been racking my old brain over this for some time, but I’m “quite” sure that is all I did.
My option of choice in allservers is #3, and it seems to be working fine using my custom mirror list…so far at least.

Best regards.


#3

please ask here about it:

@fhdk is friendly and open to suggestions. feel free to recommend an easier way to implement custom mirror lists and talk to him about it.


#4

@conky57 Thanks a heap for that info. I was just starting to rewrite the Wiki page & load in the much tougher way that I’ve written above.

I’ll check it out & act accordingly. :smiley:


#5

@excalibur1234 Thanks. :slight_smile:

It looks like @conky57 has found the best way to do it. :slight_smile:

I’ll check it out & if that works then I’ll modify the appropriate Wiki page & my page above.


#6

And thank you a heap for the allservers script…I have been using it since you made it available.:relaxed:

Best regards.


#7

@conky57 @excalibur1234 YES! you nailed it conky, I think I’ll probably throw another menu entry into allservers so when we feel like updating our Custom mirrors we can just pull it out of the allservers menu & do it.

I’ll go & delete most of the OP here… :wink:

[edit:] A bug needs to be fixed in the GUI for creating Custom lists. I think it needs to be fixed before I can add a new menu item to the allservers script.


#8

Sorry guys, the edits up there in the OP got very messy there for a few minutes. (1.5 hours sleep last night…)

The OP should be pretty well right now (for the time being as there are still bugs).


#9

Does it filter mirrors for only those that are up to date, vs those that are merely fast?


#10

@jsamyth By the look of the /var/lib/pacman-mirrors/status.json there is certainly a chance that it does.


#11

Greetings .
@handy @conky57 @jsamyth

This is going to be a long post - ye be warned :slight_smile:

My background in this project

I am retired but from my work life have a decent experience in coding and application development and when @huluti asked for help to clean up the code, I volunteered, told him - I know nothing of python, but I am up for a challenge. I have since found python to be an extremely powerful and yet easy to learn language.

The concepts as I have learned them

When I joined this project I had a bit of a difficulty with some of the concepts.

Pacman-Mirrors basically consisted of 3 elements.
Feel free to correct me if I am wrong about some of the historical data

  • A number of country mirror files
  • /etc/pacman.d/mirrors/Country
  • A mirror list
  • /etc/pacman.d/mirrorlist
  • A custom mirror file
  • before my time a hybrid between a mirrorlist and a mirrorfile
  • began existence in /etc/pacman.d/Custom
  • at some time pacman-mirrors began moving the file to /var/lib/pacman-mirrors

About the time I joined a GUI for selecting mirrors had been devised and its contents was written to the custom mirror file.

As I was on Arch Linux I was used to only a mirrorlist and the concept of both a mirror list and a mirror file was confusing at first. I learned that the difference between a mirror file and a mirror list is the branch.

Server = https://mirror.uta.edu.ec/manjaro/unstable/$repo/$arch

Server = https://mirror.uta.edu.ec/manjaro/$branch/$repo/$arch

And it is at this point some of handy’s problems began, because he learned - exactly as I did, that the Custom file is no longer a mirror list but a mirror file and when writing the mirror file you have a placeholder $branch instead of an explicitly defined branch.

What has lead pacman-mirrors to json?

In the process of rewriting and cleaning up the internals for duplicate code and other shortcuts I came up with the idea of storing the calculations made by pacman-mirrors about response-time and last-repo-sync in - you get it - json. The idea was rejected by the team but not more than - this is a good idea - we take it server side. And not long after @philm and @huluti began creating what we have know at repo.manjaro.org and the data files used.

Since then I have occupied myself by rewriting pacman-mirrors and it has been totally rewritten. There is not that corner I have not been in optimizing and restructuring and the result is v4 and of course small bugs and annoyances which has been ironed out in a 4.0.1-dev.

Is it a bug?

I have been reading through this thread and it seems a little confusing exists.

When you use the –interactive argument, you are also telling pacman-mirrors you want a custom mirror file now now as custom-mirrors.json. Pacman-Mirrors then modify your pacman-mirrors.conf so every time this custom mirror file is loaded and used.

Since the device of the Custom mirrorfile it has been a manual task to create that file. And it was a manual task to get pacman-mirrors to recognize it. The how-to you have produced does it backwards since you modify the conf file to load the custom-mirrors.json. In that case you won’t get all the mirrors from the default mirror file.

You will have to reverse your view of the custom mirror file. Think this way - You don’t have a custom mirror file - how do I get one. The GUI and TUI has been created to take the hazzle of creating your custom-mirrors.json. Actually - Hugo has made some sorting functionality (only in GUI) without compromising the rank/random intent.

pacman-mirrors -i

Pick the mirrors and exit and it saves a custom-mirrors.json and your pacman mirrorlist.

If you want to rank your mirrors you run

pacman-mirrors -g

If you want to change your selection without doing the now long ranking you can do

pacman-mirrors -i -m random

From v 4.0.1 - you can click on the headers to sort on url, country, last sync - select your mirrors and exit. They will of course be random but

pacman-mirrors -g

will rank your custom mirror file and write your mirror list with mirrors ranked by response time.

Mirror quality

You are right about the status.json - Pacman-Mirrors does have an idea of which mirrors are updated and this knowledge is brought to use with

pacman-mirrors -f 10

This gives you 10 random - up2date - mirrors ranked by response time. The reasoning behind using random mirrors is that mirrors which update more frequently than others should not be overwhelmed just because they are updated more frequently - like eg netzspielplatz.de or stdout.org does.


#12

Thank you for the detailed explanation and your efforts to improve pacman-mirrors. It makes a lot more sense to me now. I have trial and errored after being used to the “old” way of creating a custom mirror list and not comprehending the new features.
Thank you once again.:relaxed:

Best regards.


#13

@fhdk Thanks for your reply & for all of your hard work/time.

Please help me to understand (grant me patience as I’m very tired at the moment?):-

My machine now has a /var/lib/pacman-mirrors/custom-mirrors.json file that I created manually as I couldn’t think of another way out of the situation.

So would you say that due to my having done that, it causes the –interactive option to only give me the ~half a dozen or so mirrors from that file (or the /etc/pacman.d/mirrorlist) to choose from?

To me that doesn’t make sense, as I would have thought that the --interactive option would read the mirrors.json file.

Anytime a person decides that they would like to update their Custom mirrorlist, & they already have one that they’ve been using (perhaps for some months), shouldn’t they be able to just run the --interactive option & be given a full list of the available Manjaro mirrors laid out for them in the GUI so they can choose a new much shorter list that will overwrite their /var/lib/pacman-mirrors/custom-mirrors.json & go on to become their /etc/pacman.d/mirrorlist ?


#14

@conky @handy
Thanks for you appreciation :slight_smile: It makes it worth the trouble

The setting in pacman-mirrors.conf

OnlyCountry = Custom

is instructing Pacman-Mirrors to use the custom-mirrors.json.

To have Pacman-Mirrors to use another file you must instruct Pacman-Mirrors to do so.

Currently there is two possible paths - a third is underway

sudo pacman -ic all  # which ranks
sudo pacman -ic all -m random  # yes - random

v4.0.1 has sorting capabililty without changing intention
v4.0.1 introduces - if accepted by the team - a --default switch
This switch only works if --interactive is present.
Its function is to load default mirrors and skip directly to GUI with an alphabetized list of mirrors

The ranking/randomizing is executed on the custom mirror file after exit from GUI


Pacman-Mirrors 4.1.0-dev
#15

@fhdk I understand better now, thanks.

One query: --default switch is to load default mirrors, what do you mean by “default mirrors”?


#16

in mean default in the sense not custom but the whole #!


#17

@fhdk Cool, I was hoping that’s what you meant. :slight_smile:

I like the --default switch, I hope the team does too. :wink:


#18

If you download

https://raw.githubusercontent.com/manjaro/packages-core/master/pacman-mirrors/PKGBUILD

to a separate folder

edit and change _branch=4.0.x-stable to _branch=master and build it with

makepkg -is

You can have a pretest - and maybe give me some feedback?


#19

FYI

It slipped through - current v 4.0.2 is rolling - Have look there for a deeper explanation


#20

@fhdk I’m just out of bed. I see that you’ve been busy whilst I’ve been trying to sleep. :slight_smile: Good on you. I’ll check out your new v 4.0.2 & see how it goes. Your deeper explanation was great. Thanks once again for all of your time & your kind attitude. :wink: