Setting up a Repo Mirror

Support
1

I have an Ubuntu File Server, and I want to set up a Local Repo Server for the multiple Manjaro OSs I have on my LAN.

The info I have read on the Forum appears to be related to creating a Local Mirror on a Manjaro OS, using for example Pacserve, and similar Manjaro/Arch oriented tool.

What I want to know, is setting up a repo Mirror for Manjaro any more complicated than setting up a Web server and serving HTTP and or FTP, and syncing back to a public Mirror. Or are there specifics I need to know?

2

Depends on what you want to serve: a full mirror of all repos or pacserve which seems like a cache in your LAN.

https://wiki.archlinux.org/title/Pacserve

3

I want to set up the local repo on an Ubuntu Server.

4

You can use any OS as a server. You can take inspiration from this post: [root tip] [How To] Create Manjaro mirror server

5

That is all there is to it.

Ensure the custom mirror is used

When you setup a mirror you want to ensure your local computers use that mirror.

You need to make a special configuration for pacman-mirrors to avoid the pamac mirrorlist timer rewriting your mirror list.

Configure pacman-mirrors to use a mirror pool with only one mirror.

sudo pacman-mirrors --country dk

Then edit the resulting pool file /var/lib/pacman-mirrors/custom-mirrors.json and remove unnecessary data.

Initially the file may look like this

[
  {
    "branches": [
      1,
      1,
      1
    ],
    "country": "Denmark",
    "last_sync": "00:59",
    "protocols": [
      "https"
    ],
    "resp_time": "00.00",
    "url": "https://mirrors.dotsrc.org/manjaro/"
  }
]

Remove status configs branches, last_sync and resp_time.

Modify url to point to your server.

JSON data is picky on format so ensure all brackets and braces are in place.

[
  {
    "country": "CUSTOM",
    "protocols": [ "https" ],
    "url": "https://your-mirror.tld/manjaro"
  }
]

Save the file and copy to a convenient location e.g. your home.

cp /var/lib/pacman-mirrors/custom-mirrors.json ~

From there you can distribute the configuration all your systems - copy the file to the same location - replacing the existing file if necessary.

cp custom-mirrors.json /var/lib/pacman-mirrors

Ensure your custom pool is never replaced

Ideally this should be done with a custom pacman-mirrors package.

The package must rebuild on python version change - but the following will work as well.

If you want to ensure the custom pool is never reset by a user running pacman-mirrors -c all you can apply some system wide changes.

REMEMBER: Modifying system files will be reverted when the package owning the files is rebuilt.

Copy the file custom-mirrors.json as mirrors.json to /usr/share/pacman.mirrors - replacing the upstream mirror pool.

sudo cp ~/custom-mirrors.json /usr/share/pacman-mirrors/mirrors.json

Locate the file /usr/lib/python3.10/site-packages/pacman_mirrors/config/configuration.py

Edit the configuration and change the URL_MIRROR_JSON and URL_STATUS_JSON to be an empty string

#URL_MIRROR_JSON = "https://repo.manjaro.org/mirrors.json"
#URL_STATUS_JSON = "https://repo.manjaro.org/status.json"
URL_MIRROR_JSON = ""
URL_STATUS_JSON = ""

Save the file.

no shiny-mirrors

Of course none of the above will work as expected if your systems use shiny-mirrors - only pacman-mirrors can do this.

6

Shiny-mirrors???

7

Please focus on one thing.

shiny-mirrors is an alternative to pacman-mirrors

8

'K

I had started to read that article, but it seemed to be about Manjaro Apps, so I discarded it… Thanks I’ll read it properly.

@ linux-aarhus thanks

I’ll let you guys know as soon as I get this set up, could take a while though.

9

… not really a repo mirror here
but may be easier and more effective

With my Manjaro virtual machines I do that kind of thing
which essentially is:
download only once and use for all

by having a separate disk, which I mount to /var/cache/pacman/pkg
on each machine

All the machines are then using the same package cache.

This same thing could be achieved using a server and (NFS or sshfs) mounting a directory on it to
/var/cache/pacman/pkg on each machine.

This way, not the whole repo needs to be cloned - but just the packages that are actually needed.

Just another idea …

10

But…but…

11

as @ mithrial says, that’s probably pacserve. I’ll go with setting up a Mirror on my File server. I have plenty of space, and I already have Apache running on it.

It will take a while 'cos of finding the time to concentrate on actually doing the job. It’s coming into Fire Season over here, so were pretty busy making sure only our Bush burns, and not the house.

12

My scenario was
VM’s, I run them one at a time, not simultaneously.
My only requirement is not to download all the updates multiple times for each of them but re-use them once they are present.
No need for the full repo and sharing the cache via pacserve would require more than one to be running at a time which is never the case.

Just a primitive re-use of the pacman cache.

13

You describe a scenario that is unique to you, and probably be too complicated for OP.

If your server is already running apache, then all you have to do is mirror one of the mirrors to any subdirectory of apache, and point your clients to this server.

14

… maybe
It’s a network mount - plenty of documentation on that.

It was just another way to do it.

And it avoids mirroring the whole mirror when only a few packages are actually needed.
That is how I understood how pacserve works.
But I’m not sure about that.
changed because only the pacman cache is shared

… would not have worked for me anyway

15

If you don’t have it already - setup a web server on your ubuntu box and create a manjaro subfolder.

Then setup a script and a systemd timer to sync from the repo.

Create a script folder

sudo mkdir /etc/systemd/scripts

Create a file /etc/systemd/scripts/rsync-manjaro.sh and make it executable

sudo touch /etc/systemd/scripts/rsync-manjaro.sh
sudo chmod +x /etc/systemd/scripts/rsync-manjaro.sh

Edit the file as root and paste below content

#!/bin/sh

# This is a simple mirroring script. To save bandwidth it first checks a
# timestamp via HTTP and only runs rsync when the timestamp differs from the
# local copy. As of 2016, a single rsync run without changes transfers roughly
# 6MiB of data which adds up to roughly 250GiB of traffic per month when rsync
# is run every minute. Performing a simple check via HTTP first can thus save a
# lot of traffic.

# EDIT to match your system
TARGET=/var/www/manjaro

# possible rsync mirrors
# https://wiki.manjaro.org/index.php/Manjaro_Mirrors
# https://forum.manjaro.org/t/root-tip-how-to-create-manjaro-mirror-server/21264#syncing-the-repo-from-another-mirror-17
#SOURCE="rsync://ftp.halifax.rwth-aachen.de/manjaro/"
SOURCE="rsync://mirror.easyname.at/manjaro/"


TMP="/var/tmp/manjaro"
LOCK="/tmp/rsync-manjaro.lock"

# NOTE: You'll probably want to change this or remove the --bwlimit setting in
# the rsync call below
BWLIMIT=100000


[ ! -d "${TARGET}" ] && mkdir -p "${TARGET}"
[ ! -d "${TMP}" ] && mkdir -p "${TMP}"

exec 9>"${LOCK}"
flock -n 9 || exit

## if we are called without a tty (cronjob) only run when there are changes
#if ! tty -s && diff -b <(curl -s "${STATE}") "${TARGET}/state" >/dev/null; then
#    exit 0
#fi

if ! stty &>/dev/null; then
    QUIET="-q"
fi

# excluded fh 2016-11-01
rsync -rtlvH --safe-links \
    --bwlimit=${BWLIMIT} \
    --delete-after --progress \
    -h ${QUIET} --timeout=300 --contimeout=120 -p \
    --delay-updates --no-motd \
    --temp-dir="${TMP}" \
    --exclude='/arm-stable' \
    --exclude='/arm-testing' \
    --exclude='/arm-unstable' \
    --exclude='/pool/overlay-arm' \
    --exclude='/pool/sync-arm' \
    ${SOURCE} \
    "${TARGET}"

rm -f ${LOCK}
EOF

Run the script as root to do the initial sync - I don’t know how much the arm-repos are worth but I think a lot - and there’s no benefit from excluding branches as branch files - except database files - are symlinks to actual files in the pool.

If you have no need for the arm branches you can get down to around 90G only for x86_64 - I have modified the sync script to exclude arm.

When you have synced the repos - create a service and a timer

sudo touch /etc/systemd/system/rsync-manjaro.service
sudo touch /etc/systemd/system/rsync-manjaro.timer

Edit the service file and paste below

[Unit]
Description=Sync manjaro repo

[Service]
Type=oneshot
ExecStart=/etc/systemd/scripts/rsync-manjaro.sh

[Install]
WantedBy=multi-user.target

Edit the timer file and paste below

[Unit]
Description=Sync Manjaro mirror on a daily base

[Timer]
Persistent=true
OnCalendar=daily

[Install]
WantedBy=timers.target

Then enable the timer

sudo systemctl enable --now rsync-manjaro.timer
16

Thanks I’ve complimented this. I’ll guess I’ll know shortly how well it works.

17

I do have one minor problem.

I am now unable to sync the AUR database.

I have a couple of AUR packages installed on two of the computers. Most importantly is Mullvad-VPN.

18

AUR has nothing to do with the package repositories.

19

@mithrial That’s correct.

and yet… on multiple computers it no longer synchronises.

To be specific, when one synchronises databases using the GUI the message returned is ‘Failed to synchronise AUR database’

20

I am sorry to butt in like this when it was already solved but to me this is just too unnecessary and complicated with no good pros only cons,
I think it doesn’t make sense to make custom pool for just one local mirror,
I would just disable the timer and set in /etc/pacman.d/mirrorlist the local IP of the mirror
like so: Server = <protocol>://<local_IP>/<branch>/$repo/$arch
And it’s done
There is no need to keep using any kind of mirror helper(pacman-mirrors or shiny-mirrors), in such case it’s useless when you are going to use only one mirror.