[HowTo] Create a Manjaro mirror server using Architect

Create a Manjaro mirror using Architect

Manjaro Architect is well suited for creating custom installations and a custom installation is needed for running a mirror of Manjaro repo.

Manjaro can benefit from more mirrors but please consider this before committing to run an official mirror.

Prerequisites

Internet connection

A static IP address and a domain name with access to setting up DNS to point to your IP. It is possible to use dynamic DNS but it is not as reliable as a static IP.

You need to be able to setup your router to forward traffic to the system running the mirror and you also need to setup your LAN interface to use static addressing.

Don't be tempted to use a wireless connection - use a wired connection directly to your router if possible.

Hardware

I won't go into details with your hardware, suffice to say it is possible to run a mirror on a Raspberry Pi using raspbian with an 128GB SSD connected using USB. And disk space is the most important remedy. You need at least 128GB preferrably 240GB to avoid running full.

diskstation> du -sh /volume1/web/public/manjaro
94.4G	/volume1/web/public/manjaro

Software

A syncronization app such as rsync (recommended) or wget and a webserver e.g. apache or nginx configured to serve directory listings.

Installation

If you have a recent Manjaro ISO available there is no need to download another as Architect has been included for quite some time. Start by booting your hardware on your Manjaro ISO.

Note
The below steps is the simplified version - however you may be a experienced user and have intimate knowledge on how you want the system - sure - you decide on how the system should be setup so please skip this section.

Before launching Architect update your mirrorlist using the terminal

sudo pacman-mirrors -f && sudo pacman -Syy

Ensure have have an internet connection and launch Architect using your terminal.

sudo manjaro-architect

Select your language and continue to the Main Menu

  • Press 1Enter on the Prepare Installation item.
  • Select keyboard using the Set Virtual Console entry and press Enter.
  • Partition Disk entry.
    • Select the Automatic Partitioning.
  • Next is Mount Partitions.
    • Mount the root partition / and format to ext4 using the default options presented.
    • Unless you are running from SSD please do create a swapfile.
    • Select Done (Do not select to mount the 512MB partition).
    • When prompted for the EFI partition, choose your mountpoint (usually /boot/efi), select the 512MB partition and format to FAT32.
    • Press 9Enter to Refresh Pacman Keys.
    • Press Enter to Choose pacman cache and select Yes.
    • Select Back and press Enter.

Back in the Main Menu press 4Enter to Install CLI System

  • Select Install Base and select linux419.
  • Select OK in the next window and wait while the base system is installed.
  • On the Network Drivers page select appropriate for your system - free drivers will usually suffice.
  • Install bootloader → select grub
  • Configure Base
    • Generate FSTAB → Device UUID (default)
    • Set your Hostname (network unique required)
    • Set your System Locale
    • Set your Timezone and Clock
    • Set Root Password
    • Create New User
    • Select Shell
    • Set User Password
    • Select Back

Back in the CLI System Menu select Install Custom Packages

  • Type in apacheTab to select the apache package.
  • Delete and type netctl Tab to select the netctl package.
  • Delete and type arp-scan Tab to select the arp-scan package.
  • Delete and type openresolv Tab to select the openresolv package.
  • Delete and type bind-tools Tab to select the bind-tools package.
  • Press Enter to install selected packages.

Chroot into your installation

Select the item Chroot into Installation (make a note on how to return to the installer - namely first exitEnter then fgEnter).

LAN configuration

As you are going to have traffic forwarded through your router you need to assign a static IP address to your network interface.

Display information on your LAN

To retrieve information on connected devices we will use arp-scan to scan your network (Sample output from my network)

~ >>> sudo arp-scan -l
[sudo] password for user: 
Interface: enp0s3, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9.5 with 256 hosts (https://github.com/royhills/arp-scan)
192.168.x.1	    fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.4	    b8:27:eb:xx:xx:xx	Raspberry Pi Foundation
192.168.x.5	    fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.10	00:11:32:xx:xx:xx	Synology Incorporated
192.168.x.30	18:e8:29:xx:xx:xx	(Unknown)
192.168.x.31	fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.32	b4:fb:e4:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.151	c8:69:cd:xx:xx:xx	Apple, Inc.
192.168.x.201	3c:d9:2b:xx:xx:xx	Hewlett Packard
192.168.x.203	14:91:82:xx:xx:xx	Belkin International Inc.
192.168.x.204	b4:fb:e4:xx:xx:xx	Ubiquiti Networks Inc.

11 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9.5: 256 hosts scanned in 1.987 seconds (128.84 hosts/sec). 11 responded

From the result you may be able to deduct a suitable IP address otherwise you need to log into your router, look for the DHCP configuration and check which range is used for DHCP. Then select an available IP outside that range.

You will also need to retrieve information on the gateway IP, subnet and DNS servers.

Get LAN device name

ip link | grep "en"

Your device name will be something like e.g. enp0s3 - make a note.

Get your current IP config - use the device name from above

ip address show dev enp0s3 | grep 'inet '

Note the address and subnet e.g.

inet 192.168.x.y/24

Setting up network

To setup a static network connection you need to create an interface configuration. Use a template to create a network interface in the netctl folder using the interface name from above

sudo cp /etc/netctl/examples/ethernet-static /etc/netctl/enp0s3

Edit the template

sudo nano /etc/netctl/enp0s3

Change the interface name to read enp0s3 matching the interface from above.

Interface=enp0s3

Modify the Address line to correspond to the ip address config from above but using the selected static IP address - like this - replace x.y accordingly

Address=('192.168.x.y/24')

Find your gateway using this command

ip route show dev enp0s3 | grep 'default'

Locate the gateway IP e.g. 192.168.x.1 and modify the gateway line

Gateway='192.168.x.1'

Use dig from bind-tools to retrieve your networks nameserver

dig manjaro.org | grep "SERVER"

Modify the DNS line with the IP address e.g. 192.168.x.1

DNS=('192.168.x.1')

Save the file and close it. Then list your profiles to verify is is found

$ netctl list
    enp0s3

Enable the profile and instruct system to wait for the connection

sudo netctl enable enp0s3
sudo netctl wait-online enp0s3

Note
Please refer to the excellent netctl configuration page on the Arch Wiki.

Setting up Apache

The default Apache configuration can be found in /etc/httpd/conf/httpd.conf. The file is very well documented and it is a very good idea to read through it. This is no place for going into depths on running an Apache web server - there is lots of resources on that subject - just use your favorite search engine.

The default folder for storing files to be served to visitors is located in /srv/http and this is as good a place as any for the mirrored packages.

As we are in a chroot, the only things we do here is

  • enable the apache service
systemctl enable httpd.service
  • create a single html file to test the service
nano /srv/http/index.html

Input some text to be displayed and save the file by pressing F2yEnter

Manjaro Mirror service

Cancel the chroot typing exitEnter. Then return to the installer by typing fgEnter. Reboot the system.

sudo reboot

Test the http service by opening a browser pointing to the IP address of the mirror server. If you are good to go you can read the text from above in your browser.

Syncing the repo from another mirror

The recommended method of syncing a mirror is using rsync. A couple of mirrors (Manjro Wiki) is offering rsync as means to sync the Manjaro repo (the list seems outdated).

Additional mirrors offering rsync (forum search) urls not verified.

Sync script

Sample script to sync the repo from University of Kent. Setup the system using a cron or a systemd timer job every 4-6 hours .

A working sample script using rsync
#!/bin/sh
# This is a sample mirroring script.
HOME="/srv/http"
TARGET="${HOME}/manjaro"
TMP="${HOME}/.tmp/manjaro"
LOCK="/tmp/rsync-manjaro.lock"

# NOTE: You'll probably want to change this or remove the --bwlimit setting in
# the rsync call below
BWLIMIT=10000

SOURCE="rsync://mirrorservice.org/repo.manjaro.org/repos/"

[ ! -d "${TARGET}" ] && mkdir -p "${TARGET}"
[ ! -d "${TMP}" ] && mkdir -p "${TMP}"

exec 9>"${LOCK}"
flock -n 9 || exit

if ! stty &>/dev/null; then
    QUIET="-q"
fi

rsync -rtlvH --safe-links \
    --bwlimit=${BWLIMIT} \
    --delete-after --progress \
    -h ${QUIET} --timeout=600 --contimeout=120 -p \
    --delay-updates --no-motd \
    --temp-dir="${TMP}" \
    ${SOURCE} \
    "${TARGET}"

Testing your mirror

Ask for help in the forum to test the stability and speed of your mirror. Preferably users from your country of origin. It is easy to add a mirror - editing the mirror list by hand.

It is only then you know if your setup will work in the long run.

Going official

When your mirror runs smooth and it is syncing successfully you create a #manjaro-development:feature-request with your mirror's country of origin, protocol(s) and internet url address.

It is also possible to create a PR or issue on Manjaro Gitlab

10 Likes

Todo (wiki - edit as necessary)

  • Instructions on getting network info using CLI
  • Instructions on setting static IP on network
  • Instructions on setting up cron job
  • Instructions on getting mirror listed as official mirror
1 Like

Hi for me this script not work i want to make a officialy mirror on my contry but with your script have this : rsync error: timeout waiting for daemon connection (code 35) at socket.c(281) [Receiver=3.1.2]
You have see this error ?
and service is started

I snipped the rsync mirror from the wiki (and I noted it seems outdated)

https://wiki.manjaro.org/index.php?title=Manjaro_Mirrors#How_is_the_mirror_synchronized.3F

Maybe I should have verified the url :slight_smile: which I have done now and the correct url seems to be

rsync://mirrorservice.org/repo.manjaro.org/repos/

I have tested the mirrors rsync and it is the url. The OP and the wiki has been edited to match this new info.

Do you think we should package a script to automatically deploy a manjaro mirror?

Nah - too many things to ensure correct functionality - yet you are the scripting guy.

Also I have no idea which mirrors have rsync - probably more than listed on the wiki - which is where I got my info.

And using rsync is pretty much mandatory to run a succesful mirror.

Everyting in the guide is using the defaults so - it might be possible to create a script which installs the base, setting up the defaults and create the network interface from arguments

  • ip/subnet/router/nameserver
  • rsync mirror
+ HOME=/tmp/http
+ TARGET=/tmp/http/manjaro
+ TMP=/tmp/http/.tmp/manjaro
+ LOCK=/tmp/rsync-manjaro.lock
+ BWLIMIT=10000
+ SOURCE=rsync://mirrorservice.org/repo.manjaro.org/repos/
+ '[' '!' -d /tmp/http/manjaro ']'
+ '[' '!' -d /tmp/http/.tmp/manjaro ']'
+ exec
+ flock -n 9
+ stty
+ rsync -rtlvH --safe-links --bwlimit=10000 --delete-after --progress -h --timeout=600 --contimeout=120 -p --delay-updates --no-motd --temp-dir=/tmp/http/.tmp/manjaro rsync://mirrorservice.org/repo.manjaro.org/repos/ /tmp/http/manjaro
rsync: getaddrinfo: mirrorservice.org 873: Name or service not known
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]

Hmm - I did test the url by running a copy of the script and it worked.

diskstation>  cat kent.sh
#!/bin/sh

HOME="/volume1/web"
TARGET="${HOME}/public/kent"
TMP="${HOME}/.tmp/kent"
LOCK="/tmp/rsync-kent.lock"

# NOTE: You'll probably want to change this or remove the --bwlimit setting in
# the rsync call below
BWLIMIT=100000

SOURCE="rsync://mirrorservice.org/repo.manjaro.org/repos/"

[ ! -d "${TARGET}" ] && mkdir -p "${TARGET}"
[ ! -d "${TMP}" ] && mkdir -p "${TMP}"

exec 9>"${LOCK}"
flock -n 9 || exit

## if we are called without a tty (cronjob) only run when there are changes
#if ! tty -s && diff -b <(curl -s "${STATE}") "${TARGET}/state" >/dev/null; then
#	exit 0
#fi

if ! stty &>/dev/null; then
    QUIET="-q"
fi

# sleep $((RANDOM%60))

# excluded fh 2016-11-01
rsync -rtlvH --safe-links \
    --bwlimit=${BWLIMIT} \
    --delete-after --progress \
    -h ${QUIET} --timeout=600 --contimeout=120 -p \
    --delay-updates --no-motd \
    --temp-dir="${TMP}" \
    ${SOURCE} \
    "${TARGET}"

Sample output from my server

diskstation> ./kent.sh 
receiving file list ... 
192226 files to consider
./
state
         259 100%  252.93kB/s    0:00:00 (total: 0%) (xfer#1, to-check=192224/192226)
pool/
pool/overlay-32/
pool/overlay-32/abiword-gtk2-3.0.2-3-i686.pkg.tar.xz
       3.73M 100%   18.32MB/s    0:00:00 (total: 0%) (xfer#2, to-check=192221/192226)
pool/overlay-32/abiword-gtk2-3.0.2-3-i686.pkg.tar.xz.sig
         543 100%    2.72kB/s    0:00:00 (total: 0%) (xfer#3, to-check=192220/192226)

You could try another server.

+ HOME=/tmp/web
+ TARGET=/tmp/web/public/kent
+ TMP=/tmp/web/.tmp/kent
+ LOCK=/tmp/rsync-kent.lock
+ BWLIMIT=100000
+ SOURCE=rsync://mirrorservice.org/repo.manjaro.org/repos/
+ '[' '!' -d /tmp/web/public/kent ']'
+ mkdir -p /tmp/web/public/kent
+ '[' '!' -d /tmp/web/.tmp/kent ']'
+ mkdir -p /tmp/web/.tmp/kent
+ exec
+ flock -n 9
+ stty
+ rsync -rtlvH --safe-links --bwlimit=100000 --delete-after --progress -h --timeout=600 --contimeout=120 -p --delay-updates --no-motd --temp-dir=/tmp/web/.tmp/kent rsync://mirrorservice.org/repo.manjaro.org/repos/ /tmp/web/public/kent
rsync: getaddrinfo: mirrorservice.org 873: Name or service not known
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]

Probably have a bad configuration on my pc...

/etc/rsyncd.conf                                     
uid = nobody
gid = nobody
use chroot = no
max connections = 4
syslog facility = local5
pid file = /run/rsyncd.pid

[ftp]
        path = /var/ftp
        comment = ftp area

This is my classique config and rsyncd is on listen on 873

If you have any idea ?

Best regards

And have try this server too : rsync://fr.mirror.babylon.network/manjaro

You don't use the rsync daemon - just the rsync command.

The rsyncd.conf only has meaning if you are offering a rsync service for others to connect to.

Yes I know it's exactly what bothers me why he tells me at the beginning that the deamon is not started while rsync is already started with all the arguments

babylon network does not mirror manjaro - I don't remember when they stopped.

I have removed them from the wiki page.

DNS issue?

im use personal dns

Your DNS resolver cannot lookup the address for server

yes is only this mirrorservice.org is locked ok sorry to disturb for that im go to open public mirror in france for manjaro community this link : https://manjaro.ynh.ovh