[root tip] [How To] Create Manjaro mirror server

Contributions Tutorials
1

Difficulty: ★★★☆☆

Create a Manjaro mirror using Architect

Manjaro Architect is well suited for creating custom installations and a custom installation is needed for running a mirror of Manjaro repo.

Manjaro can benefit from more mirrors but please consider this before committing to run an official mirror.

Prerequisites

Internet connection

A static IP address and a domain name with access to setting up DNS to point to your IP. It is possible to use dynamic DNS but it is not as reliable as a static IP.

You need to be able to setup your router to forward traffic to the system running the mirror and you also need to setup your LAN interface to use static addressing.

Don’t be tempted to use a wireless connection - use a wired connection directly to your router.

Hardware

I won’t go into details with your hardware, suffice to say it is possible to run a mirror on a Raspberry Pi using raspbian with an 128GB SSD connected using USB. And disk space is the most important remedy. You need at least 128GB preferrably 240GB to avoid running full.

diskstation> du -sh /volume1/web/public/manjaro
94.4G	/volume1/web/public/manjaro

Software

A syncronization app such as rsync (recommended) or wget and a webserver e.g. apache or nginx configured to serve directory listings.

Installation

If you have a recent Manjaro ISO available there is no need to download another as Architect has been included for quite some time. Start by booting your hardware on your Manjaro ISO.

Note
The below steps is the simplified version - however you may be a experienced user and have intimate knowledge on how you want the system - sure - you decide on how the system should be setup so please skip this section.

Before launching Architect update your mirrorlist using the terminal

sudo pacman-mirrors -f && sudo pacman -Syy

Ensure have have an internet connection and launch Architect using your terminal.

sudo manjaro-architect

Select your language and continue to the Main Menu

  • Press 1Enter on the Prepare Installation item.
  • Select keyboard using the Set Virtual Console entry and press Enter.
  • Partition Disk entry.
    • Select the Automatic Partitioning.
  • Next is Mount Partitions.
    • Mount the root partition / and format to ext4 using the default options presented.
    • Unless you are running from SSD please do create a swapfile.
    • Select Done (Do not select to mount the 512MB partition).
    • When prompted for the EFI partition, choose your mountpoint (usually /boot/efi), select the 512MB partition and format to FAT32.
    • Press 9Enter to Refresh Pacman Keys.
    • Press Enter to Choose pacman cache and select Yes.
    • Select Back and press Enter.

Back in the Main Menu press 4Enter to Install CLI System

  • Select Install Base and select linux419.
  • Select OK in the next window and wait while the base system is installed.
  • On the Network Drivers page select appropriate for your system - free drivers will usually suffice.
  • Install bootloader → select grub
  • Configure Base
    • Generate FSTAB → Device UUID (default)
    • Set your Hostname (network unique required)
    • Set your System Locale
    • Set your Timezone and Clock
    • Set Root Password
    • Create New User
    • Select Shell
    • Set User Password
    • Select Back

Back in the CLI System Menu select Install Custom Packages

  • Type in apacheTab to select the apache package.
  • Delete and type netctl Tab to select the netctl package.
  • Delete and type arp-scan Tab to select the arp-scan package.
  • Delete and type openresolv Tab to select the openresolv package.
  • Delete and type bind-tools Tab to select the bind-tools package.
  • Press Enter to install selected packages.

Chroot into your installation

Select the item Chroot into Installation (make a note on how to return to the installer - namely first exitEnter then fgEnter).

LAN configuration

As you are going to have traffic forwarded through your router you need to assign a static IP address to your network interface.

Display information on your LAN

To retrieve information on connected devices we will use arp-scan to scan your network (Sample output from my network)

~ >>> sudo arp-scan -l
[sudo] password for user: 
Interface: enp0s3, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.9.5 with 256 hosts (https://github.com/royhills/arp-scan)
192.168.x.1	    fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.4	    b8:27:eb:xx:xx:xx	Raspberry Pi Foundation
192.168.x.5	    fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.10	00:11:32:xx:xx:xx	Synology Incorporated
192.168.x.30	18:e8:29:xx:xx:xx	(Unknown)
192.168.x.31	fc:ec:da:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.32	b4:fb:e4:xx:xx:xx	Ubiquiti Networks Inc.
192.168.x.151	c8:69:cd:xx:xx:xx	Apple, Inc.
192.168.x.201	3c:d9:2b:xx:xx:xx	Hewlett Packard
192.168.x.203	14:91:82:xx:xx:xx	Belkin International Inc.
192.168.x.204	b4:fb:e4:xx:xx:xx	Ubiquiti Networks Inc.

11 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.9.5: 256 hosts scanned in 1.987 seconds (128.84 hosts/sec). 11 responded

From the result you may be able to deduct a suitable IP address otherwise you need to log into your router, look for the DHCP configuration and check which range is used for DHCP. Then select an available IP outside that range.

You will also need to retrieve information on the gateway IP, subnet and DNS servers.

Get LAN device name

  • make a note
ip link | grep "en"

Your device name will be something like e.g. enp0s3

Get your current IP config - use the device name from above

  • make a note.
ip address show dev enp0s3 | grep 'inet '

Note the address and subnet e.g.

inet 192.168.x.y/24

Find your gateway using this command

  • make a note
ip route show dev enp0s3 | grep 'default'

Find your networks DNS

  • make a note
dig manjaro.org | grep "SERVER"

Setting up network

To setup a static network connection you need to create an interface configuration. Use a template to create a network interface in the netctl folder using the interface name from above

sudo cp /etc/netctl/examples/ethernet-static /etc/netctl/enp0s3

Edit the template

sudo nano /etc/netctl/enp0s3

Change the interface name to read enp0s3 matching the interface from above and fill the data from your notes.

Interface=enp0s3
Address=('192.168.x.y/24')
Gateway='192.168.x.1'
DNS=('192.168.x.1')

Save the file and close it. Then list your profiles to verify is is found

$ netctl list
    enp0s3

Enable the profile and instruct system to wait for the connection

sudo netctl enable enp0s3
sudo netctl wait-online enp0s3

Note
Please refer to the excellent netctl configuration page on the Arch Wiki.

Setting up Apache

The default Apache configuration can be found in /etc/httpd/conf/httpd.conf. The file is very well documented and it is a very good idea to read through it. This is no place for going into depths on running an Apache web server - there is lots of resources on that subject - just use your favorite search engine.

The default folder for storing files to be served to visitors is located in /srv/http and this is as good a place as any for the mirrored packages.

As we are in a chroot, the only things we do here is

  • enable the apache service
systemctl enable httpd.service
  • create a single html file to test the service
nano /srv/http/index.html

Input some text to be displayed and save the file by pressing F2yEnter

Manjaro Mirror service

Cancel the chroot typing exitEnter. Then return to the installer by typing fgEnter. Reboot the system.

sudo reboot

Test the http service by opening a browser pointing to the IP address of the mirror server. If you are good to go you can read the text from above in your browser.

Syncing the repo from another mirror

The recommended method of syncing a mirror is using rsync. A couple of mirrors (Manjaro Wiki) is offering rsync as means to sync the Manjaro repo (the list seems outdated).

Additional mirrors offering rsync (forum search) urls not verified.

Sync script

Sample script to sync the repo from University of Kent. Setup the system using a cron or a systemd timer job every 4-6 hours .

A working sample script using rsync 
#!/bin/sh
# This is a sample mirroring script.
HOME="/srv/http"
TARGET="${HOME}/manjaro"
TMP="${HOME}/.tmp/manjaro"
LOCK="/tmp/rsync-manjaro.lock"

# NOTE: You'll probably want to change this or remove the --bwlimit setting in
# the rsync call below
BWLIMIT=10000

SOURCE="rsync://mirrorservice.org/repo.manjaro.org/repos/"

[ ! -d "${TARGET}" ] && mkdir -p "${TARGET}"
[ ! -d "${TMP}" ] && mkdir -p "${TMP}"

exec 9>"${LOCK}"
flock -n 9 || exit

if ! stty &>/dev/null; then
    QUIET="-q"
fi

rsync -rtlvH --safe-links \
    --bwlimit=${BWLIMIT} \
    --delete-after --progress \
    -h ${QUIET} --timeout=600 --contimeout=120 -p \
    --delay-updates --no-motd \
    --temp-dir="${TMP}" \
    ${SOURCE} \
    "${TARGET}"

Testing your mirror

Ask for help in the forum to test the stability and speed of your mirror. Preferably users from your country of origin. It is easy to add a mirror - editing the mirror list by hand.

It is only then you know if your setup will work in the long run.

Going official

When your mirror runs smooth and it is syncing successfully you create a #manjaro-development:feature-request with your mirror’s country of origin, protocol(s) and internet url address.

It is also possible to create a PR or issue on Manjaro Gitlab

6 Likes
2

Thank you for the tutorial.
If Manjaro uses Cloud platform like AWS to host repos and use some of their services to cater all the regions without latency, there will be no need of mirrors. :wink:

3

For completeness the link to create a PR points as moved:
tools/maintenance-tools/manjaro-web-repo was moved to webpage/manjaro-web-repo.

4
5