Issue with arch-nspawn Creating Incorrect Mirror List

thursday_bw · 13 June 2024 07:19

I am experiencing issues with the mirror list generated by arch-nspawn inside a chroot environment. The mirrors in the chroot environment are reachable, but pacman reports 404 errors when trying to synchronize package databases.

Steps to Reproduce:

Create a chroot environment using arch-nspawn.
Compare the mirror lists between the host and the chroot.
Attempt to synchronize package databases in the chroot.

Trigger accessing mirrors from inside chroot:

sudo arch-nspawn /mnt/chroot/audacity/root pacman -Sy --print
:: Synchronizing package databases...
 core
 extra
 multilib
 extra is up to dateerror: failed retrieving file 'multilib.db' from nnenix.mm.fcix.net : The requested URL returned error: 404
error: failed retrieving file 'core.db' from nnenix.mm.fcix.net : The requested URL returned error: 404r: 404
error: failed retrieving file 'core.db' from mirror.netcologne.de : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from mirror.netcologne.de : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from www.mirrorservice.org : The requested URL returned error: 404
error: failed retrieving file 'core.db' from www.mirrorservice.org : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from ftp.rz.tu-bs.de : The requested URL returned error: 404
error: failed retrieving file 'core.db' from ftp.rz.tu-bs.de : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from mirrors.ustc.edu.cn : The requested URL returned error: 404
error: failed retrieving file 'core.db' from mirrors.ustc.edu.cn : The requested URL returned error: 404
error: failed retrieving file 'core.db' from mirrors.sjtug.sjtu.edu.cn : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from mirrors.sjtug.sjtu.edu.cn : The requested URL returned error: 404
 core.db failed to downloaderror: failed retrieving file 'multilib.db' from ct.mirror.garr.it : The requested URL returned error: 404
 multilib.db failed to downloaderror: failed to synchronize all databases (failed to retrieve some files)

Trigger those mirrors on host:

[bevan@legion setup_chroot]$ sudo pacman -Sy --print
:: Synchronising package databases...
 core is up to date
 extra is up to date
 multilib is up to date

Show host mirrors list:

cat /etc/pacman.d/mirrorlist
##
## Manjaro Linux default mirrorlist
## Generated on 2024-06-07 16:07
##
## Please use 'pacman-mirrors -f [NUMBER] [NUMBER]' to modify mirrorlist
## (Use 0 for all mirrors)
##

## Country : United_States
Server = https://opencolo.mm.fcix.net/manjaro/stable/$repo/$arch

## Country : United_States
Server = https://nnenix.mm.fcix.net/manjaro/stable/$repo/$arch

## Country : Germany
Server = https://mirror.netcologne.de/manjaro/stable/$repo/$arch

## Country : United_Kingdom
Server = https://www.mirrorservice.org/sites/repo.manjaro.org/repos/stable/$repo/$arch

## Country : Germany
Server = http://ftp.rz.tu-bs.de/pub/mirror/manjaro.org/repos/stable/$repo/$arch

## Country : China
Server = https://mirrors.ustc.edu.cn/manjaro/stable/$repo/$arch

## Country : China
Server = https://mirrors.sjtug.sjtu.edu.cn/manjaro/stable/$repo/$arch

## Country : Italy
Server = https://ct.mirror.garr.it/mirrors/manjaro/stable/$repo/$arch

Show chroot mirrors list:

arch-nspawn /mnt/chroot/audacity/root cat /etc/pacman.d/mirrorlist
Server = https://opencolo.mm.fcix.net/manjaro/stable/extra/x86_64
Server = https://nnenix.mm.fcix.net/manjaro/stable/extra/x86_64
Server = https://mirror.netcologne.de/manjaro/stable/extra/x86_64
Server = https://www.mirrorservice.org/sites/repo.manjaro.org/repos/stable/extra/x86_64
Server = http://ftp.rz.tu-bs.de/pub/mirror/manjaro.org/repos/stable/extra/x86_64
Server = https://mirrors.ustc.edu.cn/manjaro/stable/extra/x86_64
Server = https://mirrors.sjtug.sjtu.edu.cn/manjaro/stable/extra/x86_64
Server = https://ct.mirror.garr.it/mirrors/manjaro/stable/extra/x86_64

Is this because arch-nspawn is incompatible with manjaro?

System Info

Get pacman version

pacman --version

 .--.                  Pacman v6.1.0 - libalpm v14.0.0
/ _.-' .-.  .-.  .-.   Copyright (C) 2006-2024 Pacman Development Team
\  '-. '-'  '-'  '-'   Copyright (C) 2002-2006 Judd Vinet
 '--'
                       This program may be freely redistributed under
                       the terms of the GNU General Public License.

Get devtools version

pacman -Qi devtools
Name            : devtools-git
Version         : 1:1.2.0.r9.gc1a3ed2-1
Description     : Tools for Arch Linux package maintainers
Architecture    : any
URL             : https://github.com/archlinux/devtools
Licenses        : GPL
Groups          : None
Provides        : devtools=1.2.0
Depends On      : bash  openssh  subversion  rsync  arch-install-scripts  git  bzr  mercurial  diffutils  util-linux  awk
Optional Deps   : btrfs-progs: btrfs support [installed]
Required By     : None
Optional For    : None
Conflicts With  : devtools
Replaces        : None
Installed Size  : 338.60 KiB
Packager        : Unknown Packager
Build Date      : Wed 12 Jun 2024 14:37:12
Install Date    : Wed 12 Jun 2024 18:10:24
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : None

Get OS information

cat /etc/os-release
NAME="Manjaro Linux"
PRETTY_NAME="Manjaro Linux"
ID=manjaro
ID_LIKE=arch
BUILD_ID=rolling
ANSI_COLOR="32;1;24;144;200"
HOME_URL="https://manjaro.org/"
DOCUMENTATION_URL="https://wiki.manjaro.org/"
SUPPORT_URL="https://forum.manjaro.org/"
BUG_REPORT_URL="https://docs.manjaro.org/reporting-bugs/"
PRIVACY_POLICY_URL="https://manjaro.org/privacy-policy/"
LOGO=manjarolinux

Moderator edit: Formatting

Nachlese · 13 June 2024 07:53

The arch-nspawn command is part of the Arch-specific devtools package.

There is however:

systemd-nspawn - ArchWiki

which can probably do what you want - of which I don’t know what it is.

linux-aarhus · 13 June 2024 08:18

Server = https://opencolo.mm.fcix.net/manjaro/stable/$repo/$arch

Your mirrorlist in arch-nspawn is holding the repo and architecture in the list - so there has been variable expansion inside the arch-nspawn.

This explains why you get a 404 on core.db - it does not exist in the extra repo.

I don’t know how arch-nspawn works - but from the command it looks like you are doing a build in a clean chroot

arch-nspawn /mnt/chroot/audacity/root cat /etc/pacman.d/mirrorlist

For use with Manjaro - you will likely have better succes with using the package - manjaro-chrootbuild.

thursday_bw · 13 June 2024 08:21

Wow, thank you for replying so fast.

Neither do I unfortunately.

what do you mean by arch-specific, there are many tools built for arch that also work on manjaro. I am new to this, as Manjaro is Arch based shouldn’t’ it be covered?

I don’t have an Arch based system to test it on, and i’m wondering if I should open an issue on devtools.

I don’t really know what systemd-nspawn looks interesting, thanks for that. I am using arch-chroot(8) — Arch manual pages instead of arch-nspawn.

I could open an issue on devtools for arch-spawn if it should work in manjaro.

thursday_bw · 13 June 2024 08:26

Frede Hundewadt:

Your mirrorlist in arch-nspawn is holding the repo and architecture in the list - so there has been variable expansion inside the arch-nspawn.

thursday_bw:

error: failed retrieving file 'core.db' from nnenix.mm.fcix.net : The requested URL returned error: 404r: 404

This explains why you get a 404 on core.db - it does not exist in the extra repo.

I don’t know how arch-nspawn works - but from the command it looks like you are doing a build in a clean chroot
arch-nspawn /mnt/chroot/audacity/root cat /etc/pacman.d/mirrorlist
For use with Manjaro - you will likely have better succes with using the package - manjaro-chrootbuild.

“so there has been variable expansion inside the arch-nspawn.” yeah… that’s why I wondered if arch’s mirror list format might be different and thus this is a manjaro only issue… I dont’ know and have spent days diagnosing so dont’ feel like installing my first ever arch machine to test it.

“I don’t know how arch-nspawn works” I kind of do… it does exactly what it’s designed to do but doesn’t work, even on really early commits… I do think it’s just incomatible with manjaro’s mirror list format… not sure.

“you will likely have better succes with using the package - manjaro-chrootbuild.” that’s a great tip!! I’m using arch-chroot for now, but if I get problems with that I’ll take a look.

linux-aarhus · 13 June 2024 08:32

Exactly that - arch-specific.

Not necessarily - as noted variable expansion causes this - so perhaps you are using the wrong tool?

The arch devtools package is not available within Manjaro repo - so unsupported.

thursday_bw · 13 June 2024 08:36

“The arch devtools package is not available within Manjaro repo - so unsupported.” now I get you.
Thank you.

OK. I don’t want to do too much more yak-shearing when arch-chroot is doing what I need (so far). I’ll keep an eye on manjaro-chrootbuild that sounds like it is probably the clue I did not have.

Nachlese · 13 June 2024 08:37

I meant: I don’t know what you want to do - you surely do.

It’s an assumption that you make - perhaps unwarranted.

… and also some that don’t …
Maybe reading the script (arch-nspawn is a bash script) will tell you.

As I said:
I don’t know what your goal is - and systemd-nspawn is definitely not Arch specific.

What you are using right now is an AUR package, a version of software that is available natively in Arch.
The AUR let’s you have it on Manjaro as well - doesn’t mean it’s useful here.

thursday_bw · 13 June 2024 08:47

OK. I’ll go really high level, I am trying to adapt https://github.com/intel/openvino-plugins-ai-audacity/blob/43de3af6087480ab2030404aba3f122fafa9bd73/doc/build_doc/linux/README.m into a PKGBUILD script to build d on manjaro. So I want to run it in a clean environment (chroot), isolated from my host so that I can get teh right dependencies sorted out.

From what I had read I was under the impression arch-nspawn as the right tool to run that clean environment… but I was wrong and made some incorrect assumptions.

I have tried searching for documentation on “developing and testing PKGBUILD scripts in Manjaro” but I haven’t found anything that tells me what I want. Hence why I think that manjaro-chrootbuild is such a good clue.
Is there manjaro specific documentation for how to build and test PKGBUILD scripts?

thursday_bw · 13 June 2024 08:56

These are the new names for renamed scripts.

    mkmanjaroroot --> mkchroot
    manjarobuild --> buildpkg
    mkset --> buildset
    pacstrap --> basestrap
    genfstab --> fstabgen
    arch-chroot --> manjaro-chroot

ah “arch-chroot → manjaro-chroot” would ya look at that.

though maybe manjaro-chrootbuild is more appropriate, I’m not sure. I have a few things to try, thank you.

I’m little over my head, thanks for the lifeline.

edit:

manjaro-chrootbuild appears to be designed explicitly for installing packages from a repo, not from a local PKGBUILD file.

Since manjaro-tools supplies manjaro-chroot as a replacement for arch-chroot I am going to try and use that. And for the moment forget about arch-nspawn, and I’ll keep systemd-nspawn on the back burner.

linux-aarhus · 13 June 2024 11:15

Creating a PKGBUILD follows same concepts as Arch.

Using chrootbuild ensures a clean chroot -from time to time you would want to start with a clean chroot.

 $ chrootbuild -h

Usage: chrootbuild [options]

     -b <branch> Branch to use:
                 (unstable/testing/stable-staging/stable;
                 arm-unstable/arm-testing/arm-stable)
                 default: unstable / arm-unstable
     -c          Start with clean chroot fs
     -d          Disable colors.
     -D          Build with debug symbols.
     -f          Force unmount chroot if busy.
     -g          Push changes to git when building lists
     -G          Generate Checksums
     -h          This help
     -H          Use the host's keyrings
     -i <pkg>    Install package(s) to chroot fs
                 (for multiple packages repeat -i flag)
     -k <repo>   Use custom repo:
                 (mobile/kde-unstable)
     -K <list>   Kernel-modules list to build
     -l <list>   List(s) to build
                 (for multiple lists repeat -l flag)
     -L          enable LinkTimeOptimization
     -m          Build a multilib package
     -M <url>    Use custom mirror
     -n          Install built pkg to chroot fs
     -p <pkg>    Package(s) to build
                 (for multiple packages repeat -p flag)
     -r          custom chrootdir path
                 default: /var/lib/chrootbuild
     -s          Sign package(s)
     -u          Build pkgs only if update available (lists only)
     -x          Remove previously built packages in $PKGDEST

thursday_bw · 13 June 2024 13:30

Thanks for that. The challenge I appear to have is that the module I am building into audacity needs older dependencies, such as gcc11 which are in the AUR.

I need my chroot to have those older dependencies installed.
I have tried buildpkg, chrootbuild as well as manjaro-chroot, but I hit a road block at every turn… and so I may be missing something really simple.

How to I build a PKGBUILD that has dependencies on AUR packages, things like buildpkg and chroot use pacman so they fail, and the build there own chroot so I have issue getting into it to add the dependenices myself.

Yochanan · 13 June 2024 14:57

Build the dependencies first, then install them in the chroot:

With manjaro-chrootbuild:

sudo chrootbuild -cp <package-name> -i </path/to/dependency/package.tar.zst> -i </path/to/dependency/package.tar.zst>

thursday_bw · 13 June 2024 15:13

OK… I really want to express my thanks for your response.

I have a script at the moment that builds a chroot installs yay in it, creates a user inside it and then runs yay as that user to install the packages in the chroot.

One of the packages takes ages to build, so I’m going to put this aside and go to bed, and check on it tomorrow.

I think I understand what you mean, something like this?:

ls /build/mypackage/PKGBUILD
cd /build
git clone https://aur.archlinux.org/gcc11
buildpkg -p gcc11
sudo chrootbuild -cp mypackage -i /var/cache/manjaro-tools/pkg/stable/x86_64/gcc.tar.zst

edit:
Adding a link to the new issue I opened: Issues with Building mod-openvino for Audacity on Manjaro - Need Help with chroot Setup

Yochanan · 13 June 2024 19:21

If you were using the proper tools to begin with, you neither would have needed to create a script nor use an AUR helper like yay . Having said that, I realize you were not aware. Now you are.

By the way, I forgot to mention buildpkg is deprecated in favor of chrootbuild.

Er… why do you think you need GCC 11?

In summary, this topic is apparently an XY Problem. Please create a new topic about what you actually need help with and link to this topic for reference.