CDN77 is changing to S3 Object storage. Therefore we have to remove our CDN based Mirror either to S3 or self-hosted. I managed now to create some sort of solution for that problem. I need now some testers to switch over to that mirror with over 110 data centers world-wide. You can report speeds and any issues you may face:
Change $branch to one of the following as needed:
sudo pacman-mirrors -aU https://mirrors2.manjaro.org
Expect pacman-mirrors --status to return a message stating the mirror does not exist.
$ sudo pacman-mirrors -aU https://mirrors2.manjaro.org
::INFO Writing mirror list
::BUILDMIRROR : https://mirrors2.manjaro.org/unstable
::INFO Mirror list generated and saved to: /etc/pacman.d/mirrorlist
$ cat /etc/pacman.d/mirrorlist
##
## Manjaro Linux default mirrorlist
## Generated on 2023-12-07 17:57
##
## Please use 'pacman-mirrors -f [NUMBER] [NUMBER]' to modify mirrorlist
## (Use 0 for all mirrors)
##
## Country : BUILDMIRROR
Server = https://mirrors2.manjaro.org/unstable/$repo/$arch
$ pacman-mirrors
Pacman-mirrors version 4.24.0
Local mirror status for unstable branch
Mirror # 1 https://mirrors2.manjaro.org/ does not exist
mirrorlist:
## Country : Global
Server = https://mirrors2.manjaro.org/repo/testing/$repo/$arch
$ LC_ALL=C sudo pacman -Syy 1 ✘
:: Synchronizing package databases...
core.db failed to download
extra.db failed to download
multilib.db failed to download
error: failed retrieving file 'core.db' from mirrors2.manjaro.org : The requested URL returned error: 404
error: failed retrieving file 'multilib.db' from mirrors2.manjaro.org : The requested URL returned error: 404
error: failed retrieving file 'extra.db' from mirrors2.manjaro.org : The requested URL returned error: 404
warning: too many errors from mirrors2.manjaro.org, skipping for the remainder of this transaction
error: failed to synchronize all databases (failed to retrieve some files)
for display mirror, $repo $arch : pacman-conf -r core
pacman-conf -r core
Usage = All
Server = https://mirrors2.manjaro.org/stable/core/x86_64
Server = https://mirrors.manjaro.org/repo/stable/core/x86_64
or best, if error, we can add option --debug to pacman
debug: core.db: url is https://mirrors2.manjaro.org/stable/core/x86_64/core.db
debug: core.db: maxsize 134217728
debug: core.db: opened tempfile for download: /var/lib/pacman/sync/core.db.part (wb)
debug: core.db: curl returned result 0 from transfer
debug: core.db: response code 200
debug: core.db.sig: url is https://mirrors2.manjaro.org/stable/core/x86_64/core.db.sig
debug: core.db.sig: maxsize 16384
debug: core.db.sig: opened tempfile for download: /var/lib/pacman/sync/core.db.sig.part (wb)
debug: core.db.sig: curl returned result 0 from transfer
debug: core.db.sig: response code 404
debug: core.db.sig: no more servers to retry
I’ve noticed, that I used wrong URL, with extra /repo/ in it. Now with correct URL:
Server = https://mirrors2.manjaro.org/testing/$repo/$arch
pacman-mirrors still gives an error:
$ pacman-mirrors ✔ 14s
Pacman-mirrors version 4.24.0
Local mirror status for testing branch
Mirror # 1 https://mirrors2.manjaro.org/ does not exist
But sudo pacman -Syy works OK:
$ LC_ALL=C sudo pacman -Syu ✔ 4s
:: Synchronizing package databases...
core is up to date
extra is up to date
multilib is up to date
:: Starting full system upgrade...
there is nothing to doThat is true, I did not even try to sync, but that works.
No more error:
$ pacman-mirrors ✔
Pacman-mirrors version 4.24.0
Local mirror status for testing branch
Mirror #1 OK 00:08 Global https://mirrors2.manjaro.org/
I tried to install a package, all went OK
$ sudo pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
multilib is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
Packages (7) curl-8.5.0-1 glib2-2.78.3-1 lib32-curl-8.5.0-1 lib32-glib2-2.78.3-1 libcurl-gnutls-8.5.0-1 libdecor-0.2.1-2 thin-provisioning-tools-1.0.8-1
Total Download Size: 7,86 MiB
Total Installed Size: 34,62 MiB
Net Upgrade Size: 0,02 MiB
:: Proceed with installation? [Y/n]
:: Retrieving packages...
libcurl-gnutls-8.5.0-1-x86_64 303,6 KiB 979 KiB/s 00:00 [#############################################################################] 100%
lib32-curl-8.5.0-1-x86_64 298,8 KiB 260 KiB/s 00:01 [#############################################################################] 100%
libdecor-0.2.1-2-x86_64 45,9 KiB 170 KiB/s 00:00 [#############################################################################] 100%
thin-provisioning-tools-1.0.8-1-x86_64 1002,6 KiB 293 KiB/s 00:03 [#############################################################################] 100%
lib32-glib2-2.78.3-1-x86_64 1248,6 KiB 269 KiB/s 00:05 [#############################################################################] 100%
curl-8.5.0-1-x86_64 1239,9 KiB 231 KiB/s 00:05 [#############################################################################] 100%
glib2-2.78.3-1-x86_64 3,8 MiB 496 KiB/s 00:08 [#############################################################################] 100%
Total (7/7) 7,9 MiB 954 KiB/s 00:08 [#############################################################################] 100%
(7/7) checking keys in keyring [#############################################################################] 100%
(7/7) checking package integrity [#############################################################################] 100%
(7/7) loading package files [#############################################################################] 100%
(7/7) checking for file conflicts [#############################################################################] 100%
(7/7) checking available disk space [#############################################################################] 100%
:: Running pre-transaction hooks...
-------------------------cut for unnecessary text--------------------------------------
:: Processing package changes...
(1/7) upgrading curl [#############################################################################] 100%
(2/7) upgrading glib2 [#############################################################################] 100%
(3/7) upgrading lib32-curl [#############################################################################] 100%
(4/7) upgrading lib32-glib2 [#############################################################################] 100%
(5/7) upgrading libcurl-gnutls [#############################################################################] 100%
(6/7) upgrading libdecor [#############################################################################] 100%
New optional dependencies for libdecor
gtk3: gtk3 support [installed]
(7/7) upgrading thin-provisioning-tools [#############################################################################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Refreshing PackageKit...
My speeds are due to my slow internet connection (multiple users on 10Mbit), nothing on your side, so they can not be used in any kind of metrics for you.
The error in pacman-mirrors may be connected with the fact the new mirror is not browsable (it returns 403 if you try to open a folder and not a specific file)?
With pacman -Syw cuda, from Helsink in a 100Mb/s connection:
opencl-nvidia-545.29.06-1-x86_64 2,8 MiB 1515 KiB/s 00:02 [-------------------------------------] 100%
gcc12-libs-12.3.0-3-x86_64 202,3 KiB 1314 KiB/s 00:00 [-------------------------------------] 100%
egl-wayland-2:1.1.13-1-x86_64 34,2 KiB 58,7 KiB/s 00:01 [-------------------------------------] 100%
eglexternalplatform-1.1-2-any 7,1 KiB 71,1 KiB/s 00:00 [-------------------------------------] 100%
gcc12-12.3.0-3-x86_64 38,2 MiB 4,28 MiB/s 00:09 [-------------------------------------] 100%
nvidia-utils-545.29.06-1-x86_64 239,9 MiB 6,20 MiB/s 00:39 [-------------------------------------] 100%
cuda-12.3.0-6-x86_64 1519,4 MiB 4,34 MiB/s 05:50 [-------------------------------------] 100%
Total (7/7) 1800,6 MiB 5,14 MiB/s 05:51 [-------------------------------------] 100%
With my normal mirror (thanks Kyberorg!) that is in my own city:
opencl-nvidia-545.29.06-1-x86_64 2,8 MiB 657 KiB/s 00:04 [-------------------------------------] 100%
gcc12-libs-12.3.0-3-x86_64 202,3 KiB 373 KiB/s 00:01 [-------------------------------------] 100%
egl-wayland-2:1.1.13-1-x86_64 34,2 KiB 89,4 KiB/s 00:00 [-------------------------------------] 100%
eglexternalplatform-1.1-2-any 7,1 KiB 22,6 KiB/s 00:00 [-------------------------------------] 100%
gcc12-12.3.0-3-x86_64 38,2 MiB 572 KiB/s 01:08 [-------------------------------------] 100%
cuda-12.3.0-6-x86_64 486,0 MiB 11,1 MiB/s 00:44 [-------------------------------------] 100%
nvidia-utils-545.29.06-1-x86_64 239,9 MiB 1655 KiB/s 02:28 [-------------------------------------] 100%
Total (7/7) 1800,6 MiB 12,1 MiB/s 02:28 [-------------------------------------] 100%
Yes, the new URL doesn’t have the extra repo folder in it. When the old gets shutdown we may have to change that URL also without the repo folder. So far it seems to work great.
Also Traffic looks good so far.
Also all Datacenters are active:
This is not an error - it is an informative message - and it is expected as the mirror is manually set and the mirror is not in the official pool of mirrors as defined by mirrors.json
@philm Do you still need us to use this server or can I move back to my normal mirrors?
Edit
Ran one more update and:
error: failed retrieving file 'linux61-6.1.66-1-x86_64.pkg.tar.zst' from mirrors2.manjaro.org : HTTP/2 stream 9 was not closed cleanly: INTERNAL_ERROR (err 2)
warning: failed to retrieve some files
error: failed to commit transaction (unexpected error)
Errors occurred, no packages were upgraded.
Not sure if it was on your or my end, but here you go.
You can switch back at any time.
I am using the same mirror - and I have zero issues …
And for the record - is has been added to the official pool
$ pacman-mirrors
Pacman-mirrors version 4.24.0
Local mirror status for unstable branch
Mirror #1 OK 00:08 Global https://mirrors2.manjaro.org/
Well, you might not use the same node, cos we have 110 data centers mapped to that one URL. So if one node has an issue, simply rerun the update process @bedna.
Yes - I know that - even so @bedna is located in Sweden - me in Denmark - pretty close - though we don’t really know what algorithms is used to match which node is used.
I am located in Aarhus DK - distance to Copenhagen in straight line is 250km - distance to e.g. Stockholm is - straight line - a rough estimate from Copenhagen 400km - does this imply Stockholm is closer to e.g. Helsinki by wire or is is closer to Copenhagen?
Well the old approach with mirrors.manjaro.org was, that we had a copy of the repositories on a CDN77 storage solution and then the nodes would download and cache from there.
With the new approach on mirrors2.manjaro.org we use the Origin server of Manjaro Packages as direct source. So now the data center nodes get the files directly from the Origin server and cache them. Since some files like DBs have the same name, we have to purge them regularly from the nodes, so not outdated files get downloaded at the users end. The purge we had to do also with the old approach, so no changes there. We can however vary the time slots. Currently every 10 minutes the DBs get purged.
We already have migrated the AUR DB, which we now also source directly from Arch to the new approach.
https://mirrors.manjaro.org/repo/$branch/$repo/$arch is now migrated to the new approach.
Please test via sudo pacman-mirrors -aU https://mirrors.manjaro.org/repo if it still works.
It works on my end:
$ pacman-mirrors
Pacman-mirrors version 4.24.0
Local mirror status for testing branch
Mirror #1 OK 00:05 Global https://mirrors.manjaro.org/repo/
Latest update:
opera-105.0.4970.48-1... 109.7 MiB 11.3 MiB/s 00:10 [------------------------------] 100%
linux66-6.6.7-1-x86_64 130.6 MiB 11.8 MiB/s 00:11 [------------------------------] 100%
linux61-6.1.68-1-x86_64 138.8 MiB 11.0 MiB/s 00:13 [------------------------------] 100%
mesa-1:23.1.9-1-x86_64 28.7 MiB 11.8 MiB/s 00:02 [------------------------------] 100%
vulkan-intel-1:23.1.9... 7.1 MiB 8.22 MiB/s 00:01 [------------------------------] 100%
thunderbird-115.5.2-1... 55.8 MiB 10.7 MiB/s 00:05 [------------------------------] 100%
lib32-mesa-1:23.1.9-1... 24.6 MiB 10.0 MiB/s 00:02 [------------------------------] 100%
libva-mesa-driver-1:2... 4.4 MiB 8.31 MiB/s 00:01 [------------------------------] 100%
lib32-vulkan-intel-1:... 6.6 MiB 8.78 MiB/s 00:01 [------------------------------] 100%
lib32-libva-mesa-driv... 4.2 MiB 12.4 MiB/s 00:00 [------------------------------] 100%
lib32-mesa-vdpau-1:23... 4.1 MiB 9.33 MiB/s 00:00 [------------------------------] 100%
mesa-vdpau-1:23.1.9-1... 4.1 MiB 6.80 MiB/s 00:01 [------------------------------] 100%
lib32-vulkan-radeon-1... 2.8 MiB 11.6 MiB/s 00:00 [------------------------------] 100%
xorg-server-21.1.10-1... 1770.9 KiB 10.2 MiB/s 00:00 [------------------------------] 100%
vulkan-radeon-1:23.1.... 2.7 MiB 11.7 MiB/s 00:00 [------------------------------] 100%
vulkan-swrast-1:23.1.... 2.1 MiB 8.03 MiB/s 00:00 [------------------------------] 100%
thin-provisioning-too... 996.1 KiB 10.8 MiB/s 00:00 [------------------------------] 100%
thunderbird-i18n-de-1... 637.5 KiB 7.78 MiB/s 00:00 [------------------------------] 100%
zxing-cpp-2.2.1-1-x86_64 598.8 KiB 1460 KiB/s 00:00 [------------------------------] 100%
thunderbird-i18n-en-u... 563.7 KiB 7.44 MiB/s 00:00 [------------------------------] 100%
vulkan-virtio-1:23.1.... 418.7 KiB 5.84 MiB/s 00:00 [------------------------------] 100%
languagetool-6.3.a-1-any 202.4 MiB 11.6 MiB/s 00:17 [------------------------------] 100%
linux66-virtualbox-ho... 313.9 KiB 8.29 MiB/s 00:00 [------------------------------] 100%
vulkan-mesa-layers-1:... 311.1 KiB 5.33 MiB/s 00:00 [------------------------------] 100%
linux61-virtualbox-ho... 290.8 KiB 4.06 MiB/s 00:00 [------------------------------] 100%
lib32-lcms2-2.16-1-x86_64 169.8 KiB 3.53 MiB/s 00:00 [------------------------------] 100%
lib32-libdrm-2.4.118-... 147.6 KiB 5.34 MiB/s 00:00 [------------------------------] 100%
xorg-server-common-21... 31.8 KiB 1059 KiB/s 00:00 [------------------------------] 100%
Gesamt (28/28) 734.8 MiB 41.7 MiB/s 00:18 [------------------------------] 100%