Pamac fails to synchronise due unacceptable TLS certificate

Any updates on this? I started to receive the same error today, when updating from terminal. curl and firefox work fine though

Just wait and try later after a few hours.

I’ve never seen the error in the last 2 weeks.

[alex@alex-b450aoruselite ~]$ pamac upgrade --force-refresh
Preparing...
Sincronización de bases de datos de paquetes...
Actualizando core.db...                                                                                                                                               
Actualizando extra.db...                                                                                                                                              
Actualizando community.db...                                                                                                                                          
Actualizando multilib.db...                                                                                                                                           
Actualizando repo-ck.db...                                                                                                                                            
Actualizando core.files...                                                                                                                                            
Actualizando extra.files...                                                                                                                                           
Actualizando community.files...                                                                                                                                       
Actualizando multilib.files...                                                                                                                                        
Actualizando repo-ck.files...                                                                                                                                         
https://aur.manjaro.org/packages-meta-ext-v1.json.gz: Unacceptable TLS certificate                                                                                    
Failed to synchronize AUR database
Nothing to do.
Transaction successfully finished.

It is only manjaro aur

Yes, I know, but I think there is some pamac bug here that is not being addressed. But, who knows…

1 Like

It’s just the CDN that is delivering a error page with a bad SSL certificate, when there are too many requests, if you ask me…

There are too many requests for those AUR databases. Here the last 24h of traffic you guys generate just for that:

image

Remember that those DBs are just 8 MB in size.

Here a traffic snapshot for the last 30 days just for AUR DB files:

image

Certificates are all valid:

AUR, Download and Mirrors use the same as our main homepage.

So why do I get those errors?

CDN works like this:

  • we have one storage server which gets every 10 Minutes a new DB file from the Arch server
  • we purge the files every 15 mins from all CDN nodes
  • CDN nodes will fetch the updated DB files from the storage server and cache them again

If you hit a node which is in progress to purge the file you hit an error page. pamac should in that case ignore the error and try to fetch that file again. I assume if you retry to fetch the file you won’t have that error. You only wonder why you have that error and complain.

2 Likes

The certificates of the domains the CDN serves might be all correct, but how about the certificate of the default error pages of the CDN itself?
Because if there are no wrong certificates at all, why the error message about the certificate then?
“Simple logical deduction dedective Holmes” :wink:

We don’t maintain the error page nor that cert. If you fetch the error page in source I can ask the cdn provider.

Incomplete/in-progress synchronization.
Or that’s my guess anyway.

That was exactly my idea also…
But unfortunatly, im not on Manjaro anylonger…
You can still ask them to check theirs tobe sure.

I documented it more in the issue tracker. Failed to synchronize AUR database (#1305) · Issues · Applications / pamac · GitLab

not so easy if you have to do it manually. I can check if their API supports something so we can use cron jobs from our servers and upload them that way, else it is a paid service to let them maintain the CERTs. However, that is not the problem here. More or less our strict domain rules don’t accept other CERTs from the CDN provider not matching our domain when a user hits an error page when a node is in purge mode.

This updating seems like it’s a solved problem within CDNs. The updates should be atomic: serve the old file as long as the new one isn’t completely available.

Well I only assume that the non existing file might lead to an error page. CDNs are designed for files which have an unique filename and to be cached at the nodes. However since DB files never change the names but their content we have to purge them on a regular basis. Even just a millisecond might create the error.

There is also a way to pre-fetch files from the file server if they are important. However due to the high requests the CDN explicitly told us to stop pre-fetching and let the system fetch when needed, otherwise our requests DDOS the nodes. That already happend many times with the Arch AUR server in the past due to the huge demand of Manjaro users fetching that file.

Arch is updating the file every 5 minutes, we fetch it every 10 minutes and redistribute it thru the CDN network. Even there we create a lot of traffic as already documented. So I can check if there is a solution on backend side or pamac has to jump thru hoops.

That’s why people use a “create a temporary file and rename over old one” approach in those cases… :stuck_out_tongue_winking_eye:
(But yea true it’s a lot harder todo that on a CDN distribution from the mirror sides i guess)

Wont work as you have to purge the cached file to get the updated on. The file server works as a regular storage system. There we have no issue to distribute the normal files via regular rsync. This is the regular way as we also do with normal mirrors. What ever happens within the CDN network is out of our control.

  • We provide the files to the storage via rsync
  • We keep the CERTs up-to-date
  • We maintain the cron jobs on our Servers to instruct the API to purge DB files and other files as needed which have the same name in the server structure

All the magic with quirks happens on the CDN side which we can trigger with API cmds.

1 Like

Just trying to help, keep that in mind :wink:

Nice rsync should already be using the rename over old file technique :+1:
Do you use the --delay-updates option? if not try that to see if it makes a difference. :woman_shrugging:
(If you on your end already do use that option, check with the CDN if they do too)

I’m just wondering what you actually mean by that:
Are you removing the DB-files before you upload a new version?
If so why??

Maybe switch to systemd-timers and stop the timer/service while your side sets up things and re-start it afterwards when you are done.
That way the job in the cron entry won’t have any chance to fire in-between your work.
Just like inhibiting the sleep on a system :wink:


Again im just trying to help and find-out where things go wrong :v:

Well, the issue is this: If the file doesn’t change its name, like AUR DBs, you have to purge them from the CDN nodes so they cache the new files. Otherwise the nodes will keep the filename as nothing changed for them. After chatting with the support a little we gained the following information:

  • they will never show an error page for a file that’s being purged, instead CDN will either serve the old version or fetch the new one from the origin if the purge is done. There are no more than two outcomes in this situation
  • the error page is only triggered when the file is not cached and the origin returns 404 when fetching the file
  • 404 responses are returned through the CNAME used and with our own certificate we also use for manjaro.org

They are pretty confident whatever happened has nothing to do with error pages.That leave us with changes on the SSL transition, so we check any recent changes to the SNI …

Seraching the forums for “https://aur.manjaro.org/packages-meta-ext-v1.json.gz” returned several threads. I see the tag “unstable” is applied here, I am on Manjaro stable (KDE).

Error messages thrown by pamac upgrade --force-refresh are not consistent, results vary (at least) between Socket I/O timed out and Unacceptable TLS certificate.

A direct download of the package via browser is fast and fine, no issues or warnings.

Checking the SSL certificate (SSL Server Test: aur.manjaro.org (Powered by Qualys SSL Labs)) takes a long time, more than 85 sec.

Hope this info can help with finding the cause.

There might be some issue with libsoup3, which got introduced with Gnome 43 and sha256 CERTs as @codesardine pointed out once. Similar to this one: PBB radio: Handle SSL certificate error with untrusted certificate (#128) · Issues · Goodvibes / Goodvibes · GitLab