Using distcc to speed up AUR builds, include server gave up analyzing

Support AUR
1

Hello,

I have a laptop and a desktop both using Manjaro. My systems have a few AUR packages, and as you know many of them build packages from source much like Gentoo. My laptop is pretty old (2017 i7) with a dual core processor, and certain packages take a very long time to build. Updates in particular are very time-consuming. So I decided to try using distcc to try and speed things up by utilizing my desktop with its 7800X3D CPU which has 16 cores.

After setting things up in pump mode, it does seem to work, but I’m getting the following warning repeatedly

distcc[64283] (dcc_talk_to_include_server) Warning: include server gave up analyzing
distcc[64283] (dcc_build_somewhere) Warning: failed to get includes from include server, preprocessing locally

This indicates to me that pump mode and preprocessing is not happening properly on the remote server and much of the work is happening on the local machine. I’m not sure if this is because of misconfiguration, or if it’s because of the package I’m trying to build. As a result I feel that I’m not getting a proper speedup.

This is an excerpt of my laptop’s makepkg.conf

BUILDENV=(distcc color !ccache check !sign)
DISTCC_HOSTS="localhost/4 9a3eedi-linux/16,cpp,lzo"
MAKEFLAGS="-j32"

Note I’m using a hostname here, but I made sure that it’s resolvable.

On my desktop (the volunteer) I have the following in /etc/conf.d/distccd

DISTCC_ARGS="--allow 127.0.0.1 --allow-private --log-file /tmp/distccd.log --jobs 32"

Looking at distccmon-text on my laptop (the client), it seems that most of the compilation work is happening on my local machine (the laptop). Some work is happening on the remote machine, but it’s utilizing only like 3-4 of the 16 cores that it can handle. I can also see on the remote machine that cc1plus processes are running, but I expected around 16 processes running at a time, but I only see around 4 or less.

Unforutnately I really couldn’t find much information online on how to diagnose this problem, particularly with Manjaro and pamac, so I was wondering if anyone else had experience with using distcc with Manjaro.

2

The bandwidth of the network limits the amount of processes as well. You can check the timestamps by setting DISTCC_VERBOSE=1.

3

I don’t think the network bandwidth is the issue here, I’m on a local area network. My laptop is on wifi with a good signal and can typically do 20-30MB/s of bandwidth. My desktop is on ethernet. Since we’re dealing with small text files, I would say latency might be a problem, but my latency is around 2ms.
EDIT: after performing the below changes, and increasing efficiency, I did notice that network usage increased significantly to the point where it reaches my laptop’s wifi limits. So you’re right there! But IMO it’s not the core of the problem here… it seems it’s pump mode that’s not working properly.

Anyway, I tried playing around with the config, and I set the following for DISTCC_HOSTS

DISTCC_HOSTS="9a3eedi-linux/32"

This effectively means that my laptop will not do any compilation by default, the remote computer will not do any preprocessing, and is only dedicated to compiling. I also specified 32 jobs.

Now what I’m seeing is that my laptop is 100% busy with preprocesisng while there are much more parallel jobs on my desktop. The load is distributed better, and the compilation runs much faster, but it’s definitely bottlenecked by the preprocessing on my laptop.

I tried hard to get the remote computer to do preprocessing, but I simply could not figure it out, I keep getting the same error I mentioned earlier. Any tips are welcome.

4

Why not build AUR Packages on your Desktop? You know which packages you need and which versions, after the build, use pacman -U to install the packages. It can be an URL or a path to a share file system or you copy the files to your laptop. You can also create your own local repository for packages you need and install them from there.

5

Actually, I’m just trying to update my AUR installed packages with pamac. To do that, I run pamac update --aur. So it’s one command.

Doing the compilation steps on the desktop then copying teh resulting package can work, but doing it in an automated way is tedious, and I’m lazy… but perhaps I can write a script that does that for me :wink:

6

Have you confirmed / did you confirm
that distcc actually works?

I last used that when I was using Gentoo, some decades ago…

With todays hardware the difference in speed might be hard to discern.

7

It does work… sometimes. You’re right that with today’s hardware, it might not be practical… but I’m trying to get this to work on an 8 year old laptop to take advantage of my 0 year old desktop. I probably wouldn’t try to do the opposite.

I dont have numbers, but it does feel like there’s a speed up particularly for C+±based packages. But it’s not as significant as I would have hoped.

I’m starting to think that this is more trouble than it’s worth.

8

… very likely

Is not an answer (to yourself, actually) I’d just accept and run with …

1 Like
9

Thanks for the reality check. Well it does seem that it’s not worse than local compilation, and can be better in some cases, so I’ll keep it enabled.

I feel that if I can get pump mode to work reliably, there will definitely be a speed bump