Speed up installations from AUR (Speed up xz)


#1

Edit /etc/environment add line:

XZ_OPT="--threads=0"

Reboot.

In my limited tests on my core i5 (two cores/4 threads) firefox-dev installed in 1:48 seconds, with the line 1:01 seconds - saving 47 seconds.

What it does - tells xz to use all available cores/threads.


#2

Another way is to edit /etc/makepkg.conf and modify the line

COMPRESSXZ=(xz -c -z -)

like this

COMPRESSXZ=(xz -c -z --threads=0 -)

also if the goal is speed you might want to add a lower compression preset like -0 … -3. The default is -6.

COMPRESSXZ=(xz -c -z --threads=0 -3 -)

But keep in mind, a lower compression preset result in a lager package file.


#3

Like it, will test after work. 2TB hd so lots of space.


#4

With xz/lzma2, more threads also means (much) more RAM usage while compressing, and lower overall compression ratio.


#5

Like 6MB/200MB more RAM and 10KB/300MB bigger files?
Because that’s the result of my testing with 7-zip and going from 1 to 2 cores.


#6

I don’t know, never tested, just something I’ve picked up somewhere (probably 7-zip on Windows).
But perhaps useful info for users of old computers.
I’ve settled for 6 threads for xz (i.e. half of the available threads).


#7

If you don’t need maximum compression, try a different archive format.

.tar is uncompressed (and so much faster) and so good for AUR package installation.
.lz is a good middle-ground.

All you need to do is change the PKGEXT variable in /etc/makepkg.conf.


#8

I did some more tests with xz. I used the google-chrome package. I did that on a Intel Core i5-4670 CPU and all files were stored on a HDD. My System has 16 GB of RAM. On systems with less memory, xz should also use less memory there. It is not really scientific since I run all commands only once, so there is no Error analysis.

The google-chrome.pkg.tar file has a size of 273489920 byte.

Options Filesize (byte) Filesize (%) Memory (kbytes)  Elapsed time (m:ss)
-0      80925944        29.59        5156             0:13.28
-1      76392572        27.93        11068            0:17.25
-2      74742432        27.33        18688            0:24.70
-3      73795824        26.98        34220            0:34.91
-4      70223428        25.68        50672            0:59.99
-5      67601016        24.72        97892            1:18.29
-6      67135752        24.55        97716            1:27.21

and with all threads enabled

Options Filesize (byte) Filesize (%) Memory (kbytes)  Elapsed time (m:ss)
-0 -T 0 81608184        29.84        26316            0:03.86
-1 -T 0 76903792        28.12        71872            0:05.89
-2 -T 0 75199528        27.50        127432           0:08.11
-3 -T 0 74148868        27.11        226572           0:11.86
-4 -T 0 70653500        25.83        290324           0:18.67
-5 -T 0 67943584        24.84        549004           0:25.09
-6 -T 0 67567832        24.71        552440           0:27.11 

the complete output of all commands
https://pastebin.com/MgWw0QAr


#9

Thanks for the explanation.

Can you please give me a hint what to use, when i want not to use all cores with “–threads=0” - but more than one?

Afaik compression can set from 0 to -9 - higher number means higher compression.
How is the influence on this if i use a SSD with Btrfs mounted with the options
defaults,noatime,space_cache,autodefrag,compress=lzo ?
There is enough space left to use the 2, 3 packages i built from AUR.

Thanks in advance


#10

-threads=N where N is the number of threads you want.


#11

That is true, but it is of course slower. And the goal of this threat is faster not better compression.

I/O is not the bottleneck of xz on higher presets, your CPU is it. And it might even be slower, because the filesystem is compressed, too.
The same computer, but with little more going on. So it is not really comparable to the other test.
Option -6 same tar file in RAM 1:29.22 and on my SSD 1:31.80 and again HDD 1:32.70. I would say it is nearly the same.

If you build aur packages, first try building in RAM. Normally you can use your /tmp, which is mounted with tmpfs.
If you only want to install these packages and don’t want to save or use these packages on other computer, don’t compress at all.

@jonathon mentioned a way in this post.

When space is the issue and you have time, compress with higher compression presets. But keep the memory use in mind. With -6 and T 4, on my computer xz uses around 550 MB of RAM.