Autogit - Bash tool to automatically build & update PKGBUILD's from Github & Gitlab

Contributions Software
, ,
#1

Autogit is a Bash tool to automatically build and update Pacman applications from PKGBUILD’s available on Github and Gitlab via configurable Makepkg or Manjaro Buildpkg commands. It can also create automatically repo names like core , extra , community and create/update a repo database file via repo-add . Additionally it clears each local repo with paccache -v -r -k 1 -c to only keep the latest package version.

It checks for the pkgver of the PKGBUILD to update applications and can either run once or in a loop with a timer setting.

More information on how to set it up is available in autogit.conf, examples are included.

Currently 2 Github and 3 Gitlab sources are available to use. There is no limitation of packages for each source, which need to be added to the reponames folder.

It can currently create 2 different Github repo folders and 3 different Gitlab repo folders.

How to install:

  • sudo pacman -Syu base-devel git
  • git clone https://github.com/puxplaying/autogit.git
  • cd autogit
  • makepkg -srci

How to run:

  • autogit

To run it automatically without a password prompt for buildpkg , it can be added to sudoers via:

echo "$USER ALL = NOPASSWD: /usr/bin/buildpkg" | sudo tee /etc/sudoers.d/$USER

This little tool is still in early development and needs a lot of testing, any feedback to improve it is welcome.

14 Likes
Tools for building Arch repos?
#2

Nice job, thanks for posting :+1: , something to look at @Manjaro-Team

2 Likes
#3

It looks quite useful.

I would try breaking that code into the smallest functions. It would be the easiest to read and to test.

1 Like
#4

Added check for pkgrel in addition to pkgver to update applications and minor makepkg fixes for all archive types (gz, zst …etc).

2 Likes
#6

Autogit 0.25 was released and is considered to have stable core functionality. I’m currently running it with buildpkg 24/7 on 3 Manjaro Gitlab repos (core, community, extra) with 15 different packages plus my own Github and it seems to do it’s job fine. As always feedback is welcome, happy building! :nerd_face:

2 Likes
#7

I assume you don’t mind this going to the community repo?

#8

Absolutely not, I’m glad if it’s helpful for you guys!
Thanks to @Yochanan i’ve changed some things around buildpkg, so that it now shouldn’t matter for which architecture one is building. As long as it’s set in the right place of the config the binary’s will be moved to the preset repo.
Next step is to implement up to 6 repo’s for Gitlab and Github and add a error handling in case the online source is not available.

Please let me know if there is still something missing or should be implemented!

2 Likes
#9

A new release is out with added check for server availability and support for 6 Gitlab repos, which are preset to all available Manjaro repos in the config file and reponames folder.
In theory one could auto-build all manjaro software into separate local repos and repoadd should do the rest. At least in my tests with a added local repository to pacman.conf it works pretty well and the apps get updated through pacman when i run a system update.

2 Likes
#10

Uploaded to unstable

1 Like
#11

0.29 is out with a reworked server-check which should hopefully work reliable.

#12

Updated in unstable.

1 Like
#13

0.30 is out with a fix for the makepkg installation option, support for 6 Github repos and AUR support.
Happy building! :nerd_face:

1 Like
#14

New version is out with a massive cleaning and improvements.

  • removed duplicate code where possible
  • repoadd now only will be triggered for the specific repo when a update or new build has been done - if enabled. This should be much more resource friendly and faster if a lot repos are enabled.
  • the info output was reworked
  • other small changes and fixes

example output:

autogit-0.40
autogit-0.40786×601 49 KB

Enjoy!

Ahem… I forgot to ping @Chrysostomus :stuck_out_tongue_winking_eye:

2 Likes
#15

Thanks! Rebuild pending tonight

EDIT: uploaded to unstable. Thanks again!

2 Likes
#16

The New Year release is out with some eye-candy!

  • Improved visual output
  • More Information about the last run
  • Suppressed unwanted “mv” messages when compiling

Bildschirmfoto vom 2020-12-31 08-06-51
Bildschirmfoto vom 2020-12-31 08-06-51786×701 71.5 KB
Bildschirmfoto vom 2020-12-31 07-42-03
Bildschirmfoto vom 2020-12-31 07-42-03786×601 47.6 KB

Happy New Year! :fireworks:

@Chrysostomus, mea culpa forgot to ping you :see_no_evil:

1 Like
#17

Thanks for the feedback, I think your suggestion is the current state right now after implementing all the wanted functionality and a big cleaning.
I still need to improve on my comments, but the code should be in readable state - at least I hope so. :wink:

From my point of view this tool is currently feature complete as it does what it was meant to do.
Still there is one downside, which it can’t handle like a AUR helper would do for example. Building git packages is not really something it is made for, although it can do it. But as it checks for the pkgver and pkgrel it will always rebuild git packages.

#18

Thank you for sharing!

Questions that may help:

  • For avoiding rebuilds, won’t be enough checking the package version of already built packages?

  • Why exactly 2 github and 3 gitlab sources? Why not an arbitrary amount?

  • Which code repeats in different places? What if that code was a single function or variable by itself?

  • In the configuration file, which variables could be automatically inferred or have a default when not explicitly typed?

  • What if the sources where expressed in one line, like in PKGBUILDS?

  • Should be the AUR url be configurable by the user? Furthermore, could the AUR just be treated as any other source?

  • Which comments or instructions are self explained by the code or context, so removing them really has no consequence?

  • Could a function be expressed in 4-8 lines? Would this program code be easier to find if split into different files? Example here.

  • Pro level: Could these sources be built using multiple machines in parallel?

And some inspiration. Have a nice day :+1:

1 Like
#19
  • This case only applies to git packages which have no “normal” versioning. There is currently no way to check the latest online version, as the online PKGBUILD refers to a older version compared to the local one. Checking the latest built package might be an indicator, but won’t solve the problem as it needs to know when a package is out of date. I will think about how to solve that case with the current approach.
  • This is not the current state anymore, right now it supports 6 Gitlab, 6 Github and the AUR source.
  • The repeating code is one single function with variables in itself to allow different parameter for all the different sources.
  • The most things have a default.
  • The Servercheck is by default off and needs to be un-commented for each server, as i have no idea who uses what.
  • the default settings for “buildpkg” and “makepkg” can either be used as they are, or changed to user preference.
  • After that the default build method shoud be set to the preferred one.
  • repo-add is by default off
  • Finally un-comment the first variable of the preferred source(s) and change the binary folder name to own preferences, that should be all for Github and AUR. If you use Gitlab you also should check the URL as it is preset to Manjaro’s Gitlab. The source filenames can also be changed to user preference in the settings, but also need to be renamed in the reponames folder.
  • Basically everything that is in the settings, can be changed for each source.
  • The sources files in the “reponames” folder are interpreted line by line - for each application one line. Expressing them in one line would lead to a whole new complexity in order to read them accurate. Also as this are all independent packages which can have different Github sources etc., it would be hard to keep an overview and to not make mistakes when adding/deleting packages.

Yes and yes :wink:
The AUR can be seen as any other source. The URL itself shouldn’t normally be touched as long as it doesn’t change.

That is currently not possible other than manually splitting sources to different machines, but a nice idea that i will think about.

Thanks for the feedback!

#20

What I do is using the following for generating pkvers. Call me heretic.