Manjaro Git split-up: repo = packagesource (?)

development
manjaro
#4

It’s more convenient but not necessarily better in terms of git history - for example, it’s very difficult to see exactly when a particular package was changed, and see how it changed, if it’s in a large commit with a commit message of e.g. “Modifications”.

Normally, with git you’d split the commit up into smaller chunks that are more specific, so e.g. a per-package commit. This way it keeps related changes together in one place, and separate from unrelated changes.

Plus, you should work in a “feature branch” for each package, then commit that to master once it’s ready. That way, if you git pull and there are modifications to the repo it’s unlikely to affect your work and require a merge. If you bunch a large set into one commit you’re much more likely to hit a merge conflict.

For example:

cd packages-community/tlp
git pull
git branch tlp
git checkout tlp
vim PKGBUILD
git add PKGBUILD
git commit -S -m "[tlp] Randomly break some systems"
git checkout master
git pull
git merge tlp
git push
git branch -d tlp
0 Likes

#5

Is also possible reduce the clone space i think … time to time we can do an optimization of github repo ( don’t remember the command ); we already do for some of our repo like the old manjaro-iso profiles in the past …

0 Likes

#6

Thinking on, I wonder if a submodule setup would work. That way you have one “top-level” repo that contains sub-modules that are individually accessible, but also work as part of a larger repo.

I’ll have to have a read up on git submodules.

Submodules would work but might well end up making the repo structure far more complicated.

https://git-scm.com/book/en/v2/Git-Tools-Submodules

0 Likes

#7

Well, SVN supports to work only with one package by using for example svn up <pkgname>. Arch still uses SVN for that particular purpose and simply convert it to GIT. However, we may ask the guys from frugalware on how they manage it.

0 Likes

#8

Looking at their commit log (with ~250K commits…) it looks like they follow the one-package-per-commit workflow.

0 Likes

#9

Seems so. Check also their documentation here. They mostly use a tool called repoman.

0 Likes

#10

Another option could be to set up an aurweb for PKGBUILD management, then use GitHub with submodules to collect each repo’s packages in one place. That way we also get per-package comments and out-of-date notification support.

0 Likes

#11

You can trigger us all via @Manjaro-Team

1 Like

#12

Well, this would be a project to accomplish first. I’m open for it, if it makes things easier.

0 Likes

#13

But wouldnt that split the userbase to “who know the manjaro aurweb” and “who finds the github”?
For me one of the top priorities would be to make is easier to contribute / work with manjaros development environment. So it would be nice to have one contact point to interact with.
For aurweb, would it import changes made by users on github / export changes made in aurweb to github? :thinking:

Right now I only know https://github.com/flathub/ wich follows the 1 repo for 1 package, that works great in case of bugfixing/packaging/finding software

0 Likes

#14

If you read the submodule documentation you’ll see how it works. Essentially, the “big” repo which adds a submodule makes what’s essentially a symlink to the “small” package repo, and then treats it as a part of the current repo. The “small” package repo operate independently.

Pushes to the “big” repo are actually pushed to the submodule repo (like following a symlink).

Pushes to the package repo would have to be pulled in manually by anyone with a clone of the “big” repo (e.g. git submodule update).

However, this does make things much more complicated than necessary, but would still be useful if packages are split into separate repos within GitHub (i.e. you have a core repo which adds each of the core packages as submodules).

1 Like

#15

Finally found time to play a bit with this. But I dont see the usecase for this. I mean why do we even need a “big” repo?
From what I see we have following things to deal with:

  • Repo for Code
  • Repo for PKGBUILD

So why not do that? 2 “sub” Projects: Manjaro Code(MC), Manjaro PKGBUILD. Given that many people host the code outside of Manjaros infrastructure anyways, the Code repo would be minimal.

The PKGBUILD repo would only contain PKGBUILD and install files, no patches or other things, to keep it clean. So in case of a buildbot, you could easily crawl for changes inside the pkgbuild repo.

So for example:

Steam-manjaro

PKGBUILD:
would be in https://github.com/manjaro/steam-manjaro/blob/master/PKGBUILD

pkgname=steam-manjaro
_package=steam
pkgver=1.0.0.54
...
source=(
"https://github.com/manjarocode/${pkgname}/archive/${pkgver}.zip"
"http://repo.steampowered.com/${_package}/pool/${_package}/s/${_package}/${_package}_${pkgver}.tar.gz"
)
...

Pro:

  • we can create “in-development” packages, based on the same sources, that is not possible right now
  • we can (or have to) work with tags, for releases. So no checksum=('SKIP') anymore
  • devs can change the code in MC:master if they want to without to screw up the pkgbuild
  • it’s easier to merge pull requests for single packages
  • if done right, we could export a changelog for pamac
  • we could hve a lot more devs, because you can set different collaborators for each repo=package (wich is a real burden right now, to wait that your own stuff is pulled to the git repo, but it sucks to work outside because of constant new code and conflicts)

Con:

  • it would take some time to set that up
  • Poeple would have to work with 2 Repos(Code and BUILD)
  • Issuetracking could be a bit confusing for users and devs
1 Like

#16

I think you search for SVN. This application supports to checkout non and decide which exact package you want to have downloaded from the repo. I’ve to check if GIT supports a similar feature.

0 Likes

#17

I dont think its supports that other than submodules wich is kind of overkill and would not solve a thing.

Actually the 2 Repo-> Code & PKGBUILD feels like a smooth way to go :thinking: Also for transitioning to this, we could do a first step and look what software we have with its full code in the repo and rebuild the package. So everything would stay cool and working until the final split up. :thinking:

I dont think going to SVN (or SVN2Git) is something we should consider.

0 Likes

#18

We have had similar thoughts. The problem, arch use subversion one package in one repo.
To achieve something similar in git, there are submodules, and there is a 3rd party git subrepo which is basically more suitable for the task. We are are still with all packages in a git repo, since a jenkins pipeline don’t work out so well with subrepos, and major block on the road, the GH api is ugly, not easily accessible from shell, 3rd party prog there too, plus, you can’t remove repos easily on GH from shell. Needs admin confirmation, and at best, you have a package org on GH for the package repos.
However, it may be viable for manjaro to consider subrepos, its different needs for you.
I wouldn’t recommend submodules for the task.

0 Likes

#19

Well, we may move to GitLab, which supports those kind of things better. For example you can see the Gnome Project. You can create a project called Packages and place the repos to it. Inside of the repos you may add for each package one package repo. The structure will be the same. Also the issue tracker is for the whole project. However, this needs testing.

2 Likes

GitLab @ Manjaro
#20

Is there a roadmap to gitlab? (I’d help with this)

0 Likes

#21

There is no roadmap yet. We still have to decide what the benefit would be and what is needed to do so. If we do so, it would implement an buildserver and other things.

Let me comment on some of your requests:

We use the same structure of repos as Arch does. To find a package you simply use pacman and search for it. Then you find the repo it is included. Having one big repo for all PKGBUILDs is the way git works without subprojects. Splitting it up per package I don’t see the benefit yet.

In the end you would change the complete workflow as we have now. For the applications we have separate repos. The packages are splitted by package repos but stored as we have them now. Currently everybody works with the current way we do packaging.

But lets see if there is a better way to improve all of this.

0 Likes

#22

I’ve created now a new gitlab instance. All tools got migrated except packages.

2 Likes

#23

I started now with core-packages

2 Likes