Getting started developing pacman-mirrors


#1

Greetings all!

I mentioned before that I have varying degrees of experience with languages other than Ruby, and Python is one that I’m not incredibly versed in. I’ve used it for only a couple of things: a SublimeText2 plugin a number of years ago and some Ansible tooling when I was DevOps at my current employer. It should probably come as no surprise that I’m struggling a bit to get a Python development environment setup so I can hack on pacman-mirrors.

I’m following the instructions at the bottom of https://gitlab.manjaro.org/applications/pacman-mirrors/blob/master/CONTRIBUTING.md in a VM based on manjaro-gnome-17.1.10-stable-x86_64.iso. What I’ve done:

  • Installed base OS with 4G RAM and 20G disk space
  • Updated everything, but did not switch to 4.17 kernel (caused mouse problems with VM)
  • Installed git
  • Cloned and cdd into the repo
  • sudo pacman -Syu python-pip
  • sudo pip install virtualenvwrapper
  • source /usr/bin/virtualenvwrapper.sh (else next command not found)
  • mkvirtualenv pacman-mirrors
  • python setup.py develop
  • pip install mkdocs
  • pip install tox
  • pip install coverage
  • pip install babel
  • pip install flake8
  • pip install npyscreen
  • pip install transifex-client
  • flake8 pacman-mirrors tests (nothing reported)
  • python setup.py test [one failure: test_geoip_available (tests.test_httpfn.TestHttpFn)]
  • tox (failed due to not having python3.4)
  • installed python34 from AUR
  • tox (lots of red output, ultimately “error: could not delete ‘/etc/pacman-mirrors.conf’: Permission denied”)

My first few attempts were with pyenv (I’m a big fan of rbenv, and I’ve used pyenv and ndenv with some success), but just could not get it working. I figured I’d give the official steps a go and see what happens, and after three failures, I’m still scratching my head.

Help very much appreciated.
Johann


#2

Hi Johan.

I think it is awesome what you are trying to do. I have absolutely no experience with what you are attempting to do so I’m certainly not the one to offer advice. From reading the forum though I’ve seen in many posts that it is not recommended to do installs using pip. The recommended way is to do the installs via Pacman. Using pip can cause dependency issues down the road. This often causes problems. I’m not the one to really say, but I have seen this advice on the forum many times. Hopefully someone with expience in these matters will give you first hand advice.

Again many thanks for trying to help out with this project.


#3

echo “do not install using pip”


#4

Please discuss with the team what you have in mind regarding pacman-mirrors as it is a vital component of the Manjaro ecosystem :slight_smile: and it has no outstanding issues.


There is some things to be aware of in regards to the installation things.

When you have created the virtual environment you need to change to the folder where you cloned pacman-mirrors source.

You must ensure the virtual environment is active - you can see it is by (pacman-mirrors) gets added to the prompt.

workon pacman-mirrors
(pacman-mirrors) ~ >>> 

In that source folder you run

python setup.py develop
pip install mkdocs tox coverage babel flake8 npyscreen transifex-client

This installs the needed tools into the virtual environment not your running system.

For installation of python tools in your running system, you should use pacman when it is possible.

sudo pacman -S  python-some-python-pkg

Many python packages have been build to be installed with pacman and installing a package with pip might yield trouble later on as the --force argument has been removed from pacman effectively blocking an update if files in the update already exist on the system but not installed with pacman.

EDIT: clarification --force is deprecated and a new --overwrite can be used instead.

       --overwrite <glob>
           Bypass file conflict checks and overwrite conflicting files. If the package that is about to be
           installed contains files that are already installed and match glob, this option will cause all those
           files to be overwritten. Using --overwrite will not allow overwriting a directory with a file or
           installing packages with conflicting files and directories. Multiple patterns can be specified by
           separating them with a comma. May be specified multiple times. Patterns can be negated, such that files
           matching them will not be overwritten, by prefixing them with an exclamation mark. Subsequent matches
           will override previous ones. A leading literal exclamation mark or backslash needs to be escaped.

#6

So how would you manage requirements?


#7

I think what @c00ter refers to is the recent update scenarios where pip installed packages suddenly conflicts with updates.


#8

I understand what you’re saying but as far as python packaging go you should manage your python package with a requirements.txt file that autoinstalls all the required versions of packages.


#9

The requirements.txt are for redistribution - not for setting up the development environment.

And with the exception for npyscreen, the packages in mention are only for development.


#10

I see, reproduction is a very important role in software development but as long as it’s unnecessary in your case it’s fine :slight_smile:


#11

@fhdk

When I started this adventure, there was an open issue that I wanted to take a stab at. It has now been closed. Either way, though, I want to continue getting this environment working so I know how it’s done.

I did just what you said. I was in the pacman-mirrors directory, the virtual environment was active (I saw (pacman-mirrors) before my prompt), and I used pip install ... to install the packages. I followed the instructions (linked to in the original post) exactly. Even though I did try to get things working my own way (pyenv) at first, when I failed I deferred to the published documentation and did not deviate from it. (Ultimately, I am a rule follower and will do what I’m told. :wink:) However, those instructions do not work for me. Can you (or someone else) please verify they are, in fact, correct?

After people began saying “don’t use pip”, I then tried to install all the specified packages with Add/Remove software (I am becoming familiar with pacman, but haven’t looked too closely at yaourt yet, so am not sure if the options are the same), but I got a failure at some point. I was tired last night and didn’t make note of it. I can try to reproduce if necessary.

Thanks so much for your input!

Tschüss,
Johann

EDIT: Upon reflection, I realized that I left out some of the steps that I took either because I thought they were implied in the process or because they were indicated earlier in the contribution guide. From #3 in the “Get Started” section, I understood that I needed to cd pacman-mirrors before running python setup.py develop. Well, actually, I would have known that anyway, but I’m trying to explain that I did what the guide told me to do, but it didn’t work (at least for me).

I apologize for unnecessary confusion.


#12

3 posts were merged into an existing topic: Looking to get involved in Manjaro


#14

Yes sorry about that - but I was actually working on that one :).


When you open the pacman-mirrors repo you will open in a 3.0.x-stable branch.

To get the most recent version of the code - including the CONTRIBUTING.md document - you have to select master branch.

Yesterday I rewrote the section on setting up the enviroment by changing yaourt to yay and simplified the pip installation command.

I originally created a detailed set of instructions because I needed them myself.

So I can vouch for them as they work as intended. As a maintainer and developer I often reinstall my system so I regularly need to reinstall the pieces needed for developing on pacman-mirrors.


#15

@fhdk

I’m sorry it has taken so long for me to respond. Right when I thought things were settling down, they picked back up. And the World Cup is on now :wink: … I just watched the Spain/Russia match. Wow. What a nail-biter!

Anyway, I recorded a screencast showing you exactly (heh, there’s that word again) what I’m doing and the results I’m seeing. Maybe with this you’ll be able to tell what I’m doing wrong.

Thanks for your involvment!
Johann


#16

Some of those documentation is from before my time and I think we are looking at some obsolete documentation - I have not been aware of that - I have not been that eager to change something I don’t know about.

I use Makefile and the make test - just type make to get a list of possibles.

Some of these requirements I have no idea about (flake, tox, coverage) - maybe they are from the boilerplate the project is created from.

I can see some of that documentation is crap - and those parts I have never understood. Eventully I have gotten a decent understand and skills with python - only 3.6

It is not required the project runs on 3.4 - we always has the latest anyway.


#17

Thanks @fhdk. My first PR, then, is to remove tox :wink: