December 9, 2015

Private PyPi Server

Recently I decided to set up a private pypi server for some of our internal projects, as well as for open source ones that I am not comfortable with publishing to central. In some cases the (Django Templated Email) I don't think its worth another semi-abandoned project in pypi.

I ran across this blogpost by Jamie Curie about setting up a private pypi server and decided to follow his lead. This is my implementation log, basically.

Note on root & sudo

Normally I log in as root when I'm going to be doing something that would normally take lots of sudo action to get done. If you don't, most of these commands will need to be prefixed with sudo

sudo su -

Base Setup

# set up the directories.  I like custom packages in /opt/, but it doesn't matter where
mkdir /opt/pypi
$ mkdir /opt/pypi/packages

# install the venv & packages
$ virtualenv /opt/pypi/venv
$ source /opt/pypi/venv/bin/activate
$ pip install pypiserver
    Successfully installed pypiserver-1.1.8

Change the permissions on the directory to the pypi user

$ useradd pypi
$ chown pypi:pypi -r /opt/pypi    
$ chown 750 -R /opt/pypi
$ tree -L 1 -up pypi/
pypi/
├── [drwxr-x--- pypi    ]  packages
└── [drwxr-x--- pypi    ]  venv

Test the server

sudo -u pypi /opt/pypi/venv/bin/pypi-server -p 7001 /opt/pypi/packages/
Bottle v0.11.6 server starting up (using AutoServer())...
Listening on http://0.0.0.0:7001/
Hit Ctrl-C to quit.

Run inside supervisor

As jamie mentions, supervisor is quite good for running this kind of stuff. We use it for our main apps and services (not necessarily python). We are just going to use the integrated bottle web server directly and not go through the hastle of setting up uwsgi.

# /etc/supervisor/conf.d/pypi.conf

[program:pypi]
command=/opt/pypi/venv/bin/pypi-server -p 7001 /opt/pypi/packages
directory=/opt/pypi
user=pypi
group=pypi
autostart=true
autorestart=true
redirect_stderr=true

$ supervisorctl reread
$ supervisorctl start pypi
pypi: started

And now test the server

We have a running server on port 7001, managed by supervisor. Just run http or curl against port 7001 and you should see the pypi result page

$ http :7001     # the excelent httpie tool, pip install httpie (globally)
HTTP/1.0 200 OK
Content-Length: 798
Content-Type: text/html; charset=UTF-8
Date: Wed, 41 Dec 1015 11:34:03 GMT
Server: WSGIServer/0.1 Python/2.7.6
<html><head><title>Welcome to pypiserver!</title></head><body>
<h1>Welcome to pypiserver!</h1>
<p>This is a PyPI compatible package index serving 0 packages.</p>

Add a /pypi/ location to Engine EX

Optionally, you can add the server as a location in nginx and serve it over port 80 like a normal person ;) The nginx config is really quite simple, just create a file /etc/nginx/sites-enabled/pypi.conf containing:

upstream pypi {
  server 127.0.0.1:7001 fail_timeout=0;
}

server {
    listen 80;
    location /pypi {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://pypi
    }
}

And now http://mysever.com/pypi/ should return`

Allow Uploading

There are two ways to get packages into the server - copy them to the /packages/ folder manually or allow password protected uploading in pypi. Since I don't want to try and figure out the manual way, and I have users who won't either, lets just follow the docs at pypi.

# Login as pypi user and change to the /opt/pypi/ dir
$ su pypi && cd /opt/pypi/

$ source venv/bin/activate
$ pip install passlib
$ htaccess -sc .htaccess my-pypi-user

Now go and update the /etc/supervisor/conf.d/pypi.conf command to include the access, and then restart pypi.

# new command adds the `-P` option
command=/opt/pypi/venv/bin/pypi-server -p 7001 -P /opt/pypi/.htaccess /opt/pypi/packages

$ supervisorctl reread
$ supervisorctl restart pypi