Monday, May 24, 2010

From Python package to Ubuntu package in 3-ish easy steps

My friend Tim is working on a very cool Bazaar-backed wiki project and he asked me to package it up for Ubuntu. I'm getting pretty good at packaging Python projects, but I always like the practice because each time it gets a little smoother. This one I managed to package in about 10 minutes so I thought I'd outline the very easy process.

First of all, you want to have a good setup.py, and if you like to cargo cult, you can start with this one. I highly recommend using Distribute instead of setuptools, and in fact the former is what Ubuntu gives you by default. I really like adding the distribute_setup.py which gives you nice features like being able to do python setup.py test and many other things. See lines 18 and 19 in the above referenced setup.py file.

The next thing you'll want is Andrew Straw's fine stdeb package, which you can get on Ubuntu with sudo apt-get install python-stdeb. This package is going to bootstrap your debian/ directory from your setup.py file. It's not perfectly suited to the task (yet, Andrew assures me :), but we can make it work!

These days, I host all of my packages in Bazaar on Launchpad, which is going to make some of the following steps really easy. If you use a different hosting site or a different version control system, you will have to build your Ubuntu package using more traditional means. That's okay, once you have your debian/ directory, it'll be fairly easy (but not as easy as described here ). If you do use Bazaar, you'll just want to make sure you have the bzr-builddeb. Just do sudo apt-get install bzr-builddeb on Ubuntu and you should get everything you need.

Okay, so now you have the requisite packages, and a setup.py, let's build us a deb and upload it to our personal package archive so everyone on Debian and Ubuntu can easily try it out.

First, let's create the debian directory. Here's the first little icky bit:

% python setup.py --command-packages=stdeb.command sdist_dsc

Notice that this leaves us with a deb_dist/ directory, not the debian/ directory we want. The latter is in there, just buried a bit. Let's dig it out:

% mv deb_dist/wikkid-0.1/debian .
% rm -rf deb_dist
% bzr add debian
% bzr commit -m'Debianize'

Note that "wikkid-0.1" will be replaced by the name of your package. In order to build the .deb package, you need an "orig.tar.gz" file. Packaging sort of assumes that you've got an original upstream tarball somewhere and you're just adding the necessary Debian goo to package the thing. In this case, we don't have an upstream tarball, although we could easily create one, and upload it to the Cheeseshop or Launchpad or wherever. However, that just slows us down so let's skip that for now! (Aside: if you do have an upstream tarball somewhere, you'll want to add a debian/watch which points to it; that'll eliminate the need to do the next step, by downloading the tarball instead).

Let's create the tarball right now and copy it to where the following step will expect it:

% python setup.py sdist
% mv dist/Wikkid-0.1.tar.gz ../wikkid_0.1.orig.tar.gz

Here's the second icky bit. Building a Debian source package imposes a very specific naming convention on the tarball. Wikkid's setup.py happens to build a tarball with an incompatible name, while the sdist command leaves it in a place where the next step can't find it. The rename just gets everything into the proper place. YMMV.

Now we can build the Debian source package. It's the source package that we'll upload to our Launchpad PPA. Launchpad will then automatically (if we've done everything right) build the binary package from the uploaded source package, from which Ubuntu and Debian users can easily install.

Oops! Before we do this, please edit your debian/changelog file and change unstable to lucid. You should also change the version number by adding a ~ppa1 to the end of it. Yeah, more ickiness.

Alright now we're ready to build our source package:

% bzr bd -S

Now let's upload it (assuming you've enabled a PPA):

% cd ..
% dput ppa:barry/python wikkid_0.1-1~ppa1_source.changes

That's it! If you've done everything successfully, you'll have the package in your PPA in 5 minutes or so. Then anybody who's added your PPA can just apt-get install wikkid (or whatever your package is called).

I do hope to work with the appropriate developers to make some of the ickiness go away. Please do contact me if you want to help!

Addendum (2010-06-10)

Let's say you publish your tarball on the Cheeseshop or Launchpad, and you don't want to have to build a different tarball locally in order to package it. Here's what I think works:

Create a debian/watch file that points to the download location you publish to. If your package is not yet available in Debian or Ubuntu, then use this command to build your source package:

bzr bd -S -- -sa

The bit at the end tells the Debian packaging primitives to include your tarball when your source package is uploaded. The debian/watch file is used to download your published tarball and automatically renamed to the required .orig.tar.gz name. When you dput your package, your tarball will be uploaded too, and everything should build properly.

Oh, and don't forget to look carefully at the lintian output. Try to make this as clean as possible. The Debian and Ubuntu packaging guides can help here.

Addendum 2 (2010-06-10)

Andrew Straw has added a debianize command to his stdeb package, which makes things much nicer. With this you can create the debian/ directory right next to your setup.py. AFAIK, this version of stdeb isn't released yet, so you need to install his git head in a virtualenv, and it has a few minor buglets, but it does seem like the best-of-breed solution. I'll post another article with a more detailed follow up later.

8 comments:

  1. Thanks for posting this. I would like to see either python-stddeb or python-mkdebian become blessed, and the other one go away. Pitti worked on python-mkdebian, I think it is in the python-distutils-extra package, and I had been making improvements there, but if python-stddeb is the future it would be nice to consolidate.

    ReplyDelete
  2. Wow, I never even knew about python-mkdebian. It's pretty well hidden (no manpage, even kind of hidden in the source branch). Thanks for the pointer, I will have a look at it. I definitely agree there should be OOWTDI. :)

    ReplyDelete
  3. Elliot, fwiw, python-mkdebian has problems with namespace packages (or it might be a more general problem):

    https://bugs.edge.launchpad.net/python-distutils-extra/+bug/592414

    ReplyDelete
  4. See addendum 2 above. stdeb seems like the answer. What is missing from it that python-mkdebian gives you?

    ReplyDelete
  5. I have just released stdeb 0.6.0 which has the debianize command. Thanks for the testing, feedback, and ideas.

    ReplyDelete
  6. See also 'bzr dh-make' in newer versions of the bzr-builddeb plugin. 'bzr merge-upstream' is also a useful command.

    ReplyDelete
  7. I think is an OT :) but... take a look on quicky!
    https://wiki.ubuntu.com/Quickly

    Cheers!

    ReplyDelete
  8. Hi, great post - helped me a lot! I got stuck near the end, though. How do you properly include icons in the Ubuntu package?

    I added the files as package_data in my setup.py (googling told me that was the OK way of doing it) and it works if I pip install my application. When I try to package it, though, it breaks with a message

    dpkg-source: error: cannot represent change to popcorn_data/popcorn.png: binary file contents changed

    It points me at adding this stuff manually using dpkg-source and some quilt patches that add more files and folders, which just doesn't seem like a really elegant solution.

    I tried looking at different existing packages, but most of them have custom-made functions or imported modules (such as DistutilsExtra in case of Quickly apps) in setup.py.

    Is there some nice, canonical way of doing this?

    You can see my code so far at:

    http://bazaar.launchpad.net/~kermit666/popcorn.py/trunk/files

    ReplyDelete