Tuesday, April 24, 2012

Python 3 on the desktop for Quantal Quetzal

So, now all the world now knows that my suggested code name for Ubuntu 12.10, Qwazy Quahog, was not chosen by Mark.  Oh well, maybe I'll have more luck with Racy Roadrunner.

In any case, Ubuntu 12.04 LTS is to be released any day now so it's time for my semi-annual report on Python plans for Ubuntu.  I seem to write about this every cycle, so 12.10 is no exception.  We've made some fantastic progress, but now it's time to get serious.

For Ubuntu 12.10, we've made it a release goal to have Python 3 only on the desktop CD images.  The usual caveats apply: Python 2.7 isn't going away; it will still probably always be available in the main archive.  This release goal also doesn't affect other installation CD images, such as server, or other Ubuntu flavors.  The relatively modest goal then only affects packages for the standard desktop CD images, i.e. the alternative installation CD and the live CD.

Update 20120425: To be crystal clear,  if you depend on Python 2.7, the only thing that changes for you is that after a fresh install from the desktop CD on a new machine, you'll have to explicitly apt-get install python2.7.  After that, everything else will be the same.

This is ostensibly an effort to port a significant chunk of Ubuntu to Python 3, but it really is a much wider, Python-community driven effort.  Ubuntu has its priorities, but I personally want to see a world where Python 3 rules the day, and we can finally start scoffing at Python 2 :).

Still, that leaves us with about 145 binary packages (and many fewer source packages) to port.  There are a few categories of packages to consider:

  • Already ported and available.  This is the good news, and covers packages such as dbus-python.  Unfortunately, there aren't too many others, but we need to check with Debian and make sure we're in sync with any packages there that already support Python 3 (python3-dateutil comes to mind).
  • Upstream supports Python 3, but it is not yet available in Debian or Ubuntu.  These packages should be fairly easy to port, since we have pretty good packaging guidelines for supporting both Python 2 and Python 3.
  • Packages with better replacements for Python 3.  A good example is the python-simplejson package.  Here, we might not care as much because Python 3 already comes with a json module in its standard library, so code which depends on python-simplejson and is required for the desktop CD, should be ported to use the stdlib json module.  python-gobject is another case where porting is a better option, since pygi (gobject-introspection) already supports Python 3.
  • Canonical is the upstream.  Many packages in the archive, such as python-launchpadlib and python-lazr.restfulclient are developed upstream by Canonical.  This doesn't mean you can't or shouldn't help out with the porting of those modules, it's just that we know who to lean on as a last resort.  By all means, feel free to contribute to these too!
  • Orphaned by upstream.  These are the most problematic, since there's essentially no upstream maintainer to contribute patches to.  An example is python-oauth.  In these cases, we need to look for alternatives that are maintained upstream, and open to porting to Python 3.  In the case of python-oauth, we need to investigate oauth2, and see if there are features we're using from the abandoned package that may not be available in the supported one.
  • Unknowns.  Well, this one's the big risky part because we don't know what we don't know.
We need your help!  First of all, there's no way I can personally port everything on our list, including both libraries and applications.  We may have to make some hard choices to drop some functionality from Ubuntu if we can't get it ported, and we don't want to have to do that.  So here are some ways you can contribute:
  • Fill in the spreadsheet with more information.  If you're aware of an upstream or Debian port to Python 3, let us know.  It may make it easier for someone else to enable the Python 3 version in Debian, or to shepherd the upstream patch to landing on their trunk.
  • Help upstream make a Python 3 port available.  There are lots of resources available to help you port some code, from quick references to in-depth guides.  There's also a mailing list (and Gmane newsgroup mirror) you can join to get help, report status, and have other related discussions. Some people have asked Python 3 porting questions on StackOverflow, using the tags #python, #python-3.x, and #porting
  • Join us on the #python3 IRC channel on Freenode.
  • Subscribe to the python-porting mailing list.
  • Get packages ported in Debian.  Once upstream supports Python 3, you can extend the existing Debian package to expose this support into Debian.  From there, you or we can make sure that gets sync'd into Ubuntu.
  • Spread the word!  Even if you don't have time to do any ports yourself, you can help publicize this effort through social media, mailing lists, and your local Python community.  This really is a Python-wide effort!
Python 3.3 is scheduled to be released later this year.  Please help make 2012 the year that Python 3 reached critical mass!


On a more personal note, I am also committed to making Mailman 3 a Python 3 application, but right now I'm blocked on a number of dependencies.  Here are the list of dependencies from the setup.py file, and their statuses.  I would love it if you help get these ported too!
Of course, these are only the direct dependencies.  Others that get pulled in include:


  1. Just to get this clarified: this will not affect the ability to run and develop Python 2 applications (including virtualenv) in any way, right?

    Much of computational neuroscience runs on Python 2 specifically, and porting stuff to Python 3 (which is, essentially, a different language) is not even in sight. My most used package, for instance, is right now dropping support for Python 2.4 and will require at least 2.5, after over a year of agonizing. I would not hold my breath for a shift to Python 3 for at least another decade.

    1. Jan, it will affect you in only one minor way: after a fresh install of the desktop CD on a new machine, you'll need to apt-get install python2.7 since it won't be there by default. Other than that, *nothing* else changes. /usr/bin/python will still point to Python 2.7, virtualenv will still work (and default to using Python 2.7), etc.

  2. The zope.interface 3.8.0 release has the basic zope component architecture features. So, it could be possible to remove dependency on zope.component. For example, Pyramid has removed the dependency on zope.component in their latest stable release which support Python 3.

    1. Thanks for the update Baiju! The only thing I use is zope.component.getUtility(). Oh, my configure.zcml also includes meta.zcml from zope.component, but maybe that's unnecessary too. I've updated the status of the Mailman 3 dependencies below and will try to rewrite the code to use just zope.interface. Hopefully, one down... :)

  3. If this means /usr/bin/python is no longer going to be real python but some form of py3k I guess I will not be staying with ubuntu. Shame i have used it has my desktop distro since breezy.

    1. Jan, garylinux: Python 2 will definitely still be supported, and AFAIK will still be /usr/bin/python, so there's no cause for concern there. This is about what core Ubuntu applications like Update manager run on.

      The only likely change to Python 2 is that, once it's not needed for the default applications, it won't be part of a standard install. It will still be there in the repositories, and if you install a package that requires it, Python 2 will be pulled in automatically, like any other dependency.

    2. Thomas K is exactly right. /usr/bin/python will absolutely still point to Python 2.7.

    3. So what will /usr/bin/python point to after a default CD install, i.e. *before* you install python2.7 from the repositories?

    4. @Vinay, it won't exist. /usr/bin/python3 will though. I should also mention that you may not have to explicitly install Python 2.7. It will of course get pulled in automatically as a dependency if you subsequently install any other tool that hasn't yet been ported to Python 3.

  4. For you only python 2 is the "real python" ? Say that to archlinux who use python 3 as default python for more than a year. And guess what, it's working (except with developers thinking like you that use "#/usr/bin/env python" instead of python2).

    1. Note that IMHO, no deployed application should every use "#!/usr/bin/env python". That's just asking for disaster, especially in tools that the operating system requires to function correctly. I think that's common knowledge now in 2012. It's fine to develop code that uses that #! line, but once in production, it's wise to fix the Python version to what you know will work.

      I haven't tried Arch, but my understanding is that they changed /usr/bin/python to be Python 3. Debian and Ubuntu will never do that; you'll have to use #!/usr/bin/python3 to get Python 3. See also PEP 394.

    2. I think Arch has done the right thing actually in making /usr/bin/python point to Python 3. People just need to get over Python 2. It's actually getting quite annoying to hear these people hanging on stubbornly.

    3. Consider also: http://www.python.org/dev/peps/pep-0394/

      There, the recommendation is basically that /usr/bin/python should point to Python 2 (even if some "bleeding edge" distributions do otherwise) for at least the next few years. I personally happen to agree, and if it's an official recommendation, it's best if we all stick to it.

    4. PEP 394 came from a python-dev discussion about Arch switching /usr/bin/python. The conversation would have come up eventually, but Arch did not have the support of the Python community when it switched.

  5. mart, there is no "python2" target. Only python[specific version for this distro], python2.7 for Ubuntu 11.10 for instance. So developers can't target "python2".

    Better than breaking a huge amount of existing software, install Python 3 as "python3" and have scripts looking for the new language explicitly use that.

  6. Barry,

    I needed OAuth 1a for Python3. I looked at python-oauth, then at python-oauth2, and then decided to write my own. I only implemented the oauth client functionality, but I imagine that's all anything on the CD needs. I did it in just 35 lines of Python3:


    It has comprehensive tests using example values from the OAuth 1a spec, and I've given it serious beatings with Novacut and Dmedia (but that also means it's only been used to authenticate to CouchDB, so testing with other servers/services would be needed).

    I didn't try to be API compatible with python-oauth as I felt its API was crufty, needlessly OO, and way over complicated. But I'd be happy to split this functionality out of Microfiber and perhaps rework the API slightly or write a compatibility layer to make porting from python-oauth easier.

    We should chat about this at UDS :^)

    1. That's great to hear. Let's absolutely do that Jason! I have two UDS-Q blueprints, so I'll see you there. :)

  7. FWIW, zc.buildout 2.X does already run on Python 3. It has a bdist-only release on pypi at http://pypi.python.org/pypi/zc.buildout/2.0.0a1 . It's bdist-only for some complicated reason I don't understand having to do with what packaging tools are willing to pull down b default.

    1. I'll make you a deal. If I can get you to care about https://bugs.launchpad.net/zc.buildout/+bug/988380 I'll make sure to drive the desktop team to fix their #! lines :)

      % find /usr/bin -type f | xargs grep "/usr/bin/env python"
      /usr/bin/u1sdtool:#! /usr/bin/env python
      /usr/bin/gtk-builder-convert:#!/usr/bin/env python
      /usr/bin/purple-remote:#!/usr/bin/env python
      /usr/bin/gdbus-codegen:#!/usr/bin/env python
      /usr/bin/pastebinit:#!/usr/bin/env python
      /usr/bin/gsettings-schema-convert:#!/usr/bin/env python
      /usr/bin/gwibber-accounts:#!/usr/bin/env python
      /usr/bin/isoquery:#!/usr/bin/env python
      /usr/bin/purple-url-handler:#!/usr/bin/env python
      /usr/bin/byobu-select-session:#!/usr/bin/env python
      /usr/bin/invest-chart:#!/usr/bin/env python

      /me goes to file some bugs and lean on people!

    2. I think the rationale is that Jim doesn't want the Python 3 compat version to be the default that people get via easy_install, and I'm not sure what it's going to take for him to want to make 2.X the default. Maybe we should take up a collection.

    3. Yeah, he "Won't Fix"ed LP: #988380. I can understand his rationale, although ISTM that this is a flaw in the underlying tool set. You should be able to declare a package version as Python 3 only and then pip/easy_install/whatever would never match on it for Python 2 builds. Oh well, I let someone else fight that particular battle.

    4. A flaw in Python distribution installation tools? Surely you jest.

  8. Oh and Barry. gwibber-service, gwibber-accounts, and other various Canonical-driven Python GUI thingies use "#!/bin/env python" instead of "/usr/bin/python". That's destined to break, and always does. I realize I should post a bug report about this. But I did back in the 10.04 days and apparently nobody cared. So maybe I can make you care. ;-)

  9. Thanks for this great post it was very informative and helped me with my own project I am attempting to complete.

  10. @Liso: the quick answer is that we made some great progress, but still have more to go. After UDS-R in a couple of weeks I'll post an update for 13.04. I still think we'll get there by 14.04 LTS.

  11. Hello Barry! Could we hear something good news? :)