Wednesday, November 23, 2011

An update on Ubuntu's Python plans

Earlier this month, I attended UDS-P (the Ubuntu Developers Summit for 12.04 Precise Pangolin).  12.04 is an LTS release, or Long Term Support, meaning it will be officially supported on both the desktop and server for five years.

During the summit we reiterated our plans for Python on Ubuntu, both for 12.04 LTS, and our vision of Python two years from now for the next LTS, 14.04.  I'm here to provide an update from my last report, which was written after UDS-O for 11.10.

As per those previous plans, we've removed Python 2.6 from Ubuntu 12.04.  Now you have only Python 2.7 and 3.2.  Dropping Python 2.6 may cause some inconvenience for data centers which like to upgrade only between LTS's but don't want to have to upgrade both their operation system and their Python version.  Canonical services such as Launchpad and Landscape fall into this camp.  The biggest problem is that Python 2.7 is not available for the last LTS, i.e. 10.04.  To mitigate this, we decided to create a PPA containing Python 2.7 and a bunch of packages that services such as Launchpad will need to do their porting.  This now exists, although it hasn't yet been tested with any development branches of Launchpad or Landscape as far as I'm aware.  If you have your own data center porting task ahead of you, you can also use this PPA, and if there are additional packages you need for 10.04, you can create your own PPA which depends on ours, and build those extra packages there.

But that's all boring stuff.  Let's have some fun!

The other decision we made concerns Python 3.  I strongly feel that the Python community is very close to the tipping point in Python 3 adoption.  Yes, there's still tons of code that needs porting, but we're seeing more and more activity every day.  Even large frameworks such as Twisted are working on Python 3 branches.  I wrote PEP 404 that (somewhat tongue in check) expresses official Python pronouncement on the migration path from Python 2.7 to Python 3.  There will not be a Python 2.8, so the time to begin porting your code is now.

And it's really not that difficult.  There are several great guides available to help you with porting your pure Python packages to Python 3, available on both python.org and python3porting.com.  There are packages such as six available to ease the trickier bits of porting, and I have another blog post coming that describes my experiences in porting dbus-python. This latter task is made more complex by being a fairly involved C extension module.  Again, both python.org and python3porting.com have useful guides for porting extension modules, although they have missed a few things.  I wrote a mailing list article on some of the things that I had to do, but the blog post will include more detail.

Other distros such as Fedora are also starting to work on porting essential packages to Python 3, and here's where free software and open source rules.  Our distros can work together to help nudge the Python community over that tipping point.  Fedora is working from a bottom-up approach, where they are porting packages as needed by request, and we in Ubuntu are working top-down, by identifying several desktop applications that we'll port, along with their dependency stack.  We'll both be pushing our changes to the upstream projects so every Python developer can benefit, no matter what platform they are on.

My long term vision is that Ubuntu 14.04 won't even have Python 2 on the CD images (if we even have CD images in two years), and we'll also move Python 2.7 off to universe.  It will never completely go away, but the operating system and all the applications you get in a default install will run on Python 3.  I'm much more sanguine about this goal now that I was even six months ago, but that's not to say it still won't be a lot of work.

The time is now to move to Python 3, if for no other reason than to fix all those pesky UnicodeErrors.  I think there's a lot more of value in Python 3 too of course, and Python 3.3 will be even more awesome.  I expect Python 3.3 or 3.4 to be in Ubuntu 14.04.

The place to be for all Python 3 porting efforts is the python-porting mailing list.

UPDATE: Fixed Python 3.3/3.4 plans to correctly read for Ubuntu 14.04 (thanks Nick!)

19 comments:

  1. Getting 3.3 into 12.04 would be a neat trick, given it isn't expected to be released until 12.08!

    ReplyDelete
  2. Whoops! Thanks Nick, that was a typo. I mean to say that I expect Ubuntu 14.04 to have Python 3.3 or 3.4.

    ReplyDelete
  3. Also, you forgot to throw in Ubuntu One. :)

    ReplyDelete
  4. Please put PyPy into your roadmap as well, it is already a much faster Python and making great strides lately. The drawback is that it still lacks support for many of the C extensions of Python, such as numpy, PyQt4 so I understand that it won't replace cpython anytime soon. But consider it please, it could make Ubuntu an even faster and more fun experience :-)

    ReplyDelete
  5. I assume (hope?) that ubuntu has no plans of doing what arch did to /usr/bin/python ?

    ReplyDelete
  6. Having Python 2.7 in 10.04 is great. This way we can port everything and be ready when 12.04 comes out Our software is already running fine in 2.7 but we need to test our deployment to it.

    Thanks for this!

    About pypy, I don't see how putting it on a LTS release is going to help anyone, because you definetely want the latest version of pypy. Having a well supported ppa tracking the latest release for it though would be great. Bruno Gola has one I think, just needs more people using it.

    ReplyDelete
  7. Here's hoping Ubuntu's Python plans will help push adoption of Python3 by more libraries. We have 15 dependencies in one of our current requirements.txt files, and exactly one of them -- Jinja2 -- supports Python3.

    Until such libraries/frameworks as Django and ReportLab make a show of supporting Python3, we can't even begin to comtemplate making the transition.

    ReplyDelete
  8. I don't think PyPy supports language-versioned install layouts at the moment. Which means Python packages can't register themselves with PyPy with the current tools. Maybe that will appear when PyPy starts supporting Python3 and needs to handle multiple install locations.

    ReplyDelete
  9. @Sidnei: right! U1 too :)
    @Anonymous: No, /usr/bin/python will (probably) never point to python3. The only possible change to that would be if PEP 394 somehow mandated, but I doubt that will ever happen.
    @Wladmir: consensus seems to be that PyPy, while awesome, is moving too fast to be added to the archive, especially for an LTS. Better to put it in a PPA for now.
    @rubic: that's my ulterior motive too :)

    ReplyDelete
  10. @Barry,

    Consensus among whom? Certainly not among me :). PyPy will soon be at the point where it should be, at least server-side, everyone's default runtime. For a lot of cases it's there already. It's certainly got more software ready for production use today than Python 3. Having it in Ubuntu would be great! Like bzr, more demanding users could always add a pypy backport ppa later if they need one.

    ReplyDelete
  11. @glyph,

    That depends very much on what you're doing. For anything that requires C extensions to Python (GUI toolkits, plotting, messaging libraries, and loads more stuff), the picture for Python 3 looks much better than that for pypy. I'd be happy to see PyPy included, but it certainly shouldn't be the default runtime any time soon.

    ReplyDelete
  12. Barry, FWIW, it's not just possible "inconvenience for data centers which like to upgrade only between LTS's". It's a more fundamental issue about Ubuntu not providing stable APIs between releases.

    Any developer who wants to target multiple supported releases of Ubuntu with Python, can't. Doesn't matter whether they are on server or on client. Often doesn't matter if one release is an LTS and the other is not.

    They either have to:
    * make sure their software works on multiple major releases of Python (tricky, hard to test, problems with dependencies that don't always work)
    * get their users to set up PPAs (cumbersome, undesirable, rules out selling things in the Software Center)
    * port every time Ubuntu makes a change like this (extra work to stay in the same spot)

    I don't think this is an easy problem to solve. I'd like to give it a try though.

    ReplyDelete
  13. @glyph: Consensus among core pypy developers afaict. It really comes down to how stable their releases are. I don't think it helps to have a quickly moving target in the archives (too much overhead to keep updated). If that's not the case, then it should go into Debian first, and then get sync'd to Ubuntu.

    ReplyDelete
  14. PyPy RFP in Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=639315

    ReplyDelete
  15. jml: I know it sucks, but the alternative is unattractive too. If we keep 2.6 in 12.04, it means we have to commit to maintain it for about 4 years past the date that upstream is committed to *just* security fixes (it's not even being bug fixed any more). EOL for 2.6 is 2013, but 12.04 will be supported until 2017.

    The other possibility is to put 2.7 into lucid-backports, which I think makes more sense. However, the PPA is the place to start with that and afaik, nobody's even tried the packages I've put there.

    That's only part of the problem too, because you have to support a bunch of packages as well. Do you choose the Lucid versions, many of which aren't compatible with Python 2.7, or do you choose say the Oneiric versions, which might cause other upgrade problems?

    I think this is a very difficult problem, and one that doesn't affect only Python, although it's probably one of the most visible because of Python's reach. I'm certainly open to suggestions, but I haven't heard anything that I think is workable yet.

    ReplyDelete
  16. @Barry, @jml: Is there a problem including 2.6 with a deprecation notice saying it is unsupported and not for production use but is included just for getting people over the line to 2.7?

    Bottom line is 12.04 is coming out right at the time Python is in transition. The ladder is as follows -- (all previous Pythons) -> (2.7) -> (3.x) and you could easily argue that 2.6 represents all previous Pythons and without it you are missing the first rung on the ladder.

    We all want to drop 2.x as soon as possible so we should make it easy by including the 2.6 rung.

    ReplyDelete
  17. @Anonymous: It's too late to add Python 2.6 back to 12.04, even if we wanted to, which we don't :). One of the main reasons is that Python 2.6 is out of bug fix maintenance upstream, so supporting it for another 5 years on an LTS is simply infeasible. Other technical concerns come up, for example, adding a new Python version requires rebuilds of all packages with shared libraries, etc.

    I think Python 2.7 represents a good basis for migrating to, and it's not too hard to do that for Python 2 code. We have provided Python 2.7 in a semi-official PPA for Lucid for those folks who want to migrate their Python version before migrating from LTS->LTS.

    Note too, we're going to get rid of Python 2 from the CD in 12.10. It will still be in main most likely and definitely in the archive, but this will be a great first step to moving all development to Python 3. (I.e. 3.2 only on the CD).

    ReplyDelete
  18. I have a large script that uses pexpect to both send and receive ASCII characters through a telnet connection to a device that will probably always use ASCII. Will python 3 still allow strings that contains plain ASCII 8-bit characters without having to resort to bytearray, or some other such construct?

    The script is full of "harcoded" strings both for responses from the other machine and for what is sent through telnet to the other machine. It would be a massive job to do a major change to the format off all these strings. The program is quite large.

    Doing a web search, I found "future_builtins.ascii(object)", which I expect might be what I need to use? Is this correct?

    Bill H

    ReplyDelete
  19. Crap. Blogger lost my original comment. :( Bill H, it sounds like what you have are actually bytes, and you'll be better off in the long run using bytes() - as an alias for str() in Python 2 - and b'' prefixes. It's difficult to say exactly what's right for your application without seeing it though, and I'm not that familiar with pexpect. Perhaps 2to3 can help you automatically convert your code?

    ReplyDelete