Posts Tagged With 'ubuntu'

Building Debian packages with local dependencies

Written by Barry Warsaw in technology on Mon 28 November 2016. Tags: ubuntu, debian, packaging,

Way back in 2011 I wrote an article describing how you can build Debian packages with local dependencies for testing purposes. An example would be a new version of a package that has new dependencies. Or perhaps the new dependency isn't available in Debian yet. You'd like to test both packages together locally before uploading. Using sbuild and autopkgtest you can have a high degree of confidence about the quality of your packages before you upload them.

Here I'll describe some of the improvements in those tools, and give you simplified instructions on how to build and test packages with local dependencies.

What's changed?

Several things have changed. Probably the biggest thing that simplifies the procedure is that GPG keys for your local repository are no longer needed.

Another thing that's improved is the package testing support. It used to be that packages could only be tested during build time, but with the addition of the autopkgtest tool, we can also test the built packages under various scenarios. This is important because it more closely mimics what your package's users will see. One thing that's cool about this for Python packages is that autopkgtest runs an import test of your package by default, so even if you don't add any explicit tests, you still get something. Of course, if you do want to add your own tests, you'll need to recreate those default tests, or check out the autodep8 package for some helpers.

I've moved the repository of scripts over to git.

The way you specify the location of the extra repositories holding your local debs has changed. Now, instead of providing a directory on the local file system, we're going to fire up a simple Python-based HTTP server and use that as a new repository URL. This won't be …

Continue reading »


Creating Python snaps

Written by Barry Warsaw in technology on Thu 02 April 2015. Tags: debugging, python, python3, ubuntu,

Background

Snappy Ubuntu Core is a new edition of the Ubuntu you know and love, with some interesting new features, including atomic, transactional updates, and a much more lightweight application deployment story than traditional Debian/Ubuntu packaging. Much of this work grew out of our development of a mobile/touch based version of Ubuntu for phones and tablets, but now Ubuntu Core is available for clouds and devices.

I find the transactional nature of upgrades to be very interesting. While you still get a perfectly normal Ubuntu system, your root file system is read-only, so traditional apt-get based upgrades don't work. Instead, your system version is image based; today you are running image 231 and tomorrow a new image is released to get you to 232. When you upgrade to the new image, you get all the system changes. We support both full and delta upgrades (the latter which reduces bandwidth), and even phased updates so that we can roll out new upgrades and quickly pull them from the server side if we notice a problem. Snappy devices even support rolling back upgrades on a single device, by using a dual-partition root file system. Phones generally don't support this due to lack of available space on the device.

Of course, the other part really interesting thing about Snappy is the lightweight, flexible approach to deploying applications. I still remember my early days learning how to package software for Debian and Ubuntu, and now that I'm both an Ubuntu Core Developer and Debian Developer, I understand pretty well how to properly package things. There's still plenty of black art involved, even for relatively easy upstream packages such as distutils/setuptools-based Python packages available on the Cheeseshop (er, PyPI). The Snappy approach on Ubuntu Core is much more lightweight and easy, and …

Continue reading »


Resource management in Python 3.3, or contextlib.ExitStack FTW!

Written by Barry Warsaw in technology on Fri 10 May 2013. Tags: python, python3, ubuntu,

I'm writing a bunch of new code these days for Ubuntu Touch's Image Based Upgrade_ system. Think of it essentially as Ubuntu Touch's version of upgrading the phone/tablet (affectionately called *phablet*) operating system in a bulk way rather than piecemeal `apt-get s the way you do it on a traditional Ubuntu desktop or server. One of the key differences is that a phone has to detour through a reboot in order to apply an upgrade since its Ubuntu root file system is mounted read-only during the user session.

Anyway, those details aren't the focus of this article. Instead, just realize that because it's a pile of new code, and because we want to rid ourselves of Python 2, at least on the phablet image if not everywhere else in Ubuntu, I am prototyping all this in Python 3, and specifically 3.3. This means that I can use all the latest and greatest cool stuff in the most recent stable Python release. And man, is there a lot of cool stuff!

One module in particular that I'm especially fond of is contextlib. Context managers are objects implementing the protocol behind the with statement, and they are typically used to guarantee that some resource is cleaned up properly, even in the event of error conditions. When you see code like this:

with open(somefile) as fp:
    data = fp.read()

you are invoking a context manager. Python was clever enough to make file objects support the context manager protocol so that you never have to explicitly close the file; that happens automatically when the with statement completes, regardless of whether the code inside the with statement succeeds or raises an exception.

It's also very easy to define your own context managers to properly handle other kinds of resources. I won't go …

Continue reading »


Python 3 Language Gotcha -- and a short reminisce

Written by Barry Warsaw in technology on Thu 18 April 2013. Tags: python, python3, ubuntu,

There's a lot of Python nostalgia going around today, from Brett Cannon's 10 year anniversary of becoming a core developer, to Guido reminding us that he came to the USA 18 years ago. Despite my stolen time machine keys, I don't want to dwell in the past, except to say that I echo much of what Brett says. I had no idea how life changing it would be -- on both a personal and professional level -- when Roger Masse and I met Guido at NIST at the first Python workshop back in November 1994. The lyric goes: what a long strange trip it's been, and that's for sure. There were about 20 people at that first workshop, and 2500 at Pycon 2013.

And Python continues to hold little surprises. Just today, I solved a bug in an Ubuntu package that's been perplexing us for weeks. I'd looked at the code dozens of times and saw nothing wrong. I even knew about the underlying corner of the language, but didn't put them together until just now. Here's a boiled down example, see if you can spot the bug!

import sys

def bar(i):
    if i == 1:
        raise KeyError(1)
    if i == 2:
        raise ValueError(2)


def bad():
    e = None
    try:
        bar(int(sys.argv[1]))
    except KeyError as e:
        print('ke')
    except ValueError as e:
        print('ve')
    print(e)

bad()

Here's a hint: this works under Python 2, but gives you an UnboundLocalError on the e variable under Python 3.

Why?

The reason is that in Python 3, the targets of except clauses are del'd from the current namespace after the try...except clause executes. This is to prevent circular references that occur when the exception is bound to the target. What is surprising and non-obvious is that the name is deleted …

Continue reading »


UDS Update #1 - OAuth

Written by Barry Warsaw in technology on Wed 21 November 2012. Tags: debian, python, python3, ubuntu, uds-r,

For UDS-R for Raring (i.e. Ubuntu 13.04) in Copenhagen, I sponsored three blueprints. These blueprints represent most of the work I will be doing for the next 6 months, as we're well on our way to the next LTS, Ubuntu 14.04.

I'll provide some updates to the other blueprints later, but for now, I want to talk about OAuth and Python 3. OAuth is a protocol which allows you to programmatically interact with certain website APIs, in an authenticated manner, without having to provide your website password. Essentially, it allows you to generate an authorization token which you can use instead, and it allows you to manage and share these tokens with applications, so that you can revoke them if you want, or decide how and which applications to trust to act on your behalf.

A good example of a site that uses OAuth is Launchpad, but many other sites also support OAuth, such as Twitter and Facebook.

There are actually two versions of OAuth out there. OAuth version 1 is definitely the more prevalent, since it has been around for years, is relatively simple (at least on the client side), and enshrined in RFC 5849. There are tons of libraries available that support OAuth v1, in a multitude of languages, with Python being no exception.

OAuth v2 is much less common, since it is currently only a draft specification, and has had its share of design-by-committee controversy. Still, some sites such as Facebook do require OAuth v2.

One of the very earliest Python libraries to support OAuth v1, on both the client and server side, was python-oauth (I'll use the Debian package names in this post), and on the Ubuntu desktop, you'll find lots of scripts and libraries that use python-oauth. There are major problems with …

Continue reading »


The right way to internationalize your Python app

Written by Barry Warsaw in technology on Fri 22 June 2012. Tags: i18n, mailman, python, python3, ubuntu,

Recently, as part of our push to ship only Python 3 on the Ubuntu 12.10 desktop, I've helped several projects update their internationalization (i18n) support. I've seen lots of instances of suboptimal Python 2 i18n code, which leads to liberal sprinkling of cargo culted .decode() and .encode() calls simply to avoid the dreaded UnicodeError s. These get worse when the application or library is ported to Python 3 because then even the workarounds aren't enough to prevent nasty failures in non-ASCII environments (i.e. the non-English speaking world majority :).

Let's be honest though, the problem is not because these developers are crappy coders! In fact, far from it, the folks I've talked with are really really smart, experienced Pythonistas. The fundamental problem is Python 2's 8-bit string type which doubles as a bytes type, and the terrible API of the built-in Python 2 gettext module, which does its utmost to sabotage your Python 2 i18n programs. I take considerable blame for the latter, since I wrote the original version of that module. At the time, I really didn't understand unicodes (this is probably also evident in the mess I made of the email package). Oh, to really have access to Guido's time machine.

The good news is that we now know how to do i18n right, especially in a bilingual Python 2/3 world, and the Python 3 gettext module fixes the most egregious problems in the Python 2 version. Hopefully this article does some measure of making up for my past sins.

Stop right here and go watch Ned Batchelder's talk from PyCon 2012 entitled Pragmatic Unicode, or How Do I Stop the Pain? It's the single best description of the background and effective use of Unicode in Python you'll ever see. Ned does a brilliant job of …

Continue reading »


Python 3 on the desktop for Quantal Quetzal

Written by Barry Warsaw in technology on Tue 24 April 2012. Tags: canonical, debian, ubuntu, python, python3,

So, now all the world now knows that my suggested code name for Ubuntu 12.10, Qwazy Quahog, was not chosen by Mark. Oh well, maybe I'll have more luck with Racy Roadrunner.

In any case, Ubuntu 12.04 LTS is to be released any day now so it's time for my semi-annual report on Python plans for Ubuntu. I seem to write about this every cycle, so 12.10 is no exception. We've made some fantastic progress, but now it's time to get serious.

For Ubuntu 12.10, we've made it a release goal to have Python 3 only on the desktop CD images. The usual caveats apply: Python 2.7 isn't going away; it will still probably always be available in the main archive. This release goal also doesn't affect other installation CD images, such as server, or other Ubuntu flavors. The relatively modest goal then only affects packages for the standard desktop CD images, i.e. the alternative installation CD and the live CD.

Update 2012-04-25: To be crystal clear, if you depend on Python 2.7, the only thing that changes for you is that after a fresh install from the desktop CD on a new machine, you'll have to explicitly apt-get install *python2.7. After that, everything else will be the same.

This is ostensibly an effort to port a significant chunk of Ubuntu to Python 3, but it really is a much wider, Python-community driven effort. Ubuntu has its priorities, but I personally want to see a world where Python 3 rules the day, and we can finally start scoffing at Python 2 :).

Still, that leaves us with about 145 binary packages (and many fewer source packages) to port. There are a few categories of packages to consider:

Already ported and available.
This is …

Continue reading »


Lessons in porting to Python 3

Written by Barry Warsaw in technology on Wed 07 December 2011. Tags: python, python3, ubuntu,

Yesterday, I completed my port of dbus-python to Python 3, and submitted my patch upstream. While I've yet to hear any feedback from Simon about my patch, I'm fairly confident that it's going in the right direction. This version should allow existing Python 2 applications to run largely unchanged, and minimizes the differences that clients will have to make to use the Python 3 version.

Some of the changes are specific to the dbus-python project, and I included a detailed summary of those changes and my rationale behind them. There are lots of good lessons learned during this porting exercise that I want to share with you, have a discussion about, and see if there aren't things we core Python developers can do in Python 3.3 to make it even easier to migrate to Python 3.

First, some background. D-Bus is a freedesktop.org project for same-system interprocess communication, and it's an essential component of any Linux desktop. The D-Bus system and C API are mature and well-defined, and there are bindings available for many programming language, Python included of course. The existing dbus-python package is only compatible with Python 2, and most recommendations are to use the Gnome version of Python bindings should you want to use D-Bus with Python 3. For us in Ubuntu, this isn't acceptable though because we must have a solution that supports KDE and potentially even non-UI based D-Bus Python servers. Several ports of dbus-python to Python 3 have been attempted in the past, but none have been accepted upstream, so naturally I took it as a challenge to work on a new version of the port. After some discussion with the upstream maintainer Simon McVittie, I had a few requirements in mind:

  • One code base for both Python 2 and Python 3 …

Continue reading »


An update on Ubuntu's Python plans

Written by Barry Warsaw in technology on Wed 23 November 2011. Tags: python, ubuntu,

Earlier this month, I attended UDS-P (the Ubuntu Developers Summit for 12.04 Precise Pangolin). 12.04 is an LTS release, or Long Term Support, meaning it will be officially supported on both the desktop and server for five years.

During the summit we reiterated our plans for Python on Ubuntu, both for 12.04 LTS, and our vision of Python two years from now for the next LTS, 14.04. I'm here to provide an update from my last report, which was written after UDS-O for 11.10.

As per those previous plans, we've removed Python 2.6 from Ubuntu 12.04. Now you have only Python 2.7 and 3.2. Dropping Python 2.6 may cause some inconvenience for data centers which like to upgrade only between LTS's but don't want to have to upgrade both their operation system and their Python version. Canonical services such as Launchpad and Landscape fall into this camp. The biggest problem is that Python 2.7 is not available for the last LTS, i.e. 10.04. To mitigate this, we decided to create a PPA containing Python 2.7 and a bunch of packages that services such as Launchpad will need to do their porting. This now exists, although it hasn't yet been tested with any development branches of Launchpad or Landscape as far as I'm aware. If you have your own data center porting task ahead of you, you can also use this PPA, and if there are additional packages you need for 10.04, you can create your own PPA which depends on ours, and build those extra packages there.

But that's all boring stuff. Let's have some fun!

The other decision we made concerns Python 3. I strongly feel that the Python community is very close to …

Continue reading »


sbuild with local, newer, dependencies

Written by Barry Warsaw in technology on Thu 01 September 2011. Tags: canonical, debian, packaging, ubuntu,

Update 2016-11-28: I've updated this article with new instructions!

sbuild is an excellent tool for locally building Ubuntu and Debian packages. It fits into roughly the same problem space as the more popular pbuilder, but for many reasons, I prefer sbuild. It's based on schroot to create chroot environments for any distribution and version you might want. For example, I have chroots for Ubuntu Oneiric, Natty, Maverick, and Lucid, Debian Sid, Wheezy, and Squeeze, for both i386 and amd64. It uses an overlay filesystem so you can easily set up the primary snapshot with whatever packages or prerequisites you want, and the individual builds will create a new session with an overlaid temporary filesystem on top of that, so the build results will not affect your primary snapshot. sbuild can also be configured to save the session depending on the success or failure of your build, which is fantastic for debugging build failures. I've been told that Launchpad's build farm uses a customized version of sbuild, and in my experience, if you can get a package to build locally with sbuild, it will build fine in the main archive or a PPA.

Right out of the box, sbuild will work great for individual package builds, with very little configuration or setup. The Ubuntu Security Team's wiki page has some excellent instructions for getting started (you can stop reading when you get to UMT :).

One thing that sbuild doesn't do very well though, is help you build a stack of packages. By that I mean, when you have a new package that itself has new dependencies, you need to build those dependencies first, and then build your new package based on those dependencies. Here's an example.

I'm working on bug 832864 and I wanted to see if I could build the …

Continue reading »