Pumpichank

Debian packaging for Python 2 and 3

2012-01-18T19:05:00.000-05:00

Time for another installment of my ongoing mission to convert the world to Python 3! This time, a little Debian packaging-fu for modifying an existing Python 2 package to include support for Python 3 from the same source package.

Today, I added a python3-feedparser package to Ubuntu Precise. What's interesting about this is that, despite various reported problems, upstream feedparser 5.1 claims to support Python 3, via 2to3 conversion. And indeed it does (although the test suite does not).

Before today, Ubuntu had feedparser 5.0.1 in its archive, and while some work has been done to update the Debian package to 5.1, this has not been released. The uninteresting precursor to Python 3 packaging was to upgrade the Ubuntu version of the python-feedparser source package to 5.1. I'll spare you the boring details about missing data files in the upstream tarball, and other problems, since they don't really relate to the Python 3 effort.

The first step was to verify that feedparser 5.1 works with Python 3.2 in a virtualenv, and indeed it does. This is good news because it means that the setup.py does the right thing, which is always the best way to start supporting Python 3. I've found that it's much easier to build a solid Debian package if you have a solid setup.py in upstream to begin with.

Now, what I'd like to do is to give you a recipe for modifying your existing debian/ directory files to add Python 3 support to a package that already exists for Python 2. This is a little trickier for feedparser because it used an older debhelper standard, and carried some crufty old stuff in its rules file. My first step was to update this to debhelper compatibility level 8 and greatly simplify the debian/rules file. Here's what it might have looked like with just Python 2 support, so let's start there.

#!/usr/bin/make -f
export DH_VERBOSE=1

%:
    dh $@ --with python2

override_dh_auto_clean:
    dh_auto_clean
    rm -rf build .*egg-info

override_dh_auto_test:
ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
    cd feedparser && python ./feedparsertest.py
else
    @echo "nocheck set, not running tests"
endif

override_dh_installdocs:
    dh_installdocs -Xtests

This is all pretty standard stuff. dh_python2 is used (the --with python2 option to dh), and we just provide a couple of overrides for idiosyncrasies in the feedparser package. We clean a couple of extra things that aren't cleaned automatically, and we run the test suite in the slightly non-standard way that upstream requires. Also, we override the installation of a huge amount of test files that would otherwise get installed as documentation (they aren't docs).

So far so good. What do we have to do to add support for Python 3?

First, we need to make a few modifications to the debian/control file. The current convention with dh_python2 is to use an X-Python-Version header in the source package stanza, so we just need to add this header to the same stanza for Python 3:

X-Python3-Version: >= 3.2

This just says we support any Python 3 version from 3.2 onwards. You also need to add a few additional packages to the Build-Depends. In the feedparser case, I added the following build dependencies: python3, python3-chardet, python3-setuptools. Even though for Python 2 there are a couple of other build dependencies (e.g. python-libxml2 and python-utidylib) these aren't available for Python 3, but lucky for us, they are optional anyway.

Next, you need to add a new binary package stanza. There was already a python-feedparser binary package stanza for Python 2 support. In Debian, Python 3 is provided as a separate stack, meaning packages for Python 3 will always start with the python3- prefix. Thus, it is pretty easy to just copy the python-feedparser stanza and paste it to the bottom of debian/rules, changing the package name to python3-feedparser. You have to update the Depends line to use ${python3:Depends} and I updated the Recommends line to name python3-chardet, and that was about it. Here's what the new stanza looks like:

Package: python3-feedparser
Architecture: all
Depends: ${misc:Depends}, ${python3:Depends}
Recommends: python3-chardet
Description: Universal Feed Parser for Python
Python module for downloading and parsing syndicated feeds. It can
handle RSS 0.90, Netscape RSS 0.91, Userland RSS 0.91, RSS 0.92, RSS
0.93, RSS 0.94, RSS 1.0, RSS 2.0, Atom, and CDF feeds.
.
It provides the same API to all formats, and sanitizes URIs and HTML.
.
This is the Python 3 version of the package.

Again, so far so good. Now let's look at the debian/rules file.

The first thing to do is to add support for dh_python3, which is analogous to dh_python2, and is the only accepted helper for Python 3. The rules line then becomes:

%:
dh $@ --with python2,python3

Now, one problem with debhelper is that it doesn't have any built-in support for Python 3 like it does for Python 2. This means dh will not automatically build or install any Python 3 packages, so you have to do this manually. Eventually, this will be fixed, and fortunately with a solid setup.py file, you don't have to do to much, but it's something to be aware of. In the feedparser case, we need to add overrides for dh_auto_build and dh_auto_install. Here's what these rules look like:

override_dh_auto_build:
    dh_auto_build
    set -ex; for python in $(shell py3versions -r); do \
        $$python setup.py build; \
    done;

override_dh_auto_install:
    dh_auto_install
    set -ex; for python in $(shell py3versions -r); do \
        $$python setup.py install --root=$(CURDIR)/debian/tmp --install-layout=deb; \
    done;
    cp feedparser/sgmllib3.py $(CURDIR)/debian/tmp/usr/lib/python3/dist-packages/feedparser_sgmllib3.py

Not too bad, eh? You'll notice that the first thing these rules do is call the standard dh_auto_build and dh_auto_install respectively. This preserves the Python 2 support. Then we just loop over all the available Python 3 versions, doing a fairly normal equivalent of setup.py install (split into a build step and an install step). The install rule looks a little odd, but should be familiar to Debian Python hackers. It just installs the package into the proper Debian locations, and will pretty much be the same for any Python 3 package you build.

The one odd bit is the last line in the override_dh_auto_install rule. This is there just to work around an peculiarity in the feedparser 5.1 upstream package, where it depends on sgmllib.py, but that is no longer in the Python standard library in Python 3. Upstream provides an already 2to3 converted version of it, and recommends you install the module as sgmllib.py somewhere on your Python 3 sys.path. Well, I don't like the namespace pollution that would cause, so I install the file as feedparser_sgmllib3.py and add a quilt patch to the package to try an import of that module if importing sgmllib fails (as it will on Python 3).

An aside: If you look in the debian/rules file for what I actually uploaded, you'll see some additional modifications to override_dh_auto_test. This just works around the upstream bug where some test suite data files were accidentally omitted from the release tarball. You can pretty much ignore those lines for the purposes of this article.

We're almost done. The last thing we need to do is make sure that debhelper installs the right files into the right binary packages. We want the python-feedparser binary package to include only the Python 2 files, and the python3-feedparser binary package to only include the Python 3 files. Keep in mind that when a source package builds only a single binary package (as was the case before I added Python 3 support), debhelper will include everything under the build directory's debian/tmp subdirectory in the single binary package. That's why you see things get installed into $(CURDIR)/debian/tmp. But when a source package builds multiple binary packages, as is now the case here, we have to tell debhelper which files go into which binary packages. We do this by adding two new files to the debian directory: python-feedparser.install and python3-feedparser.install

Reading the manpage for dh_install will explain the reasons for this, and describe the format of the file contents. In our case, we're really lucky, because for Python 2, everything gets installed under usr/lib/python2.* and in Python 3, everything gets installed under usr/lib/python3 (relative to $(CURDIR)/debian/tmp). You'll notice a few things here. Because we could be building for multiple versions of Python 2, we have to wildcard the actual directory under usr/lib, e.g. it might be python2.6 or python2.7. But because we have PEP 3147 and PEP 3149 in Python 3.2, there's only one directory for all supported versions of Python 3, so we don't need to wildcard the subdirectory. Also, if you look at the actual .install files in the package, you'll see a few other trailing path components, so the actual contents of the files are:

usr/lib/python2.*/*-packages/*

and

usr/lib/python3/*-packages/*

for the python-feedparser.install and python3-feedparser.install files respectively. The trailing bits just wildcard what on a Debian system will always be dist-packages, just for safety (cargo culting FTW!).

And that really is it! Of course, things could be a little more complicated if you have extension modules, but maybe not that much more so, and if the package you're adding Python 3 support to isn't setuptools-based, you may have more work to do even still. The feedparser package has a few other oddities that are really unrelated to adding Python 3 support, so I'm ignoring them here, but feel free to ask for additional details in the comments, in IRC, or in email.

Hopefully this gives you some insight into how to extend an existing Python 2 Debian package into including Python 3 support, given that your upstream already supports Python 3. Now, go forth and hack!

Addendum: my colleague Colin Watson just today packaged up Benjamin Peterson's very fine Python package called six. This is a nice package that provides some excellent Python 2 and 3 compatibility utilities. You may find this helpful if you're trying to support both Python 2 and Python 3 in a single code base, especially if you have to support back to Python 2.4 (poor you :). This will be available in Ubuntu Precise, although if you're submitting patches back upstream, you may have to convince the upstream author to accept the additional dependency. It's worth it to add a little more Python 3 love to the world.

Python 3 Porting Fun Redux

2012-01-06T12:27:00.002-05:00

My last post on Python 3 porting got some really great responses, and I've learned a lot from the feedback I've seen. I'm here to rather briefly outline a few additional tips and tricks that folks have sent me and that I've learned by doing other ports since then. Please keep them coming, either in the blog comments or to me via email. Or better yet, blog about your experiences yourself and I'll link to them from here.

One of the big lessons I'm trying to adopt is to support Python 3 in pure-Python code with a single code base. Specifically, I'm trying to avoid using 2to3 as much as possible. While I think 2to3 is an excellent tool that can make it easier to get started supporting both Python 2 and Python 3 from a single branch of code, it does have some disadvantages. The biggest problem with 2to3 is that it's slow; it can take a long time to slog through your Python code, which can be a significant impediment to your development velocity. Another 2to3 problem is that it doesn't always play nicely with other development tools, such as python setup.py test and virtualenv, and you occasionally have to write additional custom fixers for conversion that 2to3 doesn't handle.

Given that almost all the code I'm writing these days targets Python 2.6 as the minimal supported Python 2 version, 2to3 may just be unnecessary. With my dbus-python port to Python 3, and with my own flufl packages, I'm experimenting with ignoring 2to3 and trying to write one code base for all of Python 2.6, 2.7, and 3.2. My colleague Michael Foord has been pretty successful with this approach going back all the way to Python 2.4, so 2.6 as a minimum should be no problem! C extensions are pretty easy because you have the C preprocessor to help you. But it turns out that it's usually not too difficult in pure-Python either. I've done this in my latest release of the flufl.bounce package, and intend to eliminate 2to3 in my other flufl packages soon too.

The first thing I've done is add print_function to the __future__ import in all my modules. Previously, I was only importing unicode_literals and absolute_import. But doctests tend to use a lot of print statements, so switching to the print() function explicitly removes one big 2to3 conversion. Aside from having to unlearn decades of print statement muscle memory, the print() function is actually rather nice. So my module template now looks like this (with the copyright comment block omitted):

from __future__ import absolute_import, print_function, unicode_literals

__metaclass__ = type
__all__ = [
]

Speaking of doctests, you really want them to have the same set of future imports as all your other code. I'll talk more about how my own packages set up doctests later, but for now, it's useful to know that I create a doctest.DocFileSuite for every doctest in my package. These suites all have a setup() function and Python's testing framework will call these at the appropriate time, passing in a testobj parameter. This argument has a globs attribute which serves as the module globals for the doctest. All you need to do to enable the future imports in your doctests is to do something like this:

def setup(testobj):
    try:
        testobj.globs['absolute_import'] = absolute_import
        testobj.globs['print_function'] = print_function
        testobj.globs['unicode_literals'] = unicode_literals
    except NameError:
        pass

The try-except really is only necessary if you keep using 2to3, since that tool will remove the future imports from all the modules it processes. The future imports still exist in Python 3 of course, since future imports are never, ever removed. So if you ditch 2to3, you can get rid of the try-except too.

In the latest release of flufl.bounce, I changed the API so that the detected email addresses are all explicitly bytes objects in Python 3 (and 8-bit strings in Python 2). This caused some problems with my doctests because the repr of Python 3 bytes objects is different than the repr of 8-bit strings in Python 2. When you print the object in Python 2, you get just the contents of the string, but when you print them in Python 3, you get the b''-prefix.

% python
Python 2.7.2+ (default, Dec 18 2011, 17:30:39)
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print b'foo'
foo
>>>
% python3
Python 3.2.2+ (default, Dec 19 2011, 12:03:32)
[GCC 4.6.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(b'foo')
b'foo'

This means your doctest cannot be written to easily support both versions of Python when bytes/8-bit strings are used. I use the following helper to get around this:

def print_bytes(obj):
    if bytes is not str:
        obj = repr(obj)[2:-1]
    print(obj)

Remember that in Python 2, bytes is just an alias for str so this code only gets invoked in Python 3.

Another fun bytes/8-bit-string issue is that in Python 3, bytes objects have no .format() method. So if you're doing something like b'foo {0}'.format(obj) this will work in Python 2, but fail in Python 3. The best I've come up with for this is to use concatenation instead, or do the format using unicodes and then encode them to their bytes object (but then you have the additional fun of choosing an appropriate encoding!).

Did you know that the re module can scan either unicodes or bytes in Python 3? The switch is made by passing in either a bytes pattern or a str pattern, and then passing in the appropriate type of object to parse. But, if you use the r''-prefix (i.e. raw strings) for saner handling of backslashes, you've got another problem when you want to parse bytes. Python does not support rb''-prefixes, meaning you can have either raw string literals or bytes string literals but not both. You have to forgo one or the other, and I usually come down on the side of ditching the raw strings and suffering the pain of backslash proliferation.

Some of the code I was porting was using itertools.izip_longest(), but this doesn't exist in Python 3. Instead you have itertools.zip_longest(). You'll have to do a conditional import (i.e. try-except) around this to get the right version.

Do you use zope.interfaces? You'll be interested to know that the syntax we've long been accustomed to for declaring that a class implements an interface does not work in Python 3. For example:

from zope.interface import Interface, implements
class MyInterface(Interface):
pass
class MyClass:
implements(MyInterface)

This is because the stack hacking that implements() uses doesn't work in Python 3. Fortunately, the latest version of zope.interface has a new class decorator that you can use instead. This works in Python 2.6 and 2.7 too, so change your code to use this:

from zope.interface import Interface, implementer
class MyInterface(Interface):
pass
@implementer(MyInterface)
class MyClass:
pass

I kind of like the use of class decorators better anyway.

Here's a tricky one. Did you know that Python 2 provides some codecs for doing interesting conversions such as Caeser rotation (i.e. rot13)? Thus, you can do things like:

>>> 'foo'.encode('rot-13')
'sbb'

This doesn't work in Python 3 though, because even though certain str-to-str codecs like rot-13 still exist, the str.encode() interface requires that the codec return a bytes object. In order to use str-to-str codecs in both Python 2 and Python 3, you'll have to pop the hood and use a lower-level API, getting and calling the codec directly:

>>> from codecs import getencoder
>>> encoder = getencoder('rot-13')
>>> rot13string = encoder(mystring)[0]

You have to get the zeroth-element from the return value of the encoder because of the codecs API. A bit ugly, but it works in both versions of Python.

That's all for now. Happy porting!

Lessons in porting to Python 3

2011-12-07T15:41:00.001-05:00

Yesterday, I completed my port of dbus-python to Python 3, and submitted my patch upstream. While I've yet to hear any feedback from Simon about my patch, I'm fairly confident that it's going in the right direction. This version should allow existing Python 2 applications to run largely unchanged, and minimizes the differences that clients will have to make to use the Python 3 version.

Some of the changes are specific to the dbus-python project, and I included a detailed summary of those changes and my rationale behind them. There are lots of good lessons learned during this porting exercise that I want to share with you, have a discussion about, and see if there aren't things we core Python developers can do in Python 3.3 to make it even easier to migrate to Python 3.

First, some background. D-Bus is a freedesktop.org project for same-system interprocess communication, and it's an essential component of any Linux desktop. The D-Bus system and C API are mature and well-defined, and there are bindings available for many programming language, Python included of course. The existing dbus-python package is only compatible with Python 2, and most recommendations are to use the Gnome version of Python bindings should you want to use D-Bus with Python 3. For us in Ubuntu, this isn't acceptable though because we must have a solution that supports KDE and potentially even non-UI based D-Bus Python servers. Several ports of dbus-python to Python 3 have been attempted in the past, but none have been accepted upstream, so naturally I took it as a challenge to work on a new version of the port. After some discussion with the upstream maintainer Simon McVittie, I had a few requirements in mind:

One code base for both Python 2 and Python 3. It's simply too difficult to support multiple development branches, so one branch must be compilable in both versions of Python. Because dbus-python is not setuptools-based, I not to rely on 2to3 to auto-convert the Python layer. This is more difficult, but given the next requirement, entirely possible.
Minimum Python versions to support are 2.6 and 3.2 (Python 2.7 is also supported). Python 2.6 contains almost everything you need to do a high quality port of both the Python layer and the C extension layer with a single code base. Python 2.7 has one or two additional helpers, but they aren't important enough to count Python 2.6 out. For dbus-python, this specifically means dropping support for Python 2.5, which is more than 5 years old at the time of this writing. Also, it makes no sense to support Python 3.0 or 3.1 as neither of those are in wide-spread use.
Minimize any API changes seen by Python 2 code, and minimize the changes needed to port clients to Python 3. For the former, this means everything from keeping Python APIs unchanged to keeping the inheritance hierarchy the same. Python 2 programs will see a few small changes after the application of my patches; I'll describe them below but they should be inconsequential for the vast majority of Python 2 applications. While it's unavoidable that Python 3 applications will see a different API, these differences have been minimized.

There are two main issues that had to be sorted out for this port, and in general for most ports to Python 3: bytes vs. strings, and ints vs. longs. For the latter, you probably know that where Python 2 has two integer types, Python 3 has only one. In Python 3, all integers are longs, and there is no L suffix for integer literals. This turned out to be trickier in the dbus-python case because dbus supports a numeric stack of various integer widths, and in Python 2 these are implemented as subclasses of the built-in int and long types. Because there are only longs in Python 3, the inheritance hierarchy a Python application will see changes between Python 2 and Python 3. This is unavoidable.

I also made the decision to change some object types to longs in both versions of Python, where I thought it was highly unlikely that Python clients would care. Specifically, many dbus objects have a variant_level attribute, which is usually zero, but can be any positive integer. For implementation simplicity, I changed these to longs in Python 2 also.

Ah, bytes vs. strings is always where things get interesting when porting to Python 3. It's the single most brain hurty exercise you will have to go through. Remember that Python 2 lets you cheat. If you not sure whether the entity you're dealing with is some bytes, or some (usually ASCII-encoded) string, just use a Python 2 str type (a.k.a. 8-bit string) and let Python's automatic conversion rules change it to a unicode when the two types meet. You can't get away with this in Python 3 though, for very good reasons - it's error prone, and can lead to data corruption or the annoyingly ubiquitous and hard to predict UnicodeErrors.

In Python 3, you must be clear about what are bytes and what are strings (i.e. unicodes), and you must be explicit when converting between the two. Yes, this can be painful at times but in my opinion, it's crucial that you do so. It's that important to eliminate UnicodeErrors that you can't defend against and your users won't understand or be able to correct. Once you're clear in your own mind as to which are strings and which are bytes, it's usually not that hard to reflect that clearly in your code, especially if you leave Python 2.5 and anything earlier behind, which I highly recommend.

dbus-python presented an interesting challenge here. It has several data types in its C API that are defined as UTF-8 encoded char*'s. At first blush, it seemed to me that these should be reflected in Python 3 as bytes objects to simplify the conversion in the extension module to and from char*'s. It turns out that this was a bad idea from an implementation stand point, and dbus-python's upstream maintainer had already expressed his opinion that these data types should be exposed as unicodes in Python 3. After having failed at my initial attempts at making them bytes, I now agree that they must be unicodes, both for implementation simplicity and for minimal impact on porting user code.

The biggest problem I ran into with the choice of bytes is that the callback dispatch code in dbus-python is complex, difficult to understand and debug, driven by external data, and written with a deep assumption of operating on strings. For example, when the dbus C API receives a signal, it must determine whether there is a Python function registered to handle that signal, and it does this by comparing a number of client-registered parameters, such as the method name, the interface, and the object path. If the dbus C API was turning these parameters into bytes, but the clients had registered strings, then the comparisons in the callback dispatch routines would fail, either loudly with an exception, or silently with failing comparisons. The former were relatively easy to track down and fix, by explicitly decoding client-registered strings to bytes. But the latter, silent failures, were nearly impossible to debug. Add to that the fact that there were so many roads into the registration system, that it was also very difficult to coerce all incoming data early enough so that coercion wasn't necessary at comparison time. I was left with the unappealing alternative of forcing all client code to also change their data from using strings to using bytes, which I realized would be much too high a burden on clients porting their applications to Python 3. Simon was right, but it was a useful exercise to fail at anyway.

(By way of comparison, it took me the better part of a week and a half to try to get the test suite passing when these objects were bytes, which I was ultimately unable to do, and about a day to get them passing when everything was unicodes. That's gotta tell you something right there, and hopefully not that "I suck" :).

Let's look at some practical advice that may help you in your own porting efforts.

Target nothing older than Python 2.6 or Python 3.2. I mentioned this before, but it's really going to make your life easier. Specifically, drop Python 2.5 and earlier and you will thank yourself[1]. If you absolutely cannot do this, consider waiting to port to Python 3. Note that while Python 2.7 has a few additional conveniences for supporting both Python 2 and Python 3 in a single code base, I did not find them compelling enough to drop Python 2.6 support.
Where you have C types with reprs, make those reprs return unicodes in both versions. Many dbus-python types have somewhat complicated reprs because they return different strings depending on whether their variant_levels are zero or non-zero. #ifdef'ing all of these was just too much work. Because most code probably doesn't care about the specific type of the repr, and because Python 2 allows unicode reprs, and because I have a very clever hack for this[2], I decided to make all reprs return unicodes in both versions of Python.
Include the following __future__ imports in your Python code: print_function, absolute_import, and unicode_literals. In Python 2.6 and 2.7, these enable features that are the default in Python 3, and so make it easier to support both with one codebase. Specifically, change all your print statements to print() functions, and remove all your u'' prefixes from your unicode literals. Be sure to b'' prefix all your byte literals[3].
Wherever possible, in your extension modules, change all your PyInts to PyLongs. In dbus-python, this means that the variant_level attributes are longs in both Python versions, as are values that represent such things as UNIX file descriptors. The only place where I kept PyInts in Python 2 (and their requisite #ifdefs to use PyLongs in Python 3) was in the numeric stack inheritance hierarchy, mostly so that Python 2 code which cares about such things would not have to change.
Define a Python variable and a C macro for determining whether you're running in Python 2 or Python 3. The former is used in dbus-python because under Python 3, there is no UTF8String type any more, among other subtle differences. The latter is used to simply the #ifdef tests where they're needed[4].
In your C code, #include <bytesobject.h> . This header exposes aliases for all PyString calls so that you can use the Python 3 idiom of PyBytes. Then globally replace all PyString_Foo() calls with PyBytes_Foo() and the code will look clean and be compilable under both versions of Python. You may need to add explicit PyUnicode calls where you need to discern between bytes and strings, but again, this code will be completely portable between Python 2 and Python 3.
Try to write your functions to accept both unicodes and bytes, but always normalize them to one type or the other for internal use, and choose one or the other to return. Some Python stdlib methods are polymorphic in that they return bytes when handed bytes, and unicodes when handed unicodes. This can be convenient in some cases, but problematic in others. Choose carefully when porting your APIs.
Don't use trailing-L long literals if you can help it.
Switch to using Py_TYPE() everywhere instead of de-references ob_type explicitly. The structures are laid out differently between Python 2 and Python 3, and this Python-supplied macro hides the ugliness from you.

Here are a few other miscellaneous issues you should be aware of:

Metaclasses are defined differently in Python 2 and Python 3, and you cannot write any Python code snippet that is even compilable between the two. That's because the syntax for defining a class that derives from a metaclass in Python 3 is illegal syntax in Python 2. Your module simply won't compile. My solution was to use exec() on a string. For this reason, I suggest keeping metaclass subclasses as simple as possible, so that string is nice and small.

Get rid of all your uses of iteritems(), iterkeys(), itervalues(), and xrange(). You probably don't need the optimization these provide, and they do not exist in Python 3. You can conditionalize around them, but I think in most cases it's not worth it. If you really need the optimization, then you'll have to figure out a way around the missing names in Python 3. But note that Python 3 is already more efficient for the first three, since you get back dictview objects instead of concrete lists.

PyArg_Parse() and friends lack a 'y' code in Python 2. In Python 3, these return bytes objects. Where I absolutely needed bytes in Python 3 and strs in Python 2, I just #ifdef'd around the PyArg_Parse() calls. In Python 3, there's no equivalent of 'z' for bytes objects (which accept Nones and set the output variable to NULL in that case). If this is important to you, you might need to write an O& converter.

Watch out for next() vs. __next__() when writing iterators. Python 2 uses the former while Python 3 uses the latter. Best to define the method once, and then support compatibility via `next = __next__` in your class definition.

operator.isSequenceType() is gone in Python 3. Here's the code I use for compatibility:

def is_sequence(obj):
    try:
        from collections import Sequence
    except ImportError:
        from operator import isSequenceType
        return operator.isSequenceType(obj)
    else:
        return isinstance(obj, Sequence)

If you by chance use PyCObjects in your extension module, you'll have to switch these to PyCapsules for Python 3. If you're lucky enough to be able to drop Python 2.6, you can use PyCapsules everywhere, since they are available in Python 2.7.

Let me close by saying that you shouldn't be frightened off by the prospect either of porting your code to Python 3, or supporting both Python 2 and Python 3 in a single code base. It's definitely doable, and we in the Python community are gaining more experience at it every day. I strongly feel that we are well on the track of Guido's original goal of mainstream Python 3 acceptance within 5 years of Python 3's release. I think we're soon going to see a critical mass of Python 3 ports, after which time, you'll just seem old and creaky if you don't port to Python 3.

There are some other excellent references for helping you port out there on the 'net, and for the most part, I've tried not to duplicate their information. Here are some useful places to start:

Enjoy!

Footnotes:

[1] It is not impossible to support both Python 3 and versions of Python 2 earlier than 2.6, just more difficult. Michael Foord has had success doing this for libraries of his such as mock. I just think it's more trouble than it's worth in most cases.

[2] Here's the clever hack, but first a set-up. The reprs of many of the dbus-python objects are conditional on whether the variant_level is zero or not. The variant_level is only included in the repr when it is greater than zero (with zero being the typical value). This just means there are usually two calls to PyUnicode_FromFormat() in each C repr implementation, and #ifdef'ing them to use PyString_FromFormat() in Python 2 would just double the pain. In addition, the reprs all include the repr of their parent objects, i.e. their base class repr. The problem is that these base-class reprs will be PyBytes in Python 2 and PyUnicodes in Python 3, and there's nothing we can do about that. As it turns out, Python 2.6 and Python 3.2 have a %V format with some very interesting semantics. %V consumes two arguments, a PyObject* and a char*, but it only uses one of them. When the first argument is not NULL, it uses that and ignores the second argument. But when the first argument is NULL, it will use the second argument.

How can this help produce portable code? I define the following macro and use this everywhere the %V format code is given:

#define REPRV(obj) \
(PyUnicode_Check(obj) ? (obj) : NULL), \
(PyUnicode_Check(obj) ? NULL : PyBytes_AS_STRING(obj))

which would be used at a call site something like this:

return PyUnicode_FromFormat("...%V...", REPRV(parent_repr));

In Python 2, where parent_repr is a PyBytes, REPRV() will return NULL as the first argument, and via PyBytes_AS_STRING(), a char* in the second argument. In Python 3, where parent_repr is a PyUnicode, the first argument will just be the object and the second argument will be NULL (but it is ignored by Python). As long as parent_repr is either a PyUnicode or a PyBytes (a.k.a. PyString), this works perfectly, and keeps the call sites simple and sane. Beware though because if parent_repr can be any other type, this will crash your program. Fortunately, Python doesn't allow for arbitrary repr types - they must be bytes or unicodes, so in practice this is pretty safe.

[3] A recent thread in python-dev points out that this recommendation may not be practical if you're building PEP 3333-compliant WSGI applications. My take on it is that PEP 3333's definition of "native strings" is a mistake, but sadly one that we have to live with for now.

[4] Here's what my Python-level flag looks like:

import sys
is_py3 = getattr(sys.version_info, 'major', sys.version_info[0]) == 3

Now I can use this in other code to switch behavior between Python 2 and Python 3. For example, in dbus-python to import the UTF8String type in Python 2 only:

from dbus import is_py3
if is_py3:
from _dbus_bindings import UTF8String

This is much easier and less error prone then doing the sys.version_info test everywhere. The other problem is that sys.version_info is a namedtuple only in Python 2.7, so in Python 2.6, it has no attribute called 'major'.

The C-level macro looks like this:

#if PY_MAJOR_VERSION >= 3
#define PY3K
#endif

So now C code only needs to do:

#ifdef PY3K
/* Do something Python 3-ish */
#else
/* Do something Python 2-ish */
#endif

You might also find the six package to be useful here, at least for writing portable Python code.

An update on Ubuntu's Python plans

2011-11-23T16:20:00.001-05:00

Earlier this month, I attended UDS-P (the Ubuntu Developers Summit for 12.04 Precise Pangolin). 12.04 is an LTS release, or Long Term Support, meaning it will be officially supported on both the desktop and server for five years.

During the summit we reiterated our plans for Python on Ubuntu, both for 12.04 LTS, and our vision of Python two years from now for the next LTS, 14.04. I'm here to provide an update from my last report, which was written after UDS-O for 11.10.

As per those previous plans, we've removed Python 2.6 from Ubuntu 12.04. Now you have only Python 2.7 and 3.2. Dropping Python 2.6 may cause some inconvenience for data centers which like to upgrade only between LTS's but don't want to have to upgrade both their operation system and their Python version. Canonical services such as Launchpad and Landscape fall into this camp. The biggest problem is that Python 2.7 is not available for the last LTS, i.e. 10.04. To mitigate this, we decided to create a PPA containing Python 2.7 and a bunch of packages that services such as Launchpad will need to do their porting. This now exists, although it hasn't yet been tested with any development branches of Launchpad or Landscape as far as I'm aware. If you have your own data center porting task ahead of you, you can also use this PPA, and if there are additional packages you need for 10.04, you can create your own PPA which depends on ours, and build those extra packages there.

But that's all boring stuff. Let's have some fun!

The other decision we made concerns Python 3. I strongly feel that the Python community is very close to the tipping point in Python 3 adoption. Yes, there's still tons of code that needs porting, but we're seeing more and more activity every day. Even large frameworks such as Twisted are working on Python 3 branches. I wrote PEP 404 that (somewhat tongue in check) expresses official Python pronouncement on the migration path from Python 2.7 to Python 3. There will not be a Python 2.8, so the time to begin porting your code is now.

And it's really not that difficult. There are several great guides available to help you with porting your pure Python packages to Python 3, available on both python.org and python3porting.com. There are packages such as six available to ease the trickier bits of porting, and I have another blog post coming that describes my experiences in porting dbus-python. This latter task is made more complex by being a fairly involved C extension module. Again, both python.org and python3porting.com have useful guides for porting extension modules, although they have missed a few things. I wrote a mailing list article on some of the things that I had to do, but the blog post will include more detail.

Other distros such as Fedora are also starting to work on porting essential packages to Python 3, and here's where free software and open source rules. Our distros can work together to help nudge the Python community over that tipping point. Fedora is working from a bottom-up approach, where they are porting packages as needed by request, and we in Ubuntu are working top-down, by identifying several desktop applications that we'll port, along with their dependency stack. We'll both be pushing our changes to the upstream projects so every Python developer can benefit, no matter what platform they are on.

My long term vision is that Ubuntu 14.04 won't even have Python 2 on the CD images (if we even have CD images in two years), and we'll also move Python 2.7 off to universe. It will never completely go away, but the operating system and all the applications you get in a default install will run on Python 3. I'm much more sanguine about this goal now that I was even six months ago, but that's not to say it still won't be a lot of work.

The time is now to move to Python 3, if for no other reason than to fix all those pesky UnicodeErrors. I think there's a lot more of value in Python 3 too of course, and Python 3.3 will be even more awesome. I expect Python 3.3 or 3.4 to be in Ubuntu 14.04.

The place to be for all Python 3 porting efforts is the python-porting mailing list.

UPDATE: Fixed Python 3.3/3.4 plans to correctly read for Ubuntu 14.04 (thanks Nick!)

Python internationalization without exploding your brain

2011-10-12T19:07:00.001-04:00

Today I helped a colleague out with a pesky internationalization (i18n) problem in a Python 2 application. The problem was subtle enough that I thought I'd write this article to save others from the same brain exploding pain.

First, your Python application is multi-language friendly, right? I mean, I'm as functionally monolinguistic as most Americans, but I love the diversity of languages we have in the world, and appreciate that people really want to use their desktop and applications in their native language. Fortunately, it's not that hard to write good i18n'd Python code, and there are many tools available for helping volunteers translate your application, such as Pootle and Launchpad translations.

So there really is no excuse not to i18n your Python application. In fact, GNU Mailman has been i18n'd for many years, and pioneered the supporting code in Python's standard library, namely the gettext module. As part of the Mailman 3 effort, I've also written a higher level library called flufl.i18n which makes it even easier to i18n your application, even in tricky multi-language contexts such as server programs, where you might need to get a German translation and a French translation in one operation, then turn around and get Japanese, Italian,and English for the next operation.

In any case, my colleague was having a problem in a typically simple command line program. What's common about these types of applications is that you fire them up once, they run then exit, and they only have to deal with one language during the entire execution of the program, specifically the language defined in the user's locale. If you read the gettext module's documentation, you'd be inclined to do this at the very start of your application:

from gettext import gettext as _

gettext.textdomain(my_program_name)

then, you'd wrap translatable strings in code like this:

print _('Here is something I want to tell you')

What gettext does is look up the source string in a translation catalog, returning the text in the appropriate language, which will then be printed. There are some additional details regarding i18n that I won't go into here. If you're curious, ask in the comments, and I'll try to fill things in.

Anyway, if you do write the above code, you'll be in for a heap of trouble, as my colleague soon found out. Just running his program with --help in a French locale, he was getting the dreaded UnicodeEncodeError:

"UnicodeEncodeError: 'ascii' codec can't encode character"

I've also seen reports of such errors when trying to send translated strings to a log file (a practice which I generally discourage, since I think log messages usually shouldn't be translated). In any case, I'm here to tell you why the above "obvious" code is wrong, and what you should do instead.

First, why is that code wrong, and why does it lead to the UnicodeEncodeErrors? What might not be obvious from the Python 2 gettext documentation is that gettext.gettext() always returns 8-bit strings (a.k.a. byte strings in Python 3 terminology), and these 8-bit strings are encoded with the charset defined in the language's catalog file. Now, it's generally best practice in Python to always deal with human readable text using unicodes, converting any bytes to unicode as early as possible. This is traditionally more problematic in Python 2, where English programs can cheat and use 8-bit strings and usually not crash, since their character range overlaps with ASCII and you only ever print to English locales, which are compatible with ASCII. As soon as you use text that's not ASCII though, you're probably going to run into trouble. By using unicodes everywhere, you can generally avoid such problems, and in fact it will make your life much easier when you eventually switch to Python 3.

So the 8-bit strings that gettext.gettext() hands you will really just hurt you, and to avoid the pain, you'd want to convert them back to unicodes before you print them to stdout or a log file. However, converting to unicodes makes the i18n APIs much less convenient, so no one does it until there's way too much code to fix.

What you really want in Python 2 is something like this:

from gettext import ugettext as _

which you'd think you should be able to do, the "u" prefix meaning "give me unicode". But for reasons I can only describe as based on our misunderstandings of unicode and i18n back in the days this module was originally written, you can't actually do that, because ugettext() is not exposed as a module-level function. It is available in the class-based API, but that's a more advanced API that again almost no one uses. Sadly, it's too late to fix this in Python 2. The good news is that in Python 3 it is fixed, not by exposing ugettext(), but by changing the most commonly used gettext module APIs to return unicode strings directly, as it always should have done. In Python 3, the obvious code just works:

from gettext import gettext as _

What can you do in Python 2 then? Here's what you should use instead of the two lines of code at the beginning of this article:

_ = gettext.translation(my_program_name).ugettext

and now you can wrap all your translatable strings in _('Foo') and it should Just Work. Or you can use the flufl.i18n API, which always uses ugettext and always returns unicode strings.

The one-liner above was in fact the solution my colleague implemented to fix his bug, so I didn't really help him much. I just explained why his fix was the correct one and the original code was buggy. At least now you know too!

(The fact that this works correctly in Python 3 is yet another reason to switch to Python 3!)

Aside: It's really fun to install a desktop in a language you cannot read. Fortunately, French, like other Latin-drived languages, has enough familiarity that I can mostly get by with the standard Ubuntu installer. It also doesn't hurt that I've done about a bazillion installs of Ubuntu Oneiric (due out tomorrow!) so I pretty much know what each screen and prompt means even without going to Google Translate.

Also interesting was that I could never reproduce the crash when ssh'd into the French locale VM. It would only crash for me when I was logged into a terminal on the VM's graphical desktop. The only difference between the two that I could tell was that in the desktop's terminal, locale(8) returned French values (e.g. fr_FR.UTF-8) for everything, but in the ssh console, it returned the French values for everything except the LC_CTYPE environment variable. For the life of me, I could not get LC_CTYPE set to anything other than en_US.UTF-8 in the ssh context, so the reproducible test case would just return the English text, and not crash. This happened even if I explicitly set that environment variable either as a separate export command in the shell, or as a prefix to the normally crashing command. Maybe there's something in ssh that causes this, but I couldn't find it.

sbuild with local, newer, dependencies

2011-09-01T18:59:00.000-04:00

sbuild is an excellent tool for locally building Ubuntu and Debian packages. It fits into roughly the same problem space as the more popular pbuilder, but for many reasons, I prefer sbuild. It's based on schroot to create chroot environments for any distribution and version you might want. For example, I have chroots for Ubuntu Oneiric, Natty, Maverick, and Lucid, Debian Sid, Wheezy, and Squeeze, for both i386 and amd64. It uses an overlay filesystem so you can easily set up the primary snapshot with whatever packages or prerequisites you want, and the individual builds will create a new session with an overlaid temporary filesystem on top of that, so the build results will not affect your primary snapshot. sbuild can also be configured to save the session depending on the success or failure of your build, which is fantastic for debugging build failures. I've been told that Launchpad's build farm uses a customized version of sbuild, and in my experience, if you can get a package to build locally with sbuild, it will build fine in the main archive or a PPA.

Right out of the box, sbuild will work great for individual package builds, with very little configuration or setup. The Ubuntu Security Team's wiki page has some excellent instructions for getting started (you can stop reading when you get to UMT :).

One thing that sbuild doesn't do very well though, is help you build a stack of packages. By that I mean, when you have a new package that itself has new dependencies, you need to build those dependencies first, and then build your new package based on those dependencies. Here's an example.

I'm working on bug 832864 and I wanted to see if I could build the newer Debian Sid version of the PySide package. However, this requires newer apiextractor, generatorrunner, and shiboken packages (and technically speaking, debhelper too, but I'm working around that), so you have to arrange for the chroot to have those newer packages when it builds PySide, rather than the ones in the Oneiric archive. This is something that PPAs do very nicely, because when you build a package in your PPA, it will use the other packages in that PPA as dependencies before it uses the standard archive. The problem with PPAs though is that when the Launchpad build farm is overloaded, you might have to wait several hours for your build. Those long turnarounds don't help productivity much. ;)

What I wanted was something like the PPA dependencies, but with the speed and responsiveness of a local build. After reading the sbuild manpage, and "suffering" through a scan of its source code (sbuild is written in Perl :), I found that this wasn't really supported by sbuild. However, sbuild does have hooks that can run at various times during the build, which seemed promising. My colleague Kees Cook was a contributor to sbuild, so a quick IRC chat indicated that most people create a local repository, populating it with the dependencies as you build them. Of course, I want to automate that as much as possible. The requisite googling found a few hints here and there, but nothing to pull it all together. With some willful hackery, I managed to get it working.

Rather than post some code that will almost immediately go out of date, let me point you to the bzr repository where you can find the code. There are two scripts: prep.sh and scan.sh, along with a snippet for your ~/.sbuildrc file to make it even easier. sbuild will call scan.sh first, but here's the important part: it calls that outside the chroot, as you (not root). You'll probably want to change $where though; this is where you drop the .deb and .dsc files for the dependencies. Note too, that you'll need to add an entry to your /etc/schroot/default/fstab file so that your outside-the-chroot repo directory gets mapped to /repo inside the chroot. For example:

# Expose local apt repository to the chroot
/home/barry/ubuntu/repo /repo none rw,bind 0 0

An apt repository needs a Packages and Packages.gz file for binary packages, and a Sources and Sources.gz file for the source packages. Secure APT also requires a Release and Release.gpg file signed with a known key. The scan.sh file sets all this up, using the apt-ftparchive command. The first apt-ftparchive call creates the Sources and Sources.gz file. It scans all your .dsc files and generates the proper entries, then creates a compressed copy, which is what apt actually "downloads". The tricky thing here is that without changing directories before calling apt-ftparchive, your outside-the-chroot paths will leak into this file, in the form of Directory: headers in Sources.gz. Because that path won't generally be available inside the chroot, we have to get rid of those headers. I'm sure there's an apt-ftparchive option to do this, but I couldn't find it. I accidentally discovered that cd'ing to the directory with the .dsc files was enough to trick the command into omitting the Directory: headers.

The second call to apt-ftparchive creates the Packages and Packages.gz files. As with the source files, we get some outside-the-chroot paths leaking in, this time as path prefixes to the Filename: header value. Again, we have to get rid of these prefixes, but cd'ing to the directory with the .deb files doesn't do the trick. No doubt there's some apt-ftparchive magical option for this too, but sed'ing out the paths works well enough.

The third apt-ftparchive file creates the Release file. I shameless stole this from the security team's update_repo script. The tricky part here is getting Release signed with a gpg key that will be available to apt inside the chroot. sbuild comes with its own signing key, so all you have to do is specify its public and private keys when signing the file. However, because the public file from

/var/lib/sbuild/apt-keys/sbuild-key.pub

won't be available inside the chroot, the script copies it to what will be /repo inside the chroot. You'll see later how this comes into play.

Okay, so now we have the repository set up well enough for sbuild to carry on. Later, before the build commences, sbuild will call prep.sh, but this script gets called inside the chroot, as the root user. Of course, at this point /repo is mounted in the chroot too. All prep.sh needs to do is add a sources.list.d entry so apt can find your local repository, and it needs to add the public key of the sbuild signing key pair to apt's keyring. After it does this, it needs to do one more apt-get update. It's useful to know that at the point when sbuild calls prep.sh, it's already done one apt-get update, so this does add a duplicate step, but at least we're fortunate enough that prep.sh gets called before sbuild installs all the build dependencies. Once prep.sh is run, the chroot will have your overriding dependent packages, and will proceed with a normal build.

Simple, huh?

Besides getting rid of the hackery mentioned above, there are a few things that could be done better:

Different /repo mounts for each different chroot
A command line switch to disable the /repo
Automatically placing .debs into the outside-the-chroot repo directory

Anyway, it all seems to hang together. Please let me know what you think, and if you find better workarounds for the icky hacks.

ONOTE and OTONE - A musical project

2011-07-20T21:37:00.000-04:00

I'm starting a new musical project, which I'm calling OTONE and/or ONOTE. Actually, I've been working on this project for several years without realizing what I wanted to do with it. It coalesced in my mind when I thought of the acronyms above. Here's what they stand for:

The One Tune One Night Experiment (OTONE)
The One Night One Tune Experiment (ONOTE)

I'm not yet sure what the difference between the two are yet (though see below), but here's the idea behind the project.

If you're like me, you can easily sweat over a song and its recording for ever, tweaking the mix, or hearing another melody, or (worst of all) agonizing over every word of a lyric that was like pulling teeth in the first place. Sometimes you think if you just do one more take of the guitar, you can get it perfect, or oh! it just needs a little bit of tamborine right there. Sometimes the arrangement just doesn't sit quite right, or you know in your gut that lurking out there somewhere there's a better way to get from the bridge to the last chorus.

Well, I'm kind of frustrated with that because it can lead to never actually finishing a song and getting it out there for folks to hear. At some point you reach diminishing returns, where the little tweaks don't really improve the song enough. Probably most importantly, the whole thing can put the brakes on the creative process. I liken it to software maintenance vs. creating a new project from scratch.

Software maintenance is important, useful, and can be fun, but the juices really get flowing when you're starting a new project. You get this rush of an idea and your fingers can't type fast enough to translate them into code. It's this latter feeling that I want to better capture with music.

A brilliant friend and awesome drummer once said, "you have to get the crappy songs out in order to get to the good ones." A similar thought is "let 1000 flowers bloom." The more you write the easier it gets, and the more likely that out will come some cool songs. Thus OTONE/ONOTE.

The idea is to produce one song in one evening, and not sweat the details (too much). I have a small backlog of tunes that I've written using this approach (without realizing it) and plan on posting them to my website and social media over the next few weeks. Going forward, I'm going to set aside one night a week or fortnight, and see what happens. I'll post those too. I'll very likely put them under a non-commercial Creative Commons license, but I haven't yet decided whether to allow derivative works or not (I'm leaning toward "yes").

I encourage my other artistic friends to take a similar approach with their music, writing, art, etc. Feel free to use the #onote or #otone tags, but if you could link back to this article as your original impetus, I'd really appreciate it.

I should mention that some of the songs were not written in one night, but all were produced in one night. I'm trying to keep it under check, but sometimes I just can't help but twiddle the chorus now and then. :) Maybe that's the distinction though. OTONE can mean a song I wrote in one night, while ONOTE can be one evening devoted to producing just one song.

A couple of other things to keep in mind:

The songs will be rough. The whole point is not to cut and paste that one early bass note into the right place because that just slows you down. Unless it doesn't and you can still finish in one night. :)
Part of the reason for doing this is to better learn other software. I'm fairly proficient at Cubase 4 (but despise the dongle), and GarageBand is easy to get started (but much more difficult to do advanced stuff like arrange), Logic 9 is cool but I don't know it that well, and there are lots of free and alternative DAWs to experiment with. Doing quick one-night projects gives me some time to explore these.
Give me feedback and maybe I'll flesh out the good ones. There's no rule against coming back around to the ones you like and obsessing over the details later.
Remember: just because the sun's come up doesn't mean the night's over.

Now, go out and create!

Ignore at your own peril

2011-07-13T15:15:00.005-04:00

On Monday, I lost my home directory on my primary development machine. I'd had this machine for a couple of years but it was still beefy enough to be an excellent development box. I've upgraded it several times with each new Ubuntu release, and it was running Natty. I had decent sbuild and pbuilder environments, and a bunch of virtual machines for many different flavors of OS.

I'd also encrypted my home directory when I did the initial install. Under Ubuntu, this creates an ecryptfs and does some mount magic after you successfully log in. It's as close to FileVault as you can get on Ubuntu, and I think it does a pretty good job without incurring much noticeable overhead. Plus, with today's Ubuntu desktop installers, enabling an encrypted home directory is just a trivial checkbox away.

To protect your home directory, ecryptfs creates a random hex passphrase that is used to decrypt the contents of your home directory. To protect this passphrase, it encrypts it with your login password. ecryptfs stores this "wrapped" passphrase on disk in the ~/.ecryptfs/wrapped-passphrase file.

When you log in, ecryptfs uses your login password to decrypt wrapped-passphrase, and then uses the crazy long hex number inside it to decrypt your real home directory. Usually, this works seamlessly and you never really see the guts of what's going on. The problem of course is that if you ever lose your wrapped-passphrase file, you're screwed because without that long hex number, your home directory cannot be decrypted. Yay for security, boo for robustness!

When you do your initial installation and choose to encrypt your home directory, you will be prompted to write down the long hex number, i.e. your unwrapped passphrase. Here's the moral of the story. 1) You should do this; 2) You should remember where you put that magic piece of paper!

I did #1 but failed to do #2.

I'm not exactly sure how my ~/.ecryptfs directory got corrupted, but somehow the wrapped-passphrase file got deleted. I'm fairly certain that I didn't do it accidentally, but you never know. I suspect that there was some bad interaction with sbuild because the only thing I'd done right before the corruption, was to update my sbuild environment, and build a package in it. Note that if you follow the Ubuntu Security Team's instructions for setting up your sbuild (which I highly recommend), your unencrypted home directory will be exposed to sbuild's chroots. This is very convenient for normal work-a-day package building. However, it's possible that some bug in sbuild caused my ~/.ecryptfs directory to get corrupted. That's only a wild guess though; I'm now rebuilding my dev box on a fresh 1.5T drive starting with Oneiric alpha 2, so we'll see if this bug bites me again.

This time, I'm prepared. I stumbled upon a clever and secure way to never lose that piece of paper again. This technique should also allow you to store and share other securely encrypted data across any of your Ubuntu machines.

The first step is to set up Ubuntu One. You get 2GB of data free, so why not use it!? I have U1 set up on all my Ubuntu desktops, and it's a great place to stash things you might need everywhere. I've been considering storing things like my Firefox, Chromium, and Claws-Mail configs in U1 because I always want to keep them in sync. I already store my Timelog Tracker data there, so I can enter my time and accomplishments from any desktop.

(I generally do not keep most of my configuration files in U1 since I keep them under Bazaar.)

Now, it would be a very simple matter of just creating a subdirectory under ~/Ubuntu One for each machine that has an encrypted home directory, and then just copy the contents of ~/.ecyptfs to that. You need a separate subdirectory per machine because even if your login password is the same everywhere, that random hex number will be different.

You can do this today, but it may not be secure as you'd like, since the hex number that unlocks your encrypted home directory is "only" protected by your login password. Crack the password, and if there's ever a security hole in U1 (or some other legal or illegal machinations) that manages to expose your wrapped-passphrase file, then your entire home directory is vulnerable. For a little extra security, we can use encfs and fuse to create an unencrypted front-end to an encrypted directory inside U1. You would then copy your ~/.ecryptfs files to the unencrypted mount, and it would get stored with an additional level of encryption (and obfuscation) to U1. Now someone would have to crack two passphrases to get to your home directory.

The instructions on how to do this couldn't be simpler, and are outlined in this wiki article, and this one. For the lazy (like myself), here are the steps:

$ sudo apt-get install encfs fuse-utils

$ sudo modprobe fuse

$ sudo adduser <your-login> fuse

$ sudo sh -c "echo fuse >> /etc/modules"

$ mkdir ~/safe  # or whatever, contents are plaintext

$ mkdir "~/Ubuntu One/safe" # or whatever, contents are encrypted

$ encfs "~/Ubuntu One/safe" ~/safe

# You will be prompted for a new encfs passphrase, this is your extra security

$ mkdir ~/safe/<your-machine>

$ cp -a ~/.ecryptfs/* ~/safe/<your-machine>

And now just let U1 do its synchronization magic.

PEP 382 sprint summary

2011-06-22T13:12:00.001-04:00

So, yesterday (June 21, 2011), six talented and motivated Python hackers from the Washington DC area met at Panera Bread in downtown Silver Spring, Maryland to sprint on PEP 382. This is a Python Enhancement Proposal to introduce a better way for handling namespace packages, and our intent is to get this feature landed in Python 3.3. Here then is a summary, from my own spotty notes and memory, of how the sprint went.

First, just a brief outline of what the PEP does. For more details please read the PEP itself, or join the newly resurrected import-sig for more discussions. The PEP has two main purposes. First, it fixes the problem of which package owns a namespace's __init__.py file, e.g. zope/__init__.py for all the Zope packages. In essence, it eliminate the need for these by introducing a new variant of .pth files to define a namespace package. Thus, the zope.interfaces package would own zope/zope-interfaces.pth and the zope.components package would own zope/zope-components.pth. The presence of either .pth file is enough to define the namespace package. There's no ambiguity or collision with these files the way there is for zope/__init__.py. This aspect will be very beneficial for Debian and Ubuntu.

Second, the PEP defines the one official way of defining namespace packages, rather than the multitude of ad-hoc ways currently in use. With the pre-PEP 382 way, it was easy to get the details subtly wrong, and unless all subpackages cooperated correctly, the packages would be broken. Now, all you do is put a * in the .pth file and you're done.

Sounds easy, right? Well, Python's import machinery is pretty complex, and there are actually two parallel implementations of it in Python 3.3, so gaining traction on this PEP has been a hard slog. Not only that, but the PEP has implications for all the packaging tools out there, and changes the API requirements for PEP 302 loaders. It doesn't help that import.c (the primary implementation of the import machinery) has loads of crud that predates PEP 302.

On the plus side, Martin von Loewis (the PEP author) is one of the smartest Python developers around, and he's done a very good first cut of an implementation in his feature branch, so there's a great place to start.

Eric Smith (who is the 382 BDFOP, or benevolent dictator for one pep), Jason Coombs, and I met once before to sprint on PEP 382, and we came away with more questions than answers. Eric, Jason, and I live near each other so it's really great to meet up with people for some face-to-face hacking. This time, we made a wider announcement, on social media and the BACON-PIG mailing list, and were joined by three other local Python developers. The PSF graciously agreed to sponsor us, and while we couldn't get our first, second, third, or fourth choices of venues, we did manage to score some prime real-estate and free wifi at Panera.

So, what did we accomplish? Both a lot, and a little. Despite working from about 4pm until closing, we didn't commit much more than a few bug fixes (e.g. an uninitialized variable that was crashing the tests on Fedora), a build fix for Windows, and a few other minor things. However, we did come away with a much better understanding of the existing code, and a plan of action to continue the work online. All the gory details are in the wiki page that I created.

One very important thing we did was to review the existing test suite for coverage of the PEP specifications. We identified a number of holes in the existing test suite, and we'll work on adding tests for these. We also recognized that importlib (the pure-Python re-implementation of the import machinery) wasn't covered at all in the existing PEP 382 tests, so Michael worked on that. Not surprisingly, once that was enabled, the tests failed, since importlib has not yet been modified to support PEP 382.

We also came up with a number of questions where we think the PEP needs clarification. We'll start discussion about these on the relevant mailing lists.

Finally, Eric brought up a very interesting proposal. We all observed how difficult it is to make progress on this PEP, and Eric commented on how there's a lot of historical cruft in import.c, much of which predates PEP 302. That PEP defines an API for extending the import machinery with new loaders and finders. Eric proposed that we could simplify import.c by removing all the bits that could be re-implemented as PEP 302 loaders, specifically the import-from-filesystem stuff. The other neat thing is that the loaders could probably be implemented in pure-Python without much of a performance hit, since we surmise that the stat calls dominate. If that's true, then we'd be able to refactor importlib to share a lot of code with the built-in C import machinery. This could have the potential to greatly simplify import.c so that it contains just the PEP 302 machinery, with some bootstrapping code. It may even be possible to move most of the PEP 382 implementation into the loaders. At the sprint we did a quick experiment with zipping up the standard library and it looked promising, so Eric's going to take a crack at this.

This is briefly what we accomplished at the sprint. I hope we'll continue the enthusiasm online, and if you want to join us, please do subscribe to the import-sig!

Python plans for Ubuntu 11.10 and 12.04

2011-05-19T14:27:00.009-04:00

TL;DR: Ubuntu 12.04 LTS will contain only Python 2.7 and 3.2, while Ubuntu 11.10 will contain Python 3.2, 2.7 and possibly 2.6, but possibly not.

Last week, I attended the Ubuntu Developer Summit in Budapest, Hungary. These semi-annual events are open to everyone, and hundreds of people participate both in person and remotely. Budapest's was called UDS-O, where the 'O' stands for Oneiric Ocelot, the code name for Ubuntu 11.10, which will be released in October 2011. This is where we did the majority of planning for what changes, new features, and other developments you'll find in the next version of Ubuntu. UDS-P will be held at the end of the year in Orlando, Florida and will cover the as yet unnamed 12.04 release, which will be a Long Term Support release.

LTS releases are special, because we make longer guarantees for official support: 3 years on the desktop and 5 years on the server. Because of this, we're making decisions now to ensure that 12.04 LTS is a stable, confident platform for years to come.

I attended many sessions, and there is a lot of exciting stuff coming, but I want to talk in some detail about one area that I'm deeply involved in. What's going to happen with Python for Oneiric and 12.04 LTS?

First, a brief summary of where we are today. Natty Narwhal is the code name for Ubuntu 11.04, which was released back in April and is the most recent stable release. It is not an LTS though; the last LTS was Ubuntu 10.04 Lucid Lynx, release back in October 2010. In Lucid, the default Python (i.e. /usr/bin/python) is 2.6 and Python 2.7 is not officially supported or available. Python 3.1 is available for Lucid, but not installed by default.

In Natty, the default Python is 2.7 with 2.6 still being officially supported. This means that you can have both Python 2.6 and 2.7 on your Natty machine, and where possible, packages were built for both Python versions. Where this was not possible, you'll almost always find a package for Python 2.7 instead of 2.6. Natty also has Python 3.2 and 3.1 available, with 3.2 being the default.

Two more bits of background are useful to know. In Ubuntu (inherited from Debian, where most packages are initially developed), we separate Python 2 support and Python 3 support into separate "stacks", meaning entirely separate binary packages even if the source packages are the same. This has many benefits, including allowing a system administrator to only install the Python 2 stack, or only the Python 3 stack if they want. It also makes for our eventual transition to Python 3 much easier, because packages don't need to be renamed. So for example, if you see a package named "python-foo" you know this is the Foo package for Python 2. You might also see a "python3-foo" which would be the Python 3 version of Foo.

Also, many packages are built for all supported versions in a particular stack. So for example, if we want to make Foo available for both Python 2.6 and 2.7, we'll include support for both in a single python-foo package. Pure Python source code is generally easily shared, so this reduces the duplication (more on this in another blog posting), however extension modules, which are usually implemented in C, must be compiled twice, and both shared libraries must be included in the same binary package. This means if we are supporting package Example which contains an extension module, for both Python 2.6 and 2.7, the binary package will contain two shared libraries, effectively doubling the disk consumption for extension module support.

Keep all that in mind as I describe what comes next!

To understand our plans for Oneiric, it's first useful to explain our goals for the next LTS, since we'll be using 11.10 as a transitional cycle. For 12.04 LTS, we want to support just one Python 2 version and just one Python 3 version. Because Python 2.6 is in security-fix only mode in upstream Python, we want to drop support for it in 12.04 LTS. This will also allow us to reclaim some space on the installation CDs because we won't need to include extension modules compiled for both Python 2.6 and 2.7. Last cycle we calculated the savings at about 10MiB, which is not insignificant on a standard CD.

For 12.04 LTS, the only Python 3 version we want to support is Python 3.2. Our thinking here is that there really isn't much code out there that depends on Python 3 yet, and Python 3.2 has many very useful features that make it (IMO) the first Python 3 to start basing production quality code on. We're going to put our money where our mouth is here, and I'll write more on that later too.

The decision to drop Python 3.1 support for 12.04 LTS is, as far as I know, completely uncontroversial, so this will happen in Oneiric. And because Python 3.3 will not be released before 12.04 LTS, we will be making that change very soon, so as to provide the longest possible period of stabilization and porting between now and April 2012. If you've been holding off on developing for Python 3, now is a great time to jump in!

Dropping Python 2.6 is somewhat more controversial for several reasons. First, in Ubuntu, we rely very heavily on Debian for the majority of packages, and we strongly encourage our developers to submit patches and new packages in Debian first, with requests for syncing to Ubuntu once they're available in Debian. My take on this relationship is that, because Ubuntu has strictly timed releases while Debian has a "release-when-ready" policy, we can often use Ubuntu's development cycle to blaze a trail (sometimes on the bleeding edge ;) but it's always critical to ensure that wherever possible, Debian contains the authoritative versions of our packages. Now that Debian has released Squeeze and is working on its Wheezy release, it's time for us to push our Ubuntu changes back into Debian, and work on getting the latest upstream versions into Debian, while syncing back to Ubuntu. For this reason, we just don't want to get too far ahead of Debian in our Python support. Our plan therefore is to continue to support Python 2.6 until Debian has completed their transition to Python 2.7 as the default version (they already support both, but Python 2.6 is still the default).

Our timeline therefore is to make a final decision on Python 2.6's fate for Oneiric by feature freeze. If Debian still hasn't completed their transition by then, we'll keep Python 2.6 for Oneiric and drop it as soon as the archive opens for 12.04 LTS. This should be pretty low risk for us, and it helps us better align ourselves with Debian, which is always a good thing! If you feel so inclined, you can help by working on some of the blocker bugs for Debian's transition to Python 2.7, as we will also be doing.

Another reason to be cautious about dropping Python 2.6 is because many of the services in our own data center are not yet ported to Python 2.7. Probably the biggest of such services is Launchpad. Our data center machines always run the previous LTS, and Lucid does not have Python 2.7, so this makes for a kind of Catch-22 for the Launchpad team. To address this, we've created a quasi-official PPA into which we'll backport Python 2.7 and many dependent modules. The Launchpad team can then use this PPA to work on their own port to Python 2.7 in plenty of time for 12.04 LTS. Anybody else out there who wants to do the same can also use our PPA, and if they need additional modules backported, they can create their own PPA which depends on ours for the base support.

So, that's what's happening, and why. Feedback is of course invited here, on the ubuntu-devel mailing list, or to me directly. If you want to follow along, you can take a look at the blueprint describing these changes, and more.

In the next articles, I plan to discuss how we're going to get to Python 3 only on the Ubuntu CDs, and how we're going to help with the migration to dh_python2. Cheers!

My Natty Upgrade Test

2011-04-15T09:21:00.005-04:00

Ubuntu 11.04 (code name: Natty Narwhal) beta 2 was just released and the final release is right around the corner. Canonical internal policy is that we upgrade to the latest in-development release as soon as it goes beta, to help with bug fixing, test, and quality assurance.

Now, I've been running Natty on my primary desktops (my two laptops) since before alpha 1, and I've been very impressed with the stability of the core OS. One of my laptops cannot run Unity though, so I've mostly been a classic desktop user until recently. My other laptop can run Unity, but compiz and the wireless driver were too unstable to be usable, that is until just before beta 1. Still, I diligently updated both machines daily and at least on the classic desktop, Natty was working great. (Now that beta 1 is out, the wireless and compiz issues have been cleared up and it's working great too.)

The real test is my beefy workstation. This is a Dell Studio XPS 435MT 12GB, quad-core i7-920, with an ATI Radeon HD 4670 graphics card, running dual-headed into two Dell 20" 1600x1200 flat panel displays. During the Maverick cycle I was a little too aggressive in upgrading it, because neither the free nor the proprietary drivers were ready to handle this configuration yet. I ended up with a system that either couldn't display any graphics, or didn't support the dual heads. This did eventually all get resolved before the final release, but it was kind of painful.

So this time, I was a little gun shy and wanted to do more testing before I committed to upgrading this machine. Just before Natty beta 1, I dutifully downloaded the daily liveCD ISO, and booted the machine from CD. On the surface, things seemed promising. I had compiz and Unity, but no dual-head. Running from the liveCD is fairly transient though; it doesn't save enough state between reboots to be a fair test of the machine.

How could I get a true test of Natty that would give me my normal development environment, run natively on the hardware, and yet be easily discarded if it turned out to not be ready yet? Here's where USB hard drives and virtual machines come in.

I'm a very heavy user of virtual machines. With plenty of disk space on this 1.5TB drive, I have maybe a dozen VMs. This let's me run the stable Ubuntu release as a host, and have VMs for several versions of Debian, several older versions of Ubuntu, and even flavors of Windows, Solaris, FreeBSD, and Fedora. (Let me know if you've successfully made a libvirt guest running as a hackintosh for OS X!). Some day I'll post about safe, copy-on-write backing disks.

Now, the cool thing about these VMs is that I bridge the network and give all of them unique IP addresses on my internal network, so for all intents and purposes, they are real machines. Most of them I access only through ssh, but virt-manager gives a nice graphical desktop when you need it. So I can mostly test just about anything on any x86 or amd64 operating system, with my full normal development environment.

So here was my thinking: create a Natty VM that wouldn't use the normal virtual disk living in a file on the host file system. Instead, its disk would actually be on a 320GB USB external drive that I had laying around. I'd install Natty to this VM using the daily liveCD, get my full development environment up and running, then I'd shut down the host, and reboot it to the USB drive. This would give me a persistent Natty running natively on the hardware, and if it didn't work out, I'd just reboot back to the internal drive. No fuss, no muss.

Let's cut to the chase: I eventually got this to work, and spent a day putting Natty through its paces. After few pre-beta-1 updates, I was satisfied that all my hardware was working great, including full resolution dual-head with Unity. That gave me the confidence to upgrade the host OS running on the internal drive, and I've been happily using it since then.

Of course, all was not smooth sailing. I did run into a few hitches along the way.

When the primary host operating system mounted the USB drive, it often got different /dev device assignments. Sometimes it would come up as /dev/sdc and sometimes as /dev/sdg. The problem with this is that libvirt wants to use the /dev name as the device for the VM's file system, so when this changed, I'd have to go into the VM's configuration and fix the storage device path. This is kind of a pain through the virt-manager U/I since the storage device has to be deleted and then re-added. You can't just change the path (I'll bet it can be done by editing the .xml configuration file directly).

When creating the storage device for the VM, you have several options for the disk type. At first, I naturally chose USB disk, since in fact, the physical device was a USB drive. But this tended to cause the Ubiquity installer no end of trouble. It hung very early in the installation process. I never did investigate that or get it working, but I did realize that if you use a virtio disk and point it at the USB device path (see above), the installer worked perfectly.

You have to be sure your host machine's BIOS can boot off of a USB drive. Luckily this is a new enough machine that it works fine, but older machines may not be able to boot off a USB drive. I think you could probably just do everything I've done above, but installing that spare drive internally and it would work just as well. But of course opening up the case is a PITA. :)

Grub was a little finicky. I first tried this just a few days before beta 1, and every time I installed the OS to the USB drive, then booted off the drive I'd immediately be dropped into a grub> prompt, with no way to complete the boot process. It was like grub was not getting installed correctly, but at the time I also wasn't sure whether this idea could even work. Note that I did not want to add the USB drive to the chain loader on the primary internal drive, I just wanted to hit F12 during the BIOS phase and select the USB drive to boot off of. I thought this *should* work, but it didn't and I thought maybe something about the virtual environment caused grub to fail when run natively. I asked around on our internal tech list, but I think this was really a transient problem with the pre-beta ISOs. I tried again a day or so later with a new liveCD and everything worked perfectly.

Of course, when booting off the USB drive natively, a few things are different. The virtual machine has a different MAC address, and thus a different host name and IP address than the native host. So after booting off the USB drive, you have to fiddle with a few things to get that to line up, depending on how you configure your machine. E.g. /etc/hosts was incorrect. These minor problems didn't really slow me down though.

Ignoring the transient, inexplicable grub issue, this turned out to be a very nice way to test drive the new Ubuntu version on the native hardware, in a completely non-destructive way. I'll bet it would work to test drive other operating systems as well, and if one of my fantastic readers gives me a clue about how to build a hackintosh in a VM, I'll give that a try for the fun of it. And of course I'll let you know how it goes!

Release that rewrite on 11.11.11

2011-04-04T12:24:00.003-04:00

I know that the Mailman 3 project is not alone in procrastinating getting out a release of its major rewrite. It's hard work to finish a rewrite on your own copious spare time. I was just chatting with Thomas Waldmann of the Moin project on IRC, and he lamented a similar story about the Moin 2 release. Then he said something that really made me sit up straight:

<ThomasWaldmann> 11.11.11 would be a great date for something :)

Yes, it would! We have the 2011 Google Summer of Code happening soon (students, you have until April 8th to submit your applications) so many free and open source software projects will get some great code coming soon. And November is far enough out that we can plan exactly what a "release" means. Here's what I propose:

Let's make November 11, 2011 the "Great FLOSS Release Day". If you're working on an open source project undergoing a major new version rewrite, plan on doing your release on 11.11.11. It can be a beta or final release, but get off your butts and make it happen! There's nothing like a good deadline to motivate me, so Mailman 3 will be there. Add a comment here if you want your project to be part of the event!

Charming Snakes and Shaving Yaks

2011-03-28T18:36:00.007-04:00

For the last couple of days I've been debugging a fun problem in the Ubuntu tool called Jockey. Jockey is a tool for managing device drivers on Ubuntu. It actually contains both a command-line and a graphical front-end, and a dbus backend service that does all the work (with proper authentication, since it modifies your system). None of that is terribly relevant to the problem, although the dbus bit will come back to haunt us later.

What is important is that Jockey is a Python application, written using many Python modules interfacing to low-level tools such as apt and dbus. The original bug report was mighty confusing. Aside from not being reproducible by myself and others, the actual exception made no fricken sense! Basically, it was code like this that was throwing a TypeError:

_actions = []

# _actions gets appended to at various times and later...

for item in _actions[:]:

# do something

Everyone who reported the problem said the TypeError was getting thrown on the for-statement line. The exception message indicated that Python was getting some object that it was trying to convert to an integer, but was failing. How could you possible get that exception when either making a copy of a list or iterating over that copy? Was the list corrupted? Was it not actually a list but some list-like object that was somehow returning non-integers for its min and max indexes?

To make matters worse, this little code snippet was in Python's standard library, in the subprocess module. A quick search of Python's bug database did reveal some recent threads about changes here, made to ensure that popen objects got properly cleaned up by the garbage collector if they weren't cleaned up explicitly by the program. Note that we're using Python 2.7 here, and after some reading of the tracker issues and the Python subprocess.py code, I just couldn't see any problem here, or at least none that could possibly lead to the error being seen.

When I originally looked at the Launchpad bug report, that was about as far as I got. I couldn't see any possible way this bug could happen, and I couldn't reproduce it, so I set the bug to Incomplete. Unfortunately, it kept hitting Ubuntu Natty beta testers so it wasn't going to go away. Fortunately, Martin Pitt found a test recipe, with which I could reproduce the bug 100% of the time. Yay! At least it probably wasn't going to be a race condition.

How to debug this though? Normally, I'd just attach gdb to the thing and start tracing, but the problem was that when I set the Jockey dbus backend to use the debug version of Python, the error went away (or rather morphed into something that was really unrelated). Here's where I started to sharpen up my favorite yak shaving blade.

The parable about shaving yaks is so entirely appropriate to problems like this that I'll take a quick detour. One of my (and my brother's) favorite shows in the 90's was Ren and Stimpy. That show was groundbreaking, and you can see elements of it in just about every NickToons cartoon on cable today. Some of the episodes are brilliant and others are horrible but as a whole, Ren and Stimpy are an undeniable classic of American animation. In one particular awesome episode, Ren and Stimpy celebrate Kilted Yaks-eve where a Yak comes up from the tub drain and shaves his stubble, leaving behind a present of the highly desired shaving scum for Stimpy in the morning. Based on this episode (most likely!) the programming term "yak shaving" typically means having to take detour after meaningless detour before you can actually solve the problem you're having.

In the context of this bug, the attempt to use the debug build of Python in the Jockey dbus backend was the first yak to shave. Because I wanted to get more information out of the process, I tried to attach to a running backend process, but this proved to be quite difficult. I was debugging this in a 64bit virtual machine, and gdb+debug-python was just not cooperating.

Now the question is: do you debug that problem (or at least get to a place where you can decide to file a bug or not), or do you punt and try a different approach? The former is most definitely shaving the yak; it gets you no closer to fixing the original problem, but certain wastes enough of your time to make you seem productive. Of course, it's inevitable that if you do follow that secondary bug, it'll lead to a third one, and a fourth one, and so on until you're deep in the shaving scum and no closer to a solution to the original problem. You need to be constantly on guard against these types of yak, er, rabbit holes.

I'll spare you the gory details about hitting bugs in my text editor, bugs in building Python on a multiarch system, and other paths that all lead to smooth chinned yaks, but not to happy Pythons. When I punted on trying to attach to the Jockey backend, I actually decided to try to figure out exactly where the TypeError was occurring by using a twist on old-fashioned print statements. By grepping the Python source I found about a dozen occurrences of the error message "an integer is required". Which one was getting tripped? I added a little marker text to every such occurrence and re-built a Python package to aid in debugging.

Here's where I did detour to yak shaving. The Ubuntu source package for Python itself, runs the full test suite on every build. That's great for ensuring a high quality Python package, but it's terrible for turnaround time when experimentally hacking Python. Building the package with DEB_BUILD_OPTIONS="nocheck nobench" is supposed to work, but for some reason did not with my sbuild environment. I guess debugging that would have been like shaving the yak's left check, but instead it was so much easier to shave his right check. So I spent some time with my razor chopping out huge sections of the debian/rules file so that Python 2.7 would build as fast as possible. While a seemingly meaningless task, this did actually help quite a bit since I was able to try out ideas with a much shorter turnaround.

Anyway, with many iterations on the exception marking idea, I finally nailed down the culprit. It was in the Python C API function PyInt_AsLong(), but even here I was unsure which arm of the conditional was getting triggered. Another round of hack-build-scp-reset-test and I found what I suspected: PyInt_AsLong() was getting passed an object that could not be turned into an integer. But what was that object?

So now back to the original problem with gdb. To solve this I downloaded and built a 32bit virtual machine, and was able to reproduce the bug there. Fortunately, in the 32bit environment I was much more successful in attaching to a running backend Jockey dbus process, and even though I didn't have the Python source available (and no, I won't talk about where that particular yak shaving detour led), I could fairly easily print the objects in the debugger at the offending code, where I learned that PyInt_AsLong() was getting called with None as its argument. And yeah, you can't turn None into an integer in Python!

But, how was this function getting called with None? Popping up the call stack led me eventually to PyArg_Parse(), an older C API function that parses a Python tuple into a set of objects based on some format flags. This is used when implementing Python functions in C, to parse an argument list. Popping up the stack again led me into some python-apt code. python-apt is a C++ library that exposes the APT system to Python programs. It's fairly mature and robust, but I wasn't as familiar with all its dark corners as I was with Python.

Now, my first instinct is never that I've found a bug in Python. That's not to say they don't exist, but just that Python has been around so long, is so well tested, and used so extensively that I'm always suspicious in cases like this (i.e. strange, inexplicable bugs that make no sense) of extension modules and third party code. And indeed, my sleuthing has led me to python-apt, exactly the kind of complex Python extension module that can have odd lurking bugs in it. Still, the problem I was now faced with was this: the call stack led me into a code path that had nothing to do with iterating over a list, or copying that list. So, what gives?

Well, it's useful to know how Python exceptions work at the C level. In general terms, when some C code raises an exception, it basically sets some global state, and then returns error codes up the stack until either something catches it and deals with it, or it percolates up to Python's top eval loop. The key thing here is that there are usually two states that exist: the global exception value currently in effect, and an error code that gets returned up the C call stack. Usually this return code is a zero or one, but it can also be NULL or -1 in certain cases. The Python C API documentation is very good at describing these.

So now that I looked closely at the python-apt code, I could see what what going on and it all began to make sense! The python-apt package was pulsing a progress meter, set up as a callback by the client of the python-apt code. Meaning, python-apt couldn't really control what this callback would return. python-apt expected the callback to return either Python's True or False, but it could return anything, including nothing! In Python, True can be coerced into the integer 1 and False into the integer 0, and python-apt wanted an integer, so indeed the call stack lead right to a call to PyArg_Parse() to turn the callback's return object into an integer. What happens if the callback didn't return something that could be turned into an integer, or worse, didn't return anything at all?

In Python, a function always returns something, even if there is no explicit return statement. In those cases, None is implicitly returned. Yes, you see it now. And if not, there was this clue in the python-apt code: "most of the time the user who subclasses the pulse() method forgot to add a return {True,False} so we just assume he wants a True." Translated: Hey guy! You forgot to add a "return True" or "return False" to your pulse() method, and it probably fell of the end, giving us None, which we dutifully passed to PyArg_Parse().

PyArg_Parse() did its thing when getting None, by correctly setting the global exception state to the TypeError, and returning a zero code to indicate an error occurred. But looking at the python-apt code, it recognizes the error code, but forgets that some global exception state was set! Meaning, even though python-apt was ignoring the exception, Python still knew about it. But because control wasn't being passed on up to the Python eval loop, the exception was just lurking there like a angry unshaven yak, waiting to be discovered. And in fact, the next time Python itself checked the exception state was in, yep, the for-loop iterating over the perfectly fine list object. Python hits the for-loop, finds this orphaned TypeError lurking there and raises it, in a place that has literally nothing to do with the original exception.

The fix is a one-liner. In python-apt, where its ignoring any exception returned by PyArg_Parse(), it must both swallow the error code (which it was doing), and clear the global exception state (which it was notdoing). By adding a call to PyErr_Clear(), python-apt was keeping the interpreter's state consistent, and properly ignoring the argument parsing error, thus fixing the bug.

As I mentioned to my colleague Colin Watson, this did turn out to be a fun one to debug, though not as "fun" as one he recently worked on.

Happy Yak Shaving.

What We Do

2010-09-09T18:15:00.017-04:00

My friends and family often ask me what I do at my job. It's easy to understand when my one brother says he's a tax accountant, but not so easy to explain the complex world of open source software development I live in. Sometimes I say something to the effect: well, you know what Windows is, and you know what the Mac is right? We're building a third alternative called Ubuntu that is free, Linux-based and in most cases, much better. Mention that you won't get viruses and it can easily breathe new life into that old slow PC you shudder to turn on, and people at least nod their heads enthusiastically, even if they don't fully get it.

I've been incredibly fortunate in my professional career, to have been able to share the software I write with the world for almost 30 years. I started working for a very cool research lab with the US Federal government while still in high school. We had a UUCP connection and were on the early Arpanet, and because we were funded by the US taxpayer, our software was not subject to copyright. This meant that we could share our code with other people on Usenet and elsewhere, collaborate with them, accept their suggestions and improvements, and hopefully make their lives a little better, just as others around the world did for us. It was free and open source software before such terms were coined.

I've never had a "real job" in the sense of slaving away in a windowless cube writing solely proprietary software that would never see the light of day. Even the closed source shops I've worked have been invested somehow in free software, and with varying degrees of persuasion, have both benefited from and contributed to the free and open source ecosystem. Thus, in many ways, my current position at Canonical feels like the perfect fit and ultimate destination for my background, skills, and passion. Canonical is open source to its core. Its central mission, as articulated by our founder Mark Shuttleworth is "to bring free software to the widest possible audience, powered by services rather than licenses, in tune with a world that was moving to services as the core economic model of a digital world."

To me, the free and open source ethos goes much deeper than just the software we write. It's about collaboration, community, sharing, learning, teaching, and having a truly positive impact on the world. It's about empowering individuals to realize their full potential, to give them the opportunity to build a merit based reputation, to carve out their own areas of interest and expertise, and relate that to the larger society, to know that they make a difference, and that their opinions and contributions matter. Open source is about having the courage of your convictions, but embracing humility to admit when you're wrong and someone else has a better idea. To encourage and actively seek out consensus, but also to cultivate a thoughtful and compassionate process for making the hard decisions when consensus can't be reached. It's about spreading enthusiasm and rallying others to your side sometimes, and at other times humbly and joyfully embracing other points of view.

I could go on with all the mushy goodness, but let's look at a few areas where the work I do for Canonical directly contributes to the broader free and open source ecosystem.

Python

I've been a core Python developer since about 1995, and have served several times as release manager for major new versions. As part of my job on the Ubuntu Platform Foundations team, I'm keenly concerned with issues involving Python's deployment on Ubuntu and its upstream ancestor Debian. One of the distinction between Ubuntu/Debian and other Linux distributions is that Ubuntu/Debian often provides more than one Python version in a release. For example, in Natty Narwhal, it's likely that Ubuntu will officially support Python 2.6, 2.7, 3.1, and 3.2. The reason for this is that it does take some porting effort for applications to upgrade their version of Python, and this eases the transition for those applications. But it's complicated by the fact that upstream Python doesn't really support multiple installed versions out of the box. Recent work I've done on PEP 3147 and PEP 3149 improve this situation greatly, by allowing multiple versions of Python to more peacefully coexist. As is typical, I try to push my changes needed for the Ubuntu platform as far upstream as possible, because that benefits the most users of the open source software. In addition, a huge number of users get their Python interpreter from their operating system vendor (this is at least the case on all Linux variants as well as Mac OS X), so work done to improve the Python experience on Ubuntu directly and positively impacts users on Debian and other Linux distributions, as well as the general Python community at large.

GNU Mailman and Launchpad

I've been the lead developer for GNU Mailman since the late 90's, and I've been overwhelmed to see it become the predominant free software mailing list manager. When I was working on the Launchpad team at Canonical, my primary responsibility for the first few years was to integrate mailing lists with Launchpad, and of course Mailman was the obvious choice for underlying technology. At first, Launchpad was closed source software, but the intent was always to release it as open source, and it was with much joy that we saw Launchpad released under a free software license in the summer of 2009. It was not a design goal of the original Mailman software to be easily integrated with external systems, so a good bit of the work I did on Launchpad at the time directly benefited upstream Mailman users by making it easier for ourselves, and others, to do this type of integration. Now that I'm no longer on the Launchpad team, I work on Mailman 3 exclusively in my spare time. My experiences with that integration effort had a direct influence on the architecture of Mailman 3, and I have a goal of swapping the current Mailman 2.1 technology in Launchpad with Mailman 3. All the relevant work to do this is fed back into the upstream project, to the benefit of the larger free software community, since many other people also want a better way to integrate mailing list functionality with their web sites.

My Mailman and Launchpad work also saw many spin-offs of helpful utilities and libraries which can be used by the much wider community of open source developers. The lazr suite of Python libraries has applicability outside Launchpad and I use many of them in my extra-curricular projects, evangelizing them along the way. Many GNU Mailman utilities have also been spun off into their own libraries, and these are now or will soon be available on the Debian and Ubuntu platforms, just as they are now available on the Python Package Index. The work I'm doing, along with my Canonical and Ubuntu colleagues to reduce the barriers to opportunistic participation in projects is key to this ecosystem. We're making it much easier for others to find and participate in the projects that interest them, at whatever level they choose.

UDD

Along those lines, as part of my work on Ubuntu Platform Foundations, I've become quite enthusiastic about technology we're calling Ubuntu Distributed Development. When I moved to the Platform team, I knew next to nothing about the art of packaging, which is how upstream projects are turned into installable and manageable chunks that people can put on their Ubuntu (and Debian) machines. Packaging has a rich tradition, but it's pretty esoteric and to most people who don't package, it's a black art. Even now that I know how to package software, I still think it's way too magical. UDD is a set of tools and procedures that aim to bring packaging to the masses, by reducing the magic and exposing the important parts in much more familiar tools. Every open source developer sooner or later (hopefully much sooner!) learns how to use a revision control system. Such systems allow developers to manage their code in a principled and rigorous way, by keeping detailed records about the changes to their software over time. Revision control systems are an absolutely fundamental and required tool in the open source developer's toolbox.

The Bazaar distributed revision control system was written by Canonical, but is now a community-driven free software project. It's ease of use, superior quality, and high degree of flexibility and extensibility make it a very attractive choice for projects looking to use the next generation in revision control systems. Bazaar also integrates very nicely with Launchpad. UDD are a set of extensions to bring packaging to the Bazaar suite, so that the very tool that upstream software developers use dozens of times a day, are also the tools they will use to create packages for making their software available to the mass of Ubuntu users. I've been using Bazaar for years now, and have written a few plugins and patches. I'll soon be helping to lead an effort to improve the UDD workflows and more widely expose the benefits of UDD to more and more Ubuntu developers.

Platform

I love my work on Platform because it allows me to have a small hand in lots of different upstream free and open source projects. After so many years of hacking on such projects of all different stripes, I'm fairly good at looking at and understanding good open source code, debugging and fixing problems, and interacting with the upstream communities to get those patches pushed higher up the stack. Bug trackers, mailing lists, IRC, and revision control systems are the core technologies needed for this work, and I'm comfortable interacting on all of them. Part of our job as Platform developers is, in my opinion, to engage the upstream projects, so that the fixes and changes we need to make in order to provide the absolute best experience to Ubuntu users, are available even to those folks who don't get their software from Ubuntu. This to me is the the true meaning of being a free and open source developer for a company whose mission is to make free and open source available to the mass of computer users. Our aim is not just to make software that competes on price, or liberty, but also on quality and "sex appeal" - we want our users to flock to Ubuntu not just because it costs nothing, but because it's so compelling that you can't help but love it.

import this and The Zen of Python

2010-06-08T15:14:00.005-04:00

Richard Jones is working on a talk for PyCon Australia and asked me about the history of the Zen of Python, Tim Peters' eternal words of wisdom, often quoted, on the essential truths of Python. At first, I couldn't find a reference to the first publication of this list, but then I did a better search of my archives and found that it was originally sent to the python-list mailing list on June 4, 1999, under the subject "The Python Way".

Interestingly enough, because I couldn't find that first reference immediately, I went back into my archives and researched the "this" module. Did you know that if you type the following at a modern Python interpreter, you get the Zen of Python?

% python

Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)

[GCC 4.4.3] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import this

The Zen of Python, by Tim Peters

Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Complex is better than complicated.

Flat is better than nested.

Sparse is better than dense.

Readability counts.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

Errors should never pass silently.

Unless explicitly silenced.

In the face of ambiguity, refuse the temptation to guess.

There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.

If the implementation is easy to explain, it may be a good idea.

Namespaces are one honking great idea -- let's do more of those!

The story behind "import this" is kind of funny, and occurred totally behind the scenes, so I thought it might be interesting to relate how it happened. Maybe something to add to Guido's History of Python blog.

Anyway, back in fall 2001, Foretec was organizing the International Python Conference #10 (IPC 10, precursor to Pycon). Foretec was a conference organizing company owned by CNRI, which earlier had employed Guido, Fred, Jeremy and myself, and which by 2000 we had left to form Pythonlabs at various friendly (and occasionally disfunctional) corporate homes. By September 13, 2001 (yes, how weird!) we were working for our friends at Zope Corporation (perhaps not yet rebranded from Digitial Creations), but Foretec was still organizing the IPCs. They wanted to have a slogan for the conference, which could be printed on a t-shirt, and wanted to gather submissions from the Python community. Pythonlabs agreed to judge the entries and select a winner. However, I think Guido's wife was due any day and he didn't have much time or energy to go through the entries.

We got something like 500 entries, almost all of them terrible. Tim's exact words were "Jeez Louise, I can't look at these for more than 5 minutes w/o my brain turning to mush" and yet he still managed to do an initial cut down to something like 130 entries. While we had agreed to choose a winner, we procrastinated until the last minute and by then it was obvious that Tim and I were the only ones crazy enough to still care. Tim suggested that we trade the list back and forth between the two of us, each cutting the list in half until there was just one left. Tim is much better at math than me, and I had forgotten about Python's integer division so I was left to choose from between the last two entries: "Bite off all you like - Chewing is optional" (yes, I said most were terrible ;), and "import this". At the last minute, Tim resurrected a fun one that we had both noticed approvingly early on: "Let's we study about Python program".

While that last one had lots of appeal, I liked the irreverent, almost sneering tone of "import this". I saw the potential for a great, Michael Jackson-esque t-shirt.

As soon as we'd chosen "import this" I realized we just had to implement it. Python 2.2 was about to be released and I proposed that we turn off checkin notifications and sneak in a "this.py" module which when imported just printed the Zen of Python. Tim or Guido suggested further that we rot13 the contents of the module just for a little extra obfuscation, and we told no one outside our little group. According to my intergoogles spelunking, as soon as IPC 10 was concluded, we commemorated the event by committing this.py to what was to become Python 2.2.1, thus adding to the affront of new features in point releases. IIRC, it took a long time for someone to find our little easter egg.

That was all back in the day when the Python community had a sense of humor.

Experimental Virtual Machines

2010-06-07T15:03:00.006-04:00

I'm doing some work these days on trying to get Python 2.7 as the default Python in the next version of Ubuntu, Maverick Meerkat (10.10). This work will occasionally require me to break my machine by installing experimental packages. That's a good and useful thing because I want to test various potentially disruptive changes before I think about unleashing them on the world. This is where virtual machines really shine!

To be efficient, I need a really fast turnaround from known good state, to broken state, back to known good state. In the past, I've used VMware Fusion on my Mac to create a VM, then take a live snapshot of the disk before making my changes. It was really easy then to revert to the last known good snapshot, try something else and iterate.

But lately Fusion has sprouted a nasty habit of freezing the host OS, such that a hard reboot is necessary. This will inevitably cause havoc on the host, by losing settings, trashing mail, corrupting VMs, etc. VMware can't reproduce the problem but it happens every time to me, and it hurts, so I'm not doing that any more :).

Back to my Lucid host and libvirt/kvm and the sanctuary of FLOSS. It's really easy to create new VMs, and there are several ways of doing it, from virt-manager to vmbuilder to straight up kvm (thanks Colin for some recipes). The problem is that none of these are exactly fast to go from bare metal to working Maverick VM with all the known good extras I need (like openssh-server and bzr, plus my comfortable development environment).

I didn't find a really good fit for vmbuilder or the kvm commands, and I'm not smart enough to use the libvirt command line tools, but I think I've figured out a hack using virt-manager that will work well enough.

1. Create a disk for the baseline VM (named 'scars' in my case :) manually

% qemu-img create -f qcow2 scars.qcow2 20G

2. Create the baseline VM using virt-manager

* I use dhcp internally, so I give this thing a mac address, assign it 1GB

of RAM and 1 processor.

* For storage, I tell it to use the scars.qcow2 file I created above

* Boot from the maverick ISO of your choice, install everything you want,

and get your development environment in place

* Shut this machine down

3. Clone your baseline VM

* In the virt-manager Manager window, right click on your baseline VM and

select Clone

* You will not be given an opportunity to select a disk or a mac address,

so for now just go with the defaults.

* Do not start your clone

4. Create an 'overlay' disk that is a backed by your baseline disk.

% qemu-img create -f qcow2 -b scars.qcow2 scars.ovl

5. Edit your clone

* Delete the disk given to your clone by default

* Create a new virtio storage that points to scars.ovl

* Delete the nic given to your clone by default

* Create a new virtio network device with the mac address of your

baseline. You'll get a warning about a mac address collision, but this

can be ignored (see below).

6. Boot your clone

At this point you'll have a baseline which is your known good system, and a clone/overlay which you can break to your heart's content. When it's time to iterate back to a known good state, shut down your clone, delete the overlay disk, and create a new one from the baseline qcow2 disk. This is pretty fast, and your turn around time is not much more than the time it takes to shutdown one machine and boot another. It actually feels a lot faster by the wall clock than Fusion ever was to snapshot and restore.

One downside is that you cannot run both VMs at the same time. I think mostly this is because of the MAC address collision, but also because creating the overlay requires that both machines be powered off.

The other downside seems to be that if you want to update your known good baseline, say by installing more packages or apt-get update/upgrade, you will have to recreate your overlay disk for your next experiment. Changes to the underlying disk do not seem to propagate to the overlay automatically. Maybe that's intentional; I can't find much documentation on it. (Note too that the manpage for qemu-img does not describe the -b option.)

I guess the last downside is that I spent way too much time trying to figure all this out. The Googles were not a lot of help but did give me the qemu-img clue. But at least now you don't have to! :)

We have a winner!

2010-05-26T17:46:00.004-04:00

In the early part of 2010, we started a contest for a new GNU Mailman logo. Our old logo, donated by the Dragon de Monsyne had served us well for many years, but it felt like we needed a refresh. Besides, we wanted a nice scalable graphic that we could use in many more situations. So we solicited entries and then conducted a poll. Today I am very pleased to announce the winner!

By better than 2-to-1, this submission by Andrija Arsic was voted as best logo by the Mailman community. Congratulations Andrija!

While we have yet to re-brand the website and software to include the new logo, we'll start using it immediately. If you'd like to help with any redesign, please contact us at mailman-developers@python.org.

A little bit about Andrija: originally from Trstenik, Serbia and now studying IT technology in Belgrade, Andrija is a self-employed, part-time graphic designer, specialising in the fields of corporate identity (logo) design, web design, print design and branding with the majority of his time spent designing and implementing marketing promotions for businesses such as logos, websites, letterhead, business cards, packaging and more. I'm glad that he also contributes to free software, as I think his winning logo is spectacular.

My thanks and appreciation to all the artists who contributed logos to the contest. All the designs are very nice, and in their own way, capture the spirit of GNU Mailman.

From Python package to Ubuntu package in 3-ish easy steps

2010-05-24T21:26:00.009-04:00

My friend Tim is working on a very cool Bazaar-backed wiki project and he asked me to package it up for Ubuntu. I'm getting pretty good at packaging Python projects, but I always like the practice because each time it gets a little smoother. This one I managed to package in about 10 minutes so I thought I'd outline the very easy process.

First of all, you want to have a good setup.py, and if you like to cargo cult, you can start with this one. I highly recommend using Distribute instead of setuptools, and in fact the former is what Ubuntu gives you by default. I really like adding the distribute_setup.py which gives you nice features like being able to do python setup.py test and many other things. See lines 18 and 19 in the above referenced setup.py file.

The next thing you'll want is Andrew Straw's fine stdeb package, which you can get on Ubuntu with sudo apt-get install python-stdeb. This package is going to bootstrap your debian/ directory from your setup.py file. It's not perfectly suited to the task (yet, Andrew assures me :), but we can make it work!

These days, I host all of my packages in Bazaar on Launchpad, which is going to make some of the following steps really easy. If you use a different hosting site or a different version control system, you will have to build your Ubuntu package using more traditional means. That's okay, once you have your debian/ directory, it'll be fairly easy (but not as easy as described here ). If you do use Bazaar, you'll just want to make sure you have the bzr-builddeb. Just do sudo apt-get install bzr-builddeb on Ubuntu and you should get everything you need.

Okay, so now you have the requisite packages, and a setup.py, let's build us a deb and upload it to our personal package archive so everyone on Debian and Ubuntu can easily try it out.

First, let's create the debian directory. Here's the first little icky bit:

% python setup.py --command-packages=stdeb.command sdist_dsc

Notice that this leaves us with a deb_dist/ directory, not the debian/ directory we want. The latter is in there, just buried a bit. Let's dig it out:

% mv deb_dist/wikkid-0.1/debian .

% rm -rf deb_dist

% bzr add debian

% bzr commit -m'Debianize'

Note that "wikkid-0.1" will be replaced by the name of your package. In order to build the .deb package, you need an "orig.tar.gz" file. Packaging sort of assumes that you've got an original upstream tarball somewhere and you're just adding the necessary Debian goo to package the thing. In this case, we don't have an upstream tarball, although we could easily create one, and upload it to the Cheeseshop or Launchpad or wherever. However, that just slows us down so let's skip that for now! (Aside: if you do have an upstream tarball somewhere, you'll want to add a debian/watch which points to it; that'll eliminate the need to do the next step, by downloading the tarball instead).

Let's create the tarball right now and copy it to where the following step will expect it:

% python setup.py sdist

% mv dist/Wikkid-0.1.tar.gz ../wikkid_0.1.orig.tar.gz

Here's the second icky bit. Building a Debian source package imposes a very specific naming convention on the tarball. Wikkid's setup.py happens to build a tarball with an incompatible name, while the sdist command leaves it in a place where the next step can't find it. The rename just gets everything into the proper place. YMMV.

Now we can build the Debian source package. It's the source package that we'll upload to our Launchpad PPA. Launchpad will then automatically (if we've done everything right) build the binary package from the uploaded source package, from which Ubuntu and Debian users can easily install.

Oops! Before we do this, please edit your debian/changelog file and change unstable to lucid. You should also change the version number by adding a ~ppa1 to the end of it. Yeah, more ickiness.

Alright now we're ready to build our source package:

% bzr bd -S

Now let's upload it (assuming you've enabled a PPA):

% cd ..

% dput ppa:barry/python wikkid_0.1-1~ppa1_source.changes

That's it! If you've done everything successfully, you'll have the package in your PPA in 5 minutes or so. Then anybody who's added your PPA can just apt-get install wikkid (or whatever your package is called).

I do hope to work with the appropriate developers to make some of the ickiness go away. Please do contact me if you want to help!

Addendum (2010-06-10)

Let's say you publish your tarball on the Cheeseshop or Launchpad, and you don't want to have to build a different tarball locally in order to package it. Here's what I think works:

Create a debian/watch file that points to the download location you publish to. If your package is not yet available in Debian or Ubuntu, then use this command to build your source package:

bzr bd -S -- -sa

The bit at the end tells the Debian packaging primitives to include your tarball when your source package is uploaded. The debian/watch file is used to download your published tarball and automatically renamed to the required .orig.tar.gz name. When you dput your package, your tarball will be uploaded too, and everything should build properly.

Oh, and don't forget to look carefully at the lintian output. Try to make this as clean as possible. The Debian and Ubuntu packaging guides can help here.

Addendum 2 (2010-06-10)

Andrew Straw has added a debianize command to his stdeb package, which makes things much nicer. With this you can create the debian/ directory right next to your setup.py. AFAIK, this version of stdeb isn't released yet, so you need to install his git head in a virtualenv, and it has a few minor buglets, but it does seem like the best-of-breed solution. I'll post another article with a more detailed follow up later.

Gentoo No More

2010-05-23T17:56:00.009-04:00

Today I finally swapped my last Gentoo server for an Ubuntu 10.04 LTS server. Gentoo has served me well over these many years, but with my emerge updates growing to several pages (meaning, I was waaaay behind on updates with almost no hope of catching up) it was long past time to switch. I'd moved my internal server over to Ubuntu during the Karmic cycle, but that was a much easier switch. This one was tougher because I had several interdependent externally facing services: web, mail, sftp, and Mailman.

The real trick to making this go smoothly was to set up a virtual machine in which to install, configure and progressively deploy the new services. My primary desktop machine is a honkin' big i7-920 quad-core Dell with 12GB of RAM, so it's perfectly suited for running lots of VMs. In fact, I have several Ubuntu, Debian and even Windows VMs that I use during my normal development of Ubuntu and Python. However, once I had the new server ready to go, I wanted to be able to quickly swap it into the real hardware. So I purchased a 160GB IDE drive (since the h/w it was going into was too old to support SATA, but still perfectly good for a simple Linux server!) and a USB drive enclosure. I dropped the new disk into the enclosure, mounted it on the Ubuntu desktop and created a virtual machine using the USB drive as its virtio storage.

It was then a pretty simple matter of installing Ubuntu 10.04 on this USB drive-backed VM, giving the VM an IP address on my local network, and installing all the services I wanted. I could even register the VM with Landscape to easily keep it up-to-date as I took my sweet time doing the conversion. There were a few tricking things to keep in mind:

I use a port forwarding border router to forward packets from my static external IP address to the appropriate server on my internal network. As I prepared to move each service, I first shut the service off on the old server, twiddled the port forwarding to point to a bogus IP, then tested the new service internally before pointing the real port forward to the new service. This way for example, I had reasonably good confidence that my SMTP server was configured properly before loosing the fire hose of the intarwebs on it.
I host several domains on my server so of course my Apache uses NameVirtualHosts. The big downside here is that the physical IP address is used, so I had to edit all the configs over to the temporary IP address of the VM, then back again to the original IP of the server, once the switch was completed.
My old server used a fairly straightforward iptables configuration, but in Ubuntu, UFW seems to be the norm. Again, I use IP addresses in the configuration, so these had to be changed twice during the migration.
/etc/hosts and /etc/hostname both had to be tweaked after the move since while living in a VM, the host was called something different than when in its final destination. Landscape also had to be reconfigured (see landscape-config(8)).

You get the picture. All in all, just tedious if not very difficult. One oddness was that when the machine was a Gentoo box, the ethernet port was eth0 but after the conversion to Ubuntu, it became eth1. Once I figured that out, it was easy to fix networking. There were a few other little things like updates to my internal DNS, pointing my backup server to the new locations of the web and Mailman data on the new server (Ubuntu is more FHS-compliant than my ancient Gentoo layout), and I'm sure a few other little things I forgot to take notes on.

Still, the big lesson here was that by bouncing the services to a USB-drive backed VM, I was able to fairly easy drop the new disk into the old server for a quick and seamless migration to an entirely new operating system.

2010-01-25T12:18:00.004-05:00

I read this somewhere:

When we opened the door to this wonderful place, we were stunned! Literally. Uncle Fledge had to be led in by the hand and seated in front of the massive TV. He'd never seen anything so big. Grandma loved the "hot" tub and even with her massive girth, it came up to her chin so she could relax. The little family of hamsters or weasels kept wanting to join her, so that's what the broomstick is for (in case you're wondering).

Uncle Fledge is a <> <> skier but he loved the game room so much, he played for 36 hours straight until he got pokey finger. The Aunt Weebus dragged him out to the slopes. Good thing too because we were getting tired of White Lion on the jukebox. Uncle Fledge and Jimmie's favorite band, how did you know?

Martha was kind of obsessed with the telescope but billy brought his laptop and then it was family YouTube time. The view is spectacular though. Grandma loved watching the bears iceskate on the lake. Never seen that before she said. The twins wanted me to add that the bunk beds are fun, and the triplets loved having their own rooms. Mom and Dad retired to the jacuzzi for 8 hours, and little Moby had to keep refreshing their champagne. We loved the Bears Den and will definitely be back.

- The Kimmels, the Conans, the Boxxees, the Fledge Jrs & Srs, Phil, Sarah, Monk, Billy, Jimmy, Sandy Candy & Mandy, Uncle F. Aunt W, Grandma, Tina, Mallory, Frank, Stu, Mack T, Mack F, Tug, Willy and "Beans".

MBP

2009-12-09T18:47:00.005-05:00

So I got my new Mac Book Pro, named it "hemispheres" as mentioned in my previous post. I'm very happy with this machine, and here are my early impressions.

The SSD is pure win. It's quiet, cool, and fast. The biggest immediate downside of course is the cost, but as a refurb, this option was made much more affordable. It'll be interesting to see how it performs over time as there are known issues with the long-term use of the technology.

My two biggest concerns were the screen and the keyboard. I absolutely love my previous generation MBP keyboard, as it's about the only laptop I can comfortably use for 8 hours a day. The unibody MBPs have a completely different keyboard, with a different feel, but I hadn't used one for a long hacking session so I was unsure how it would feel after a few hours under my hands. I'm very relieved to say that while it has a different feel, it's still incredibly comfortable to use. Apple seems to know what they're doing.

I have had a few problems hitting the right keys, most notably not quite getting the shift-2 chord right to produce an @ sign. The reach is a little bit longer it seems so I often end up just typing a "2", somehow missing the shift combination. I have a few other common typing errors, but for the most part I'm getting used to it.

The other big uncertainty was the antiglare screen. I absolutely love the matte screen on my old MBP and hate hate HATE the glossy screens that seem to come on most laptops today. Honestly, if the MBP didn't offer a matte screen as an option, I probably wouldn't have bought it. Of course, now it's a $50 upcharge, but I think it's worth it. The antiglare screen has a different color temperature than the old MBP matte screen. It seems colder and I don't think there's any way to adjust it (well, maybe color calibration but my first and only attempt at that didn't do much). The screen also seems a bit more "washed out" than my old MBP, but it's also brighter because of the backlit-LED. I'm not fond of the dimming in the dark and brightening in the light (it seems backward to me), so I'm kind of riding the F1 and F2 keys, but after several days of use, I am getting used to it.

The machine does seem to have a very long battery life. For hacking in front of Sunday football, I'd estimate it's about a 5 hour run, and the battery indicator seems pretty accurate after its calibration. That's with the high performance graphics card, btw (there are two in this machine).

I'm getting used to the different arrangement of ports and media slots, and the sleep light seems much less obnoxious. I do wish the machine had come with an eSATA port, as I think FireWire at least for storage is becoming increasingly uncommon and/or expensive.

Snow Leopard (OS X 10.6) seems pretty cool, although honestly I don't notice much of a difference. There are still some kinks though, as my cursor freeze up occasionally. I'm sure Apple will fix this in subsequent patches. They did seem to fix a few Mail.app bugs I'd reported, but sadly not everything. And they broke GPGmail so currently I have no encryption in Mail.app (a release for Mail.app 4 is reportedly forthcoming).

I wanted this machine to run Ubuntu as well, so that I'd only have to take one laptop with me to conferences and such. I struggled with the best way to install Ubuntu: should I dual boot, use a virtual machine, or both? A colleague of mine had done the "both" route with VMware and bootcamp and I had enough disk space to consider this, but decided to see how the VMware-only approach worked first. The main problems are that Xorg configuration and MAC-address (and thus DHCP assigned IP addresses) would be problematic in a "both" environment.

Fortunately, Karmic Koala in a VMware 3 machine works absolutely great. It's very fast, looks and feels great, and doesn't tax the machine much that I can tell. The biggest problems are the lack of OpenGL/compiz for windowing, and a weird problem where Gnome-Do's summon chord of control-command-space has to be hit twice in order for it to work. I don't know if this is a VMware thing (since it works fine on my native Karmic desktop) or because I need to install vmware-tools. My last experience with vmware-tools was far from satisfying so I've mostly ignored it. I may try it to see if it clears up this problem though.

I should mention that I have an older version of Parallels, but ditched it because of some things that bug me about it, such as screen resolution during boot up, and more. Parallels 5 reportedly supports OpenGL/compiz for guests, so I grabbed a try-and-buy and went through the installation of Karmic in it, but was never able to enable desktop effects, so ditched it again.

In any event, I'm not going to try to set up a dual-boot until and unless I have to.

I did have one big gotcha as I was getting the new MBP set up. Apple has a very nice migration assistant that helps you move your stuff (including applications, settings, home directory, etc.) from your old machine to your new machine. The first time I tried it was massive fail. First, I did a wireless-to-wireless transfer and it never warned me that it was going to take 12+ hours to migrate, and once it started it was impossible to interrupt. Second, 12 hours later when I tried to log into the new machine, I was completely prevented from doing so. I actually could log in, but OS X was throwing up a cryptic error saying it couldn't open my FileVault. My home drive on my old MBP is encrypted and of course I wanted the same on my new machine. I figured since Migration Assistant happy proceeded it would Do The Right Thing. Yeah, not so much.

After being on the phone with Apple support for a few hours, I was finally told that this was not a supported migration option. Thank you Migration Assistant for the warning. Not. The solution was to re-install the OS on the new machine (since I could in no way log in now), turn FileVault off on the old machine, effect the transfer, and then re-enable FV on both machines. The problem with this is that you need enough disk space on the old machine to turn FV off, and this I did not have. It took quite some effort to free up the 31GB of space on my old machine (1/3 of the total hard drive size), but by temporarily moving stuff off to a NAS, I finally managed it.

This time, I did an ethernet-to-ethernet migration and it took way less than 12 hours. Of course it was transferring a lot less data, but still, it only took an hour or so maybe. I guess that's the advantage of gigabit ethernet over 802.11G :).

Hemispheres

2009-12-04T13:06:00.004-05:00

The relationship between the bass player and the drummer is one of the most profound and important relationships in rock music. I've been incredibly fortunate in my musical life to have played with a number of drummers with whom I've had a deep and usually immediate musical connection. Almost all are still close personal friends. For me, playing with a good drummer is probably one of the most important reasons to stick with a band long term (it's a necessary but not sufficient condition).

I've been with the U-Liners now for quite a few years and I've enjoyed every minute playing with our drummer Larry. He's in the Army band, so clearly he's talented, but it goes deeper than that. His feel is impeccable, his timing is great, and he hears everything. What I really love about him (and many of my other favorite drummers) is that he's solid but knows when to go for it. It's the occasional waltzes on the edge that make for an exciting, emotional musical moment. It can't be too often, but it has to be there. An element of risk is involved, and I love more than anything else, those improvisational moments within musical structure of the song, because that's where the magic happens.

This week has been extraordinary too because I've played with three of my favorite drummers of all time. Larry of course (and we have a gig this coming Saturday night), but also too great drummers and long time friends. Last night the U-Liners played a show that Larry couldn't make, so my friend Torro sat in. Torro and I go way back (he was best man at my wedding) and is an amazing musician. And at a jam party last Saturday night, my friend Keith came up from Florida and we got to play together for a while. Keith was actually the first drummer that I played with where I experienced that immediate mind meld, back when I was 16 or 17. We knew from the first song that we had Something, and now even 30 years later, it's still there.

Larry and I have been doing something very cool and fun for a while now that I don't think anybody (maybe not even our fellow band members) know is going on, but it cracks the hell out of us! At the end of songs, there's usually a little closing fill, kind of a ba-dump roll that ends thing. Larry and I have this little game going on where we hit those little closing rolls dead on. I honestly don't know how we do it, nor do I really want to know! We're not 100% together, but it's better than 9-out-of-10, and the rest are pretty darn close. It's not because that closing is the same every time; in fact, he sometimes challenges me by throwing in subtle differences, or delaying the hit, or adding a little extra flourish. The key to me seems to be to all mindfulness. I don't think. I just open my ears and my eyes and let my fingers do the right thing. If you see us laughing at the end of a song, it's because we've hit a particularly good one.

The wild thing about this is that even with the hour or so I played with Keith and Torro this week, I've managed the same connection with them. Keith and I played one song that was almost all improv, and it was one of those chord progressions where it always comes back around to the head, so you're never quite sure how you're going to get out of it. Then, out of nowhere, there it was, a clean ending where we Just Knew it was ending, Keith and I hitting the last notes perfectly in unison, perfectly unplanned, but as tight as if we'd played together every day for those 30 years. We looked up afterward and just had to laugh! Torro and I had several such moments at last nights gig.

It's moments like these that make me so thankful to be a bass player bless with good drummers, and really just great musicians in my life. It's what makes being the bass half of this whole thing such an ongoing joy.

----

It looks like my new Mac laptop will be delivered today, so it's time to name it. My naming theme has always been Rush references, with a preference going toward one word song and album titles. This laptop is going to dual-boot Mac OS X and Ubuntu, so it seems appropriate to call it "hemispheres" although I've yet to figure out which one is Apollo and which one is Dionysus. Maybe I should call the machine Cygnus instead...

Hemispheres was also the first Rush album I heard, and I've been a rabid fan ever since.

Laptops

2009-12-02T08:30:00.003-05:00

I really need a new laptop. While I love my 3.5 year old Mac Book Pro, it's by now showing its age and the abuse I've given it. It runs Fusion to give me an Ubuntu desktop, but because the disk is almost full I have to run that from a USB drive, so it's slow. I know I can upgrade the disk, but the machine is a Core Duo only, so that probably wouldn't help much. I also have an old IBM Thinkpad X40 which has an excellent form factor, but a small screen, and the motherboard power jack has an annoying habit of screaming at a high pitch when plugged in, so it lives in its dock, making it heavier and much less portable. And this thing is really slow.

What to do? Well, I've been looking around and with Cyber Monday (do you hate that marketing term as much as I do?) I knew there'd be some great deals around. One of the best, which if you're reading this the same day I post it, is on Lenovo computers (formerly IBM), including steep 35% discounts on Thinkpads. I know, I should get one of those!

I looked at the x200s and the x301, the former being closest to the X40 form factor, but with a WXGA+ 1400x900 screen (yay!), no trackpad (boo), and no built-in webcam (boo). The latter has the same WXGA+ screen (yay), a trackpad (yay), a built-in webcam (yay) and default solid state drive (yay). It's still under 3lbs (yay) and only .5lbs heavier than the x200s in its stock configuration. But it's about $800 more before you start customizing it. Still the discounts really bring both computers into the realm of possibility, especially for a work computer that I'll be using constantly.

I came this close to pulling the trigger, but ultimately decided not to. Why? Because the Thinkpads just don't feel right under my hands. I use a Microsoft Natural Keyboard on my desktop because I can type for hours with no pain. It's about the only Microsoft product that I can highly recommend . I think it's literally saved my career. There's no straight keyboard in the world that I can use for more than 5 minutes without feeling pain... except the keyboard on my Mac Book Pro. For some reason, that keyboard is also very comfortable and pain free.

What's not comfortable and pain free is the keyboard on my X40. So before I pulled the trigger on the Lenovo, I sat down and used the X40 for my most common tasks: reading email and coding. Within about 5 minutes I remembered why I've given this machine to Max (my son, who is about as anti-Windows as I am, and loves his little Ubuntu machine). The thing just hurts to use. I'm certainly not going to spend over $1000 on a machine that hurts, even if the discounts are insanely good.

I've decided instead to save my pennies and get a new Mac Book Pro some time in the next 3 months or so. The design of the 15" MBP has changed pretty significantly since I bought mine in summer of 2006. They use an aluminum unibody now, a default glossy screen, and most importantly a completely different keyboard. I honestly don't know if it will have the same comfortable feel as my current MBP. (TBH, I don't know that the Thinkpad x200s will have the same painful feel as my x40, but I think it's a good bet. Their designs haven't changed that much since then.) The little I've played on a new MBP is encouraging though.

And one thing's for sure, the Mac hardware is just better. The aluminum bodies are solid and sturdy, and I think few laptops could have put up with the abuse I've given mine. I know my old little Dell (now donated as an Ubuntu machine to a friend) failed me after about a year. Yes, you pay for it upfront, but they are reliable workhorses, so IMO well worth the money. Coming up with the money is a different matter.

Of course, Apple won't actually build the machine I want: a Mac Book Air with a 1400x900 matte screen. But the MBP, even with it's heavier weight, will have to do.

In any case, I know that the Thinkpad isn't right for me, so for now, I will choose not to decide.

Mailman and LMTP

2009-11-29T14:31:00.002-05:00

So, I spent quite a bit of time this weekend making sure that Mailman 3 works with LMTP delivery from Postfix. With some help from Patrick on the mailman-developers list, I've gotten this working pretty well. I also took the opportunity to play with virt-manager on Karmic. It was pretty darn easy to create a virtual machine, load it up with Ubuntu 9.10 server and test Mailman in a fairly pristine setting. The one gotcha that I ran into was having to run virt-manager under sudo, otherwise it could not create the VM files under /var/lib/libvert/images

I'm going to release Mailman 3.0.0a4 today. It really took me much longer to get this out than it should have. I removed the dependency on setuptools_bzr because it really sucked having to build bzr just to install Mailman. The whole reason for the setup dependency on setuptools_bzr was to allow me to be lazy in creating the MANIFEST.in file. I bit the bullet and created that file, but it took surprisingly many iterations to get it right.

On another front, after my bout with the flu and trip to Dallas, I'd been pretty remiss in updating my two Gentoo servers. They were way behind on patches, so I finally updated the public one, which was a royal pain in the butt. Still, it was doable. The internal server is running EVMS which is no longer supported by IBM or Gentoo. I think this will be the final straw that forces me to install Ubuntu on that machine. I'm dreading moving all the services and files off of it in preparation for that. It's going to be a time consuming job. Fortunately, I have a spare box that I can bounce the services to, and backups of all the files. I'm probably going to postpone that until the long holiday break.

House, Music, Hacking

2009-10-08T19:31:00.005-04:00

Well, it's been a long time since I posted anything here, and a lot has happened. I won't bore you with all the details, but suffice to say it's been an interesting ride. The good news is we were able to move back into our house in August and life is good. We're not completely done, but we're getting back on track and moving on with our lives.

Max started middle school so I am officially a morning person now, and I've even caught the Facebook bug. We got a cat.

I want to expand this blog to cover things technical and personal, and I'm going to spend a little time every week posting something here. I was going to say "something interesting" but it probably won't be, so you're just wasting time reading this. :)

The email-sig is very interesting lately. Python comes with a package to parse, generate and manipulate email messages. It's actually very functional in Python 2 but mostly because we cheat. In Python 2 we can be lazy about what's a string and what's a byte and email exploits this profusely. I know this because the email package is severely damaged in Python 3, where the distinction between strings (unicodes) and bytes is explicit. The email-sig is tasked with maintaining and developing the email package and we're struggling with many tricky issues. And y'all thought email was simple because 99% of it is spam.

Python 2.6.3 was released last Friday, but it was broken and no one should use it. It's my fault as the release manager for wanting a shortened candidate cycle, but I'm still not convinced that a long cycle would have avoided the regressions. 2.6.3 broke the logging module and setuptools, so I released Python 2.6.4rc1 on Tuesday. 2.6.4 will have a nice long candidate release and a very conservative commit policy. We'll just fix the regressions in 2.6.3 and hopefully have a solid 2.6.4 on October 18.

On a personal note, I've been taking vocal lessons from Dede Wyland for about 6 weeks. She taught Joe Uehlein (from the U-Liners) and she's fantastic. I never really understood harmony theory, but her explanations were fantastic. Tenor tends to sing a chord tone above the melody and baritone tends to sing a chord tone below the melody. So if the chord is G major and the melody is singing a B (the major 3rd), the tenor will sing a D (the 5th) and the baritone will sing a G (the root). I was pleasantly surprised that I didn't totally suck when we sang together! I'm really enjoying stepping up to the mic more often.

Roof

2009-03-06T19:09:00.002-05:00

Okay, we have roof and insulation estimates. There are basically three options. The most expensive involves using this foam insulation between the attic rafters. This would give us essentially a conditioned attic which would be usable in for living space in the future. Given the price, and what we're going to use the attic for this seems overkill.

The second option involves insulating between the joists in the second floor ceiling but there are problems with this approach because batt insulation to code won't fit in the old shorter joists. So this option will include building up the joists to accept the R38 insulation.

The final (and cheapest) option seems best to me. This involves using 12" rafters in the new roof so that it can accept the 10.5" R38 batt insulation. We're going to go this route and it'll cost us approximately $6500. I have to hand it to the builder for really working with us to make this as palatable as possible. It's definitely not something I expected to spend money on and it's definitely a painful hit, but in the long run I think it will work out by giving us much better insulation and usable attic space. I also think it will be easier for the builder.

Now we just need plans for the roof, and the county will probably have to approve it. (I wonder how much that's going to cost?!)

Insulate? Holy Moly!

2009-02-13T19:06:00.000-05:00

We got quotes on the roof insulation. Wow, it is not cheap and it makes the whole roof proposition an even harder hit. It's going to add $2000-$3000 onto the total cost of the roof. Tom will work up specific estimates and we'll go over these at our next production meeting.

Nothing is happening at the house.

Waiting

2009-02-11T19:04:00.002-05:00

Not much is happening, as we're waiting to decide about the roof. We need to get insulation numbers. Tom really thinks rebuilding the roof is the best thing. I tend to agree, but we need to make it fit the budget.

We told Tom about the placement of the kiln, which we want to put in the unconditioned space. This shouldn't be a problem, but we're still waiting to see the electrical bid.

A new roof?

2009-02-10T18:52:00.002-05:00

The builder is recommending that we rip off the old roof entirely and rebuild it from scratch. The cost came in at $4700 but that does not include insulation (or the 20% change order markup), so we'll need a complete price in order to decide if we should do it. This is a big hit to the budget, but it seems like it's really going to be the best option. We'll need architectural and structural plans for the new roof, which basically must look like the original roof in front (hip with dormer) and like the plans in the back (hip joint to match the front).

We looked at the shingle colors - I think Jane would love to go dark dark, but that's not nearly as energy efficient as a lighter color. I think we'll end up compromising on something like Forest Gray which is just a shade or two lighter. Wouldn't it be great if we could just layer solar panels over the entire roof? Maybe in Max's first house.

We met the stucco contractor today, who dropped off some samples and a color palette. Seemed like a nice guy and he offered to take us around to some of his jobs to see his work. I'm sure Jane will take him up on that. Sounds like he can basically match any paint color in the stucco; we also need to pick a finish. Jane's leaning toward the coarse grain for the majority of the house, with a fine finish for the trim. The really coarse finish just didn't do it for us.

The cabinets are killing us! We did a plan at HD for some fairly standard KM cabinets. I'm not very psyched about them, and it's really difficult to get cabinets that don't look like they belong in an apartment. The thing that sucks the most about the KMs is that they really don't give you the options you want. For example, we'd like to get the Shaker Rail doors in a dark color, but you can't do that. Not to mention that we're still at about $9000 for the quote, and even with the HD discount, we're still going to be a thousand or so over our allowance. We'll see about ditching the glass fronts, but that brings the discount down along with the price.

Really, we want to do something different but that seems impossible on our budget. Still looking.

Little things

2009-02-04T18:43:00.003-05:00

Lots of little things today, but not too much progress. Tom came up with a great, simple idea for the stair posts, involving bearing the load on the existing steel beam in the basement, then pull that load across to the outer wall. This means we won't need to break up the concrete and pour another footing, with the sleeve around the waste pipe.

We don't want to overbuild the existing roof, nor do we really want huge trusses because both options will severely limit the amount of usable space in the attic. We need to get options and costs on our alternatives.

Jane is still thinking about the RTA cabinets. Everyone's saying "Don't do it!" because of the amount of work involved and the low margin of error. Still, to get close to the cabinet allowances, we may have to do it this way. No decision on the cabinets are made yet.

The bay window will be insulated after the close-in inspection, so there's plenty of time for that.

Tom will get Jane square footage for tile, so that she can start looking into that.

We're still waiting on the electrical proposals.

We'll need to pick the shingle colors this week. This is especially important because we need to match it to the stucco color, which we also have to start thinking about.

Nope

2009-01-31T23:26:00.000-05:00

We got some cost estimates on the two alternatives we talked about previously. As suspected, both the additional insulation in the exterior walls, and the beadboard in the porch are too expensive so we'll pass on those.

The porch, with a wood flooring, is framed up to the headers. We're still waiting for structural on the bathroom and roof, and we're still waiting for the partial lien wavers.

What's goin' on

2009-01-30T23:06:00.003-05:00

The porch deck is being built now and the basement plumbing is done.

We've talked more about the structural options for keeping the existing bathroom. One option is to bring a footer down through the corner of the new basement bathroom. The unfortunate thing is that this drops right onto the waste pipe, with a sleeve to protect against any future settling. This would entail tearing out more of the original cast iron.

We also learned that the basement slab falls 2" from front-to-back but nothing significant side-to-side, so this will probably be easy to level out in the new slab.

There's a conflict in the roof drawings. We noticed that the architectural drawings say to tear the back roof off, while structural says to build over the existing roof. We'll have to think about options for resolving this and work out some numbers.

One time sensitive decision coming up is the rough-in kits for the showers, and we'll need to research the water heater a bit. We're also waiting on structural for the stair support options. In the meantime, the porch will be framed out.

I had a thought about some extra porch steps leading down from the small corner jog to the Louis side. Steve said he'd think about this, but I get the sense he's not crazy about the idea. I don't think Jane is either, so I'll probably be convinced to drop it. :) Jane is also thinking about doing beadboard between the porch roof beams and the ceiling, but again, it probably won't be cost effective to do.

Snow!

2009-01-28T23:03:00.002-05:00

Today's production meeting needs to be rescheduled due to the snow, but we did get the window list in an email and will review that.

Return from vacation

2009-01-27T22:58:00.002-05:00

We were on vacation for a few days, doing some much needed detox on the ski slopes.

On our return, we saw that the whole addition is framed out, but not yet under roof. The brick exterior on the back of the old house has been demoed but not yet cleared. There's some kind of weird contraption on the roof. I wonder what that's all about?!

There's not much other activity at the house, probably due to the snow.

More considerations

2009-01-22T22:49:00.001-05:00

We thought about a couple of extras today:

Since the front wall where the bay window is was torn out, we noticed that the existing house is basically uninsulated. One of the things we're considering is tearing out the few outside walls we aren't touching, insulating, and re-drywalling them. It's tricky though because we've got some weird 1930's pre-drywall covering on the studs. There's only a few walls we're talking about, but it's still probably more work than we want to pay for. We've asked for an estimate and will get it from the builder soon.

There's another issue with the existing bathroom that we're not touching. There's a question about how the support for the new stairs is going to tie into the perimeter wall. This beam runs under the existing bathroom, but the plumbing is essentially in the way. We've asked for an estimate to tear the bathroom out and rebuild it, but it will probably be cost prohibitive. We're looking into other options, and are consulting the structural engineer to see if we can just drop a post down into the basement.

I also walked the site with one of the electricians, looking at what we want (especially with some kind of X10 setup, the network cabling and the additional outlets in the office and computer closet. We'll review his bid when it comes in and probably get a second bid as well.

Good news! The plumber's come in and verified that we will not need an ejector pump. I'm really glad we won't have to worry about that gross little addition.

The framing out of the ground floor is continuing apace.

Thinking About Windows

2009-01-21T22:25:00.005-05:00

We had our weekly production meeting this morning, Steve, Tom, Jane and I. Tom brought the tentative window order by and we went through them. Would you believe how hard it was to remember what was in the original house, at least until I pulled up the pictures from the web! To save money, we're matching the new windows with the vinyl replacements that already exist. Those are all 3x2 on top and no mullions on the bottom. We can match all of those except for the casement windows we'll be using in the new bedrooms to conform to egress code. The small 25" wide windows will be awnings, and everything else will be double hung windows. The dormers will probably be fixed and we'll be getting half screens all around. The only tempered windows will be the small ones in the basement, which are just above grade.

We noticed that the bay window is fully framed in now, but the guys forgot to insulate the well underneath it, so Tom is going to get them to pull up that flooring and insulate there. Supposedly the plumber was coming by today, but I never did hear the verdict on that.

We talked at length about the stucco seams that are needed at all material joins, and runs over 144 square feet. Since the Louis side marries the old brick construction with the frame construction of the new part, there will be a vertical seam right there. We really need to match that well so that its as unnoticeable as possible, as it's going to be 1/2" wide the height of the house. There will be some horizontal seams as well, and we may place them low on the house, and draw them all the way around, with a color offset for aesthetics.

I read something on the 'net about stucco and the need for kickout flashing, which is installed at the transition between roof and vertical wall. It's purpose is to "kick out" any water away from the wall, so it can't seem behind the stucco and cause problems. I'll have to remember to ask about that when we get to that phase of the project.

I took a few pictures of today's progress. You can see that they've put the deck on for the new part of the house.

BHO!

2009-01-20T06:38:00.003-05:00

I just have to take a moment and say what an amazing day today was. BHO is now our 44th president and there were millions of people on the mall to witness it. Our neighbors biked down as we're only about 6 miles or so from the center of DC. Me, it was too darn cold and I had to work, but I did take a long lunch break to watch. I have nothing more profound to say than what everyone else has said, but it sure was cool.

And what's that with Cheney? Could he be any more Mr. Potter than that? George Bailey -> Mr. Smith -> Barack Obama anyone...?

Anyway, back to (as my wife would say) my boring house blog. Although we talked to Tom about the windows, there's no feedback on that yet, though we have a production meeting tomorrow morning. I went over to the house and it looked like the bay was framing was complete, including the roof. The one or two poor guys working out on that frigid day were putting a wood cap on the poured concrete walls of the conditioned area. Now the bolts sticking up make perfect sense, as the wood planks are literally screwed onto them nice and solid.

I didn't get into the house today, but will try to do so tomorrow.

Waterproofing

2009-01-19T06:28:00.002-05:00

Looks like today was a fun day with gooey black tar. This stuff is used to waterproof the foundation poured concrete walls that form the conditioned part of the house. The walls forming the unconditioned parts apparently are not waterproofed, such as the area under the entranceway, the stairs leading down from what will be the back porch, and the unconditioned area under the porch. I bet it wasn't fun to be slather that stuff all over the place in the cold we've been experiencing lately. I also hope that the stuff still works when it's slather on in that cold!

The bay window was framed out today, or mostly so. That's gonna be a really nice touch to the exterior view.

No word yet on the ejector pump. Tom dug down about 12" and did not see the top of the sewage line, but the plumber still needs to come in and take a look to give us the final word. We discussed re-arranging the bathroom in the basement to make it fit if necessary. We'll see...

The framing of the new living space was delayed because Tom was waiting for word from the stucco guy, just to make sure tolerances were set for marrying the old and new part of the house. The old part will of course be stuccoing over brick, while the new will be wood. The wood needs to be 1/2" back from the brick so that it can accept the mesh that the stucco will be applied to.

The one bummer is that there will have to be a seam between the old and new parts of the house. This makes sense because the original house has been settling for 70 years. We're concerned that the caulk seam will be too noticeable.

New picture site

2009-01-18T11:13:00.002-05:00

Everybody needs a day off. It's Sunday and there's no activity, but we did get a chance to show Jane's sister, brother-in-law, and their kids around the site.

Because of limitations in Flickr and because there's some nicer integration with blogger, I've moved the canonical location of the construction pictures to Picasa.

A Down Day

2009-01-17T11:18:00.000-05:00

Not too much happening today. The guys were going to start framing, and though there's a big pile of wood out there, there's no sign of that happening. They did start framing out the floor for the new bay window.

Steve and I did get a chance to get inside and look around. We were actually looking for a couple of stands for the dumpster drums I'm loaning his son Graham (a good friend of my son Max and a budding Neil Peart). Well, we couldn't find the stands -- they're either in the pods or gone gone gone -- but it was good to walk around with Steve. The basement step-up was jackhammered, but we haven't yet heard the gravitational verdict, and it looks like the temporary walls were built in the basement. I suspect the basement stairs will be ripped out before we know it.

We have got to wrap the sofa bench that we left in the house. It's huge and difficult to move, but not only was it heavily dusted, we noticed that there was a McDonalds drink cup left on it by one of the workers. I'll probably head to the hardware store and do that before we get too much farther.

It was also really cool to walk around the new foundation, see the poured walls from the inside and get a sense of what the new space will be like. We had a question about the window cut outs on the Louis side of the basement since there are supposed to be three windows there, but there are only two cut outs. The third is in the interface between the old and new house, so we think they'll just carve out what they need once the back wall comes down. We'll verify that with the builder.

Foundations

2009-01-16T10:51:00.000-05:00

I had a chance to see the poured foundation walls today. Originally, we were thinking to use block construction, but there's no arguing with the speed at which the poured concrete walls went up. Our builder has a lot of experience with this construction method, and our architect is cool with it so it's the way we went. Jane's not psyched about the formed brick impressions (she detests fake attempts when something tries to look like what it's not), but all that is going to get stuccoed anyway so you'll never see it.

(Aside: Jane gives me grief because this is all boring and not funny. I keep telling her that things are moving so fast, and there's so much information to get in that there isn't time to be funny. Don't worry! I'm sure there will be plenty to laugh about later.)

One big question that we have is whether we're going to need an ejector pump in the basement or not. The basement bathroom is above a step-up at the bottom of the basement stairs. We don't really know what's under that step-up. Ideally of course, the sewage line is below the grade of the new basement bathroom and we'll just let gravity do what it does best. If not, we'll need an ejector pump to get the, er, goo up from the lower bathroom into the main sewage line. If necessary, we'll hide the ejector pump under the stairs just as we're doing with the sump pump. We think there's enough room there. Tomorrow hopefully, we'll get that step-up jackhammered and take a look at what's there.

The other issue is the gas meter, which is currently inside what was the utility room, but what will be the family room if we finish the basement. Of course, we'd rather not have the gas meter in that location, but it may not be feasible to move it. There's no room outside that wall because of the new bay window, and it will probably be cost prohibitive to move the meter to the far side of the house where the new utility room is going. The plumber indicates that even with the additional BTUs of Jane's New UberOven we apparently won't need a new meter, so the gas company would be reluctant to pay for moving it. It won't be that big of a deal if we keep it where it is; we'll just hide it behind a closet or some such.

We're still thinking about finishing the basement. It's an additional cost (not part of the base contract), but it sure would be nice. We have some time to decide. We also have a bit of time to decide about the new back porch and new front walkway. Jane really wants to flagstone it, but that's also an extra cost. The difference is in the finish and depth of the concrete we're pouring in those locations. Obviously if we don't flagstone it, we'll have a nice smoother finish, but if we do cover it in flagstones, we'd want a more rough and thinner slab there. Again, we have time to decide.

As a cost saving measure we decided to keep the existing vinyl replacement windows in the old house and match the new windows as best we can. The problem is that new windows have a different reveal (the border area) than replacement windows. This won't matter on the front of the house because the only new windows are the bay and that will look different anyway. On the back, it's all new windows and on the Scutt-side it won't matter too much because it's not really visible from the street. On the Louis-side though there will be a mix of existing and new windows and Jane is concerned about the different looks. We'll have a better feel for that when we see some window options.

If the lumber arrives, we're told they may begin framing tomorrow.

Um, how the hell are the pod guys going to get their pods back when they're done?

Compromise, and progress

2009-01-15T21:47:00.000-05:00

(Happy Birthday, Mom!)

Today we had a production meeting at 7am with the builder, his project manager, and our architect. We talked about a bunch of things, including the need to raise the windows behind the kitchen banquette (for the seat backs), the fact that the original slab isn't level (but only slightly, and we don't yet know what the impact on the new slab will be), and what we'll use to frame the floor of the porch. Interestingly enough, one sheet of the plans calls for Trex while another leaves it unspecified, and our contract states Trex as an alternative. Neither our architect nor we really care too much as long as it is structurally sound and has water and ice guards. Later, our builder related that they preferred to wood frame it.

There's an unconditioned space below the porch, but my eventual hope is that in a few years we'll close that off, bring in hvac and electrical and possibly be able to finish it. It's a nice little space, but we really can't afford to finish it now, so it'll mostly be used to store lawnmowers, bikes and the lot. Someday it might make a nice little area for Jane's pottery studio perhaps.

Another interesting thing about that space: I wanted to put a concrete ramp in next to the stairs, so we've actually designed a 6' wide entrance. Unfortunately, even though that ramp was approved by the county, it turns out that it's not up to code after all, so we're going to put stairs in across the entire 6 feet and then probably put in a wooden ramp or some such for the few times I need to haul gear up and down. I'm disappointed, but we can work around it.

As mentioned yesterday, we did discuss the drywells for stormwater runoff, but since the county is pretty strict about this, we're going to leave it as designed.

I'm glad that we'll be waterproofing most of the outer foundation. Almost the entire perimeter has been dug around: the entire back of course, the side because of the new entrance footers, and half the front for where the new bay window is going. All that will be waterproofed. The neighbor side and half the front will be left alone, but they're already dry and we'll make sure the new grading will keep it that way.

There was also a question about whether code would require a sprinkler system, but we've learned that our plans will not require it.

One of the things I really want to think about is wiring the house for smart home. I mean, what would be cooler than being able to log into your house and turn off the lights when you're on vacation? I'm not sure we'll be able to afford to put in all the fixtures, but I really want to at least prep the house so that we can do this if we want. Of course, I'll also want to wire the house up for cat-6 so we can hardwire a few rooms to reach areas where wireless sucks. I'm starting now to research all this, but if you have suggestions, please leave a comment with some suggestions!

Oh, and the photo stream is up!

A big ol' pile of dirt

2009-01-14T19:53:00.000-05:00

It's amazing how quickly things have happened. The project manager (Tom) is a great guy and he's explained that we'll see a lot of progress early on, but that once the addition is framed and under roof, things will appear to slow. Plumbing, electrical and hvac all take a long time to complete. That's okay, right now it's very exciting!

Today the excavation was largely complete. There is now a BIG hole in the backyard, stretching almost all the way back to the tree line. This is sure going to eat up most of our backyard. There's also a ditch along the side of the house (which, because we're on a corner lot faces the street we are not addressed on), and a hole near the front door where the bay window will go. Footers were poured today for all of that.

We hit our first snag, namely in where the sump pump is going to go. The original plans were going to put this under the unconditioned space under the porch, but that's not allowed or recommended because it can freeze. Since that space will (for now) be open, you just don't want the sump pump to be exposed to the elements. We've decided to move the sump pump to under the basement stairs, which are barely usable as it is because it's such a small space. We'll have access to the pump from a utility door and it'll still drain out of the side of the house. The unconditioned area will still have a floor drain to handle any rain and such.

We briefly toyed with the idea of enclosing the unconditioned area, running hvac and electrical to it, so that we could use it as an additional room, but because of the way the stairs and ramp lead out of the basement, it isn't feasible or cost effective. We're talking more architectural and engineering fees, and most significantly, delays in schedule getting the changes approved by the county. We'll leave this area unconditioned for now, though our architect has assured us that should we decide to enclose it a few years from now, it will be doable (with some accommodation for the lack of an exterior landing, which we just can't fit given the yard space).

Another snag we hit was with the drainage plan, which was added at the last minute in order to gain county approval. It turns out that this will leave two clean out access ports in the front yard, each about 8" in diameter. This is unfortunate because they'll basically be front yard artifacts that will be somewhat unsightly. Tom and Larry (the builder) have told us that once the yard is given its final grade, the drainage pipes will be shorn so that they'll be at or below grass level. Hopefully they'll be easy to hide behind some gardening. We're exploring the possibility of using a single drain, but we're doubtful that the county will either give its approval, or do so in a timely manner. We really do not want to incur delays here.

Tomorrow, we've got our first production meeting with Tom and Larry, along with Steve (our architect). We've all met before, but I think it'll be good to talk about some of these outstanding issues, establish regular meetings, get a schedule, etc. It's gonna be cold tomorrow, so we're doing it at the rental house. Yay for 7am meetings!

Where'd my house go?

2009-01-13T22:39:00.000-05:00

Well, we woke up this morning to find most of our house gone. Actually, we'd sort of expected it since the county approved our plans on Friday. We've been in our house in suburban Washington DC for more than 6 years and always planned on doing a renovation project. It's taken this long for us to get off our butts and finish the plans, line up a builder, arrange financing, and so on. Given the sorry state of the housing market, this is either the smartest thing we've done, or the dumbest.

Anyway, we were expecting to break ground tomorrow, but Jane had to go up to the school this morning and saw the project manager on her way. He asked her if she'd seen the house -- which she hadn't yet. So we went over there and by noon, they had ripped off the crappy addition, taken off the awning and the concrete slab underneath, and nearly completely stripped the interior. Kitchen? Gone. Basement? Gone. Upstairs walls? Gone. Unbelievable.

By the time school had let out, the concrete to the front door was gone as was the huge azalea bush that Max and his friends had used as a secret fort. The bobcat and backhoe were hard at work digging out the big hole which will soon be the new part of our basement.

It's only the first day, but it was a good, exciting, freakout kind of day. I can say one thing though: I am so glad we decided to move out while the construction was under way! We took lots of pictures, which I'll be posting to Flickr as soon as possible.

Pain

2003-04-21T22:44:00.000-04:00

This is a repost from my earlier blog. I totally disavow everything my younger self said. He doesn't know what he's talking about.

There's a Star Trek (original series) episode that I dimly remember. I think it's the one with the rotating stoner light on the ceiling in the penal colony, but I'm not sure. I don't know what's sadder: that I used to be able to name each episode before the first commercial break, or that my memory is so far gone, that I no longer can.

Anyway, I think while Kirk's in the chair transfixed by "the colors, dude!" a man in the next room utters the single word: Pain. And Kirk feels the pain.

I'm thinking of this because I'm doing a CVS merge from the trunk of the Mailman tree into the 2.1 maintenance branch. That perverse little guy who invented CVS has just uttered "Pain" as I stare transfixed at all the pretty colors in my XEmacs buffer. And it hurts like someone knifed open your balls (which has happened to me, but that's probably too much information, huh?).

Why is this so painful? Well, part of the reason is that there are CVS id strings in the files that really shouldn't be there. They just cause conflicts for no good reason. Yeah, I know about -kv, but I know it about the same way I know the Star Trek episode, so it doesn't help me much.

The other reason is worse: there are tons of files that aren't in English and that I didn't write. Because Mailman is internationalized, there are a raft of files containing text in everything from Italian to Japanese, and now (tickling my ethnic pride nose hairs) Polish too. Now, for my own sanity, I'm trying to move to a place where the translation teams make their own updates in both the trunk and the maintenance branch, where there are essentially two copies of each file. Many teams are handling this quite well.

Me, like an idiot, did a blanket merge from the HEAD which caused all kinds of conflicts in files I can't read. So resolving those conflicts -- which must be done manually -- is basically like playing "pet the kitty" with the one evil cat of ours that is laying on my shoulders. It's all nice and fine while she's quiet, but at some point she'll expose some tendons and then the fun (and blood) will start. At some point in the conflict resolution, I'll delete the wrong block and screw up the hard work of other dedicated volunteers.

That sucks the most, because I know everyone's overworked, and my translators do a great job solely on a volunteer basis. I just hate to screw them over as thanks for a hard job well done, y'know?

I have some ideas on how to do this better, and plan on investigating The Translation Project to make our lives easier. I actually think Zope could come to the rescue here, but it now looks like I won't be working on that any time soon. So, like Kirk, I mostly just sit in agony in this chair, with one tiny part of my brain quite happy to watch the lights and enjoy the writhing of the body that surrounds me, hoping that eventually someone will turn the dial down and stop the pain.

Young Again

2003-04-04T23:11:00.000-05:00

This is a repost from my earlier blog. I totally disavow everything my younger self said. He doesn't know what he's talking about.

So I'm blogging. All the kids are doing it and it seems like a great way to rant and vent in a write-only forum. I feel like I did the first time I recompiled my Linux kernel. Once again I can put off my mid-life crisis without having to waste money on a fast car or have a divorce-inducing affair with the babysitter. Blogging is much less likely to have an impact on anybody else in the world, and that's probably a good thing.

Thanks to my colleague Jeremy Hylton for finding a nice Elisp blogging tool which makes it really easy to write these things. Expect a lot of content early on, followed by long periods of inactivity. But no one's reading this anyway, so who cares?

The main thing I've been doing lately is immersing myself in graphical IMAP based mail reading clients. I've been a VM user for years, preferring of course the One True of the One True Editors. But lately I've been disappointed with VM, primarily because of its lack of true remote IMAP support. I use at least 3 different desktop machines (Linux and MacOSX -- I'm Windows free!) regularly, as well as my laptop in various locations, and tunneling XEmacs over SSH is just not cutting it. VM has some support for IMAP but only by sucking all your mail over from the server to the local client, and that just defeats the whole purpose. I might as well burn a DVD with my gig of 20 year old email so I have it with me at all times.

I started out by using Apple's Mail application for OSX. I'm a huge OSX fan, so this felt like a good place to start. On the plus side, Mail.app is nicely integrated with the OSX desktop, has very good drag-and-drop, and has a quite nice integrated spam detection system. I've been aware of spam classifiers through my off-and-on dance with the Spambayes project.

However, Mail.app suffers from Bruised Skull Syndrome, caused by its many missing features and defects. Chief among these are the lack of message threading and folder subscriptions, virtual foldering, and its really horrible performance at times. It's unfortunately a lot like Apple's web browser Safari; really cool software that needs more baking. I'll keep an eye on the oven and hope the cookies are moist when they come out. Mmmhh, moist.

Once again, Jeremy turned me on to a cool tool, Ximian's PIM tool called Evolution. Get it? Ximian, simian, monkey icon, evolution? Evo, as it's quaintly called by us insiders, is really excellent, even for someone like me who won't buy into the whole Gnome desktop conspiracy. For a graphical mail app, it's about my ideal. Except for one huge, fatal flaw. Its imap implementation is horrendous. When it works, it's great, but it has a nasty and prolific tendency to drop its connection to the server all the time. It helpfully reconnects automatically, after shoving annoying click-away windows which let you know it hiccuped. I almost wouldn't care if it didn't display that annoying window. But really, its imap implementation should be better. As nice as Evo is, this one bug is getting to be too much so I may start looking around at other mates.

The Ximian folks claim the problem is that I tunnel cleartext imap over ssh, but I don't believe it because even running Evo on my local network in the clear, I get constant drops. And I've been tunneling XEmacs over ssh for years -- if XEmacs isn't a client sensitive to network glitches, I don't know what is (although they've gotten much better lately). Nope, the underlying connection is solid.

Mozilla's mail client, as well as its co-joined sister Netscape are just too pitiful to be usable. Enabling imap mail should be dead simple, with just a few configuration options necessary to get you started. But no, both Moz and NS were just too hard to configure properly and I couldn't get them to talk with my imap server. To give them a fair shot, I spent almost an hour trying to get the basic connections to work, but my impatient bastard side won out and I gave up.

Ditto for KDE's KMail application. Admittedly though, they have newer versions that what comes with my Redhat 7.3 version of Linux, but I haven't spent the time to install KDE 3.1 RPMs. I may come back to this though because Kmail had one extremely neat feature: I could use XEmacs as my external editor for composing email. This is great because of the huge number of Emacs hacks I've got going. I've been a heavy duty (as opposed to heavy doodie) Emacser for almost 20 years (yes, I'm a fuggin' old dinosaur) and a XEmacser/Lucid Emacser for maybe 12. So I like that environment a lot.

My netbud Jason Mastaler reminded me that Gnus has an imap backend, but treating my mail like news was just a little too jarring for me. Jason also pointed me to Wanderlust which I plan on taking a close look at. I wouldn't call either of these graphical, but there's an appeal of keeping everything in XEmacs.

I'm sure I've missed some, but if you know of any other decent clients for Linux or MacOSX, feel free to let me know. But you're not reading this anyway, so I guess you won't.

Okay, I've puked enough on this subject, so I'll wipe the chunks of lisp off my lips and see if I can clean myself up. Of course, with a blog you can turn any crap into art...