Living Code

There is a cool new language on the block, coming from The Google courtesy of (among others) Rob Pike and Ken Thompson. The language is called Go (and the debugger is called Ogle, so put ‘em together for a secret message) and it attempts to be both a low-level, type-safe, compiled language, like C, while being garbage-collected, concise, with high-level structures and idioms like Python and Javascript. There are a lot of things to like in it, and I am exploring what it can and cannot do. My first question was whether you could use it for functional programming, i.e., are functions first-class objects that can be passed to other functions, returned from functions, stored in variables, etc. This wasn’t in the FAQ or the Language Design FAQ, but there were some hints of it in the description of channels, so I went ahead and wrote my own tests to see if it would work.


package main

func main(){
takesAFunction(foo);
var f Stringy = returnsAFunction();
f();
var baz Stringy = func()string{
return "anonymous stringy\n"
};
fmt.Printf(baz());
}


This compiled and ran, giving the right output, so it looks to me like functional programming is indeed possible in Go, and on the Effective Go page we learn, “In Go, function literals are closures: the implementation makes sure the variables referred to by the function survive as long as they are active.” So that’s kind of cool.

Go also has rich, high-level concurrency support similar to Erlang, several features that prevent random memory from being read or written, and both strings and maps (dictionaries, hash tables) are built-in datatypes. It has a small, but powerful standard library.

One puzzle down, several to go. I still want to explore how I can talk to Cocoa from Go (for GUIs), how to wrap libraries such as libxml2 and cairo for Go, and basically how to map more Python/Javascript library goodness into Go’s world.

I have smaller questions too: can I add methods to existing types? How do I load the contents of a URL? It’s looking good so far! Tune in next time for more exploration.

A friend asked me for an example of AOP in Python. I started to write up my response, then realized it might be worth sharing more widely.

Briefly, AOP is about separating out Aspects which are are interspersed throughout your code (in AOP lingo, cross-cutting). The canonical example is logging, but there are others.

For Python, as long as you’re content with before/after aspects, the situation is good. In Python 2.5 and up there are two main tactics: decorators and context managers. Decorators use the @ syntax and context managers are used with the “with” keyword.

Decorators

Decorators are only usable on functions and methods in 2.4 and 2.5, but in 2.6, 3.0 and beyond they can be used on classes as well. Essentially they are callables (functions, methods, objects) that accept a function object and return a function object. They are called when the function is defined, so they get a chance to have their way with it: annotate it, replace it, or wrap it. The common case is to wrap it.

Quick example:

import logging
def before(fn):
    def wrapped(*args, **kws):
        logging.warn('about to call function %s' % fn.func_name)
        return fn(*args, **kws)
    return wrapped

def after(fn):
    def wrapped(*args, **kws):
        retVal = fn(*args, **kws)
        logging.warn('just returned from function %s' % fn.func_name)
        return retVal
    return wrapped

OK, those are three basic wrappers, you can use them like so:

@before
def foo():
    logging.warn('inside foo')

@after
def bar():
    logging.warn('inside bar')

@before
@after
def baz():
    logging.warn('inside baz')

foo()
bar()
baz()

This will result in the following output:

WARNING:root:about to call function foo
WARNING:root:inside foo
WARNING:root:inside bar
WARNING:root:just returned from function bar
WARNING:root:about to call function wrapped
WARNING:root:inside baz
WARNING:root:just returned from function baz

You will note that when we use two decorators on baz, the name of the function called by before is “wrapped.” This is because what before is called on is the result of after. The functools.update_wrapper function is useful in this case to make a wrapped function look more like the original function.

For more, please see PEP 318 Decorators for Functions and Methods: Examples and PEP 3129 Class Decorators. For convenience when creating new decorators, see the standard library functions functools.update_wrapper and functools.wraps.

Context Managers

Context Managers are used with the with statement, and are handy for resource aquisition and release. In Python 2.5 you have to “from __future__ import with_statement” to use them, but they are built-in in Python versions later than that. Also, objects such as files and locks are context managers now, so you can use patterns like

with open('example.txt') as example:
    for line in example:
        do_something(line)

This will automatically close the file when leaving the with block. And for locks the pattern is similar:

with myThreadingLock:
    do_something_threadsafely()

It is important to note that the lock will be release properly, or the file closed, even if an exception is thrown inside the with block.

If you want to create your own context managers, you can add two methods to your objects: __enter__(self) and __exit__(self, exception_type, exception_value, traceback). The return value from __enter__ will be passed to the optional as variable (seen in the file example). The __exit__ method will be called with exception info if there was an exception. If no exception is raised in the with block, then all three arguements will be None. If __exit__ returns True then any exception will be “swallowed”, otherwise the exception will be re-raised after any cleanup.

For more info, see PEP 343 The “with” Statement, especially the examples section, and also the helper functions in the standard library contextlib.

AOP and 80/20

Full-on aspect-oriented programming is beyond the scope of this post and involves join-points, code weaving, and other such arcanery. There are multiple Python libraries which target aspect-oriented coding styles, but for my money, the simplicity of the methods in the standard library, coupled with my impression that they cover at least 80% of the uses of AOP, make me favour these built-in techniques over any of the special purpose tools.

Goal: Iterate through all my (OS X Safari) browser windows and make a list of titles and urls which is then placed in the clipboard ready to be pasted into an email or blog post.

This is an update to Tab Dumping in Safari. That still works well as the basis for extending any Cocoa-based application at runtime, but it relies on SIMBL, which while it is a great bit of code, essentially is abusing the InputManager interface. Some developers and users shun such hacks, and at least one Apple application checks for them at startup and warns you from using them.

I have been running the WebKit nightlies, which are like Safari, but with newer code and features (most importantly to me right now, a Firebug-like developer toolkit). WebKit warns at startup that if you’re running extensions (such as SIMBL plugins) it may make the application less stable. I was running both Saft and my own tab dumping plugin, and WebKit was crashing a lot. So I removed those and the crashes went away. I miss a handful of the Saft extensions (but not having to update it for every Safari point release), and I found I really miss my little tab-dumping tool.

I toyed with the idea of rewriting it as a service, which would then be available from the services menu, but couldn’t figure out how to access the application’s windows and tabs from the service. So I tried looking at Safari’s scriptable dictionary, using the AppleScript Script Editor. Long ago, John Gruber had written about the frustration with Safari’s tabs not being scriptable, but a glance at the scripting dictionary showed me this was no longer the case (and probably hasn’t been for years, I haven’t kept track).

I am a complete n00b at AppleScript. I find the attempt at English-like syntax just confuses (and irritates) me no end. But what I wanted looked achievable with it, so I armed myself with some examples from Google searches, and Apple’s intro pages and managed to get what I wanted working. It may not be the best possible solution (in fact I suspect the string concatenation may be one of the most pessimal methods), but it Works For Me™.

In Script Editor, paste in the following:

set url_list to ""
-- change WebKit to Safari if you are not running nightlies
tell application "WebKit"
  set window_list to windows
  repeat with w in window_list
    try
      set tab_list to tabs of w
      repeat with t in tab_list
        set url_list to url_list & name of t & "\n"
        set url_list to url_list & URL of t & "\n\n"
      end repeat
    on error
      -- not all windows have tabs
    end try
  end repeat
  set the clipboard to url_list
end tell

I had to use AppleScript Utility to add the script menu to my menu bar. From there it was easy to create script folders that are specific to both WebKit and Safari and save a copy of the script (with the appropriate substitution, see comment in script) into each folder. Now I can copy the title and URL of all my open tabs onto the clipboard easily again, without any InputManager hacks.

I had some recollection that is a way to do this from Python, so I looked and found Appscript. I was able to install this with a simple easy_install appscript and quickly ported most of the applescript to Python. The only stumbling block was that I couldn’t find a way to access the clipboard with appscript, and I didn’t want to have to pull in the PyObjC framework just to write to the clipboard. So I used subprocess to call the command-line pbcopy utility.

#!/usr/local/bin/python
from appscript import app
import subprocess
tab_summaries = []
for window in app('WebKit').windows.get():
    try:
        for tab in window.tabs.get():
            name = tab.name.get().encode('utf-8')
            url = tab.URL.get().encode('utf-8')
            tab_summaries.append('%s\n%s' % (name, url))
    except:
        # not all windows have tabs
        pass
clipboard = subprocess.Popen('pbcopy', stdin=subprocess.PIPE)
clipboard.stdin.write('\n\n'.join(tab_summaries))

The remaining hurdle was simply to put the Python script I’d written into the same Scripting folder as my AppleScript version. For me this was ~/Library/Scripts/Applications/WebKit/. When run from the scripts folder, your usual environment is not inherited, so the #! line must point to the version of Python you are using (and which has Appscript installed). You should also make the script executable. Adding .py or any other extension is not necessary.

Overall, while I found AppleScript to be very powerful, and not quite as painful as I remembered, I found the Python version (warts and all) to be easier to work with. Combined with the fact that the script folder will run non-Applescript scripts, this opens up new worlds for me. I have hesitated in the past to write a lot of SIMBL-based plugins, tempting though it may be, because they are hacks, and they run in every Cocoa-based application. But adding application-specific (or universal) scripts, in Python, is pure, unadulterated goodness.

This is something of a followup to earlier posts Drawing Hexmaps and Saving PNG from PyGame. A recent comment from Roger Aisling on the second of those posts asked about drawing with opacity in the Python Image Library (PIL). A check of the PIL API didn’t turn up anything, but there is an optional add-on to the PIL called aggdraw, which is based on Anti-Grain Geometry (AGG), a graphics library that seems to cover similar ground to Cairo. I’ve been meaning to play with both of these, but don’t know enough about either project to know what is better about one vs. the other. I know Cairo is used by Mozilla, GTX+, Mono, Inkscape, and WebKit (under GTX+). I wasn’t able to find much about where AGG is used.

Anyway, here are some examples showing how to draw polygons with variable opacity (or alpha). I’ve got examples for NodeBox, aggdraw, and pycairo. I was going to do some for the browser canvas, Processing, PyObjC, and SVG, but a) I wanted to get this post out, and b) PyCairo can already output to SVG and Cocoa (and Processing will be very similar to NodeBox). So if there is any interest in those examples, or others I may have forgotten, let me know in the comments.

While the images below look identical (at least they do to me), each was drawn with its respective library.

For each library I will present the drawing function and the main function that sets things up and drives it. At the end I will present the utility functions used which were the same for all libraries. No optimizations here, just the simplest thing I could do that worked.

NodeBox

I began with NodeBox, because it is so easy work with. In Processing they call programs “sketches” and in NodeBox that is very much what it feels like. The first thing I worked up was a way to draw “random” polygons in the same way for each libary. I could seed the random with a set value, but decided to use random to generate starting values, then just re-use those initial values. To get 50 different polygons, I used a list of primes, chose from the list at random, then used those selected values to increment each polygon vertex and color value (red, green, blue, and alpha). All of which was probably more work than was strictly necessary, but should be stable across different versions of Python and different operating systems.

The drawing routines all have the same signature. They take a tuple of point tuples (x,y) and a tuple of colors (r,g,b,a) as ints between 0 and 255.

def nodebox_poly(pts, clr):
    fill(*ints2floats(clr)) # convert ints in range 0-255 into floats between 0.0 and 1.0
    beginpath()             # every time we want to change colours or linewidth we need to start a new path
    moveto(*pts[-1])     # start and end at the last point in the list
    for pt in pts:
        lineto(*pt)
    endpath()

def nodebox_main():
    size(WIDTH, HEIGHT) # nodebox wants to know how big a canvas you are working with
    draw_all(nodebox_poly) # No additional setup needed

PyCairo

The lines in cairo_main where we create the surface and where we save it are all we would have to change if we wanted to generate a Cocoa view, an SVG document, a PDF document, etc. Two of the main differences between this and the NodeBox example are that there is a bit more setup (Cairo doesn’t default to a white background) and we need to have a reference to the context for drawing, where NodeBox has an implicit drawing context.

def cairo_poly(pts, clr):
    ctx.set_source_rgba(*ints2floats(clr))
    ctx.move_to(*pts[-1])
    for pt in pts:
        ctx.line_to(*pt)
    ctx.close_path()
    ctx.fill()

def cairo_main():
    # Setup Cairo
    import cairo
    global ctx
    surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, WIDTH, HEIGHT)
    ctx = cairo.Context(surface)
    # fill background white
    cairo_poly(((0,0),(WIDTH,0),(WIDTH,HEIGHT),(0,HEIGHT)),(255,255,255,255))
    draw_all(cairo_poly)
    surface.write_to_png('cairo_example.png')

aggdraw

The aggdraw example is very similar to Cairo. In fact, if I’d done either of these first, I would have passed the context object around and simply ignored it in NodeBox to eliminate a global variable. Aggdraw can draw in memory without using PIL, but doesn’t appear to have a way to write to a PNG image that way, so we draw to a PIL image (which requires us to flush all drawing operations before writing the file).

def aggdraw_poly(pts, clr):
    import aggdraw
    global ctx
    b = aggdraw.Brush(clr, clr[-1])
    pts2 = []
    for p in pts:
        pts2.extend(p)
    ctx.polygon(pts2, b)

def aggdraw_main():
    import aggdraw, Image
    global ctx
    img = Image.new('RGBA', (WIDTH, HEIGHT), "white")
    ctx = aggdraw.Draw(img)
    draw_all(aggdraw_poly)
    ctx.flush()
    img.save('aggdraw_example.png')

Utilites

These were the same for all of the examples. If you put it all in one file (or use mine, linked below) you can choose what library to use by switching which main function is called.

MIN_ALPHA = 50
MAX_ALPHA = 100

WIDTH = 500
HEIGHT = 250

#
#   Utilities
#
def hex2tuple(hex_color):
    return tuple([int(hex_color[i:i+2], 16) for i in range(1,9,2)])

def tuple2hex(tuple_color):
    return "#%0.2X%0.2X%0.2X%0.2X" % tuple_color

def ints2floats(tuple_color):
    return tuple([c / 255.0 for c in tuple_color])

def inc_point(p, dp):
    return (p[0] + dp[0]) % WIDTH, (p[1] + dp[1]) % HEIGHT

def inc_triangle(t, dt):
    return tuple([inc_point(t[i], dt[i]) for i in range(3)])

def inc_color(c, dc):
    new_c = [(c[i] + dc[i]) % 256 for i in range(3)]
    new_a = (c[3] + dc[3]) % MAX_ALPHA
    if new_a < MIN_ALPHA: new_a += MIN_ALPHA
    new_c.append(new_a)
    return tuple(new_c)

def draw_all(draw_fn):
    triangle = start_t
    color = start_c
    for i in range(50):
        triangle = inc_triangle(triangle, dt)
        color = inc_color(color, dc)
        draw_fn(triangle, color)

#
#   Starting and incrementing values
#
start_c = hex2tuple('#0xE6A20644')
start_t = (127, 132), (341, 171), (434, 125)
dt = (107, 23), (47, 73), (13, 97)
dc = 61, 113, 109, 41

As you can see, these libraries are all quite capable of producing nice vector-based, anti-aliased polygons with alpha blending. I have heard anecdotal evidence that AGG is faster than Cairo, but Cairo appears to be more widely used. Aside from that I don't know why one would be preferred over the other (unless you want to get into language wars: AGG is written in C++, Cairo is written in C). Both have bindings in a number of languages besides Python, if that's your thing. NodeBox is still my favourite tool for noodling around in with graphics programming. All of these are great fun, easy to use, and handy to have in your bag of tricks.

I tested this under OS X (required for NodeBox), using pycairo installed via fink (using fink's Python 2.5) and aggdraw installed via easy_install (using the builtin in Python 2.5).

Here is my test file containing all of the code above: Alpha polygons example code

I’ve gotten a couple of messages asking why there are strange attachments in my email which may or may not prompt you to open your security settings if you try to open them. Fear not, these are not viruses! Rather, it’s the basis of email itself if we ever want to move beyond the ocean of spam we currently swim in.

First some background. When you send an email, by default there are three things which do not occur. The first is Authorization, i.e., there is no guarantee that the person you are sending the mail to is able to read it. The second is Encryption, which ensures that other people are not able to read it. The third is Authentication, proving that the mail which appears to be from me, actually is from me. None of these are present in normal email.

In fact, sending an email without these is like putting all of your standard (snail) mail on postcards, then having the postcards travel through the houses of random strangers until they (possibly) end up at their destination. Adding authentication, authorization, and encryption is like putting your email in an envelope.

So that is what the obscure attachment is in my email, it’s a digital signature, which provides some form of authentication (better than nothing). I recieved this certificate from a company called Thawte, for free, and installed it in my primary email application, Mail.app. There are very good instructions for doing this at http://www.joar.com/certificates/. Once you have installed the certificate, signing your mail happens authomatically and transparently. The process of getting the certificate could be easier, but it’s certainly not difficult.

The real benefit happens when more people start doing this. See, if I have received mail from someone who also uses digital signatures, then Mail.app remembers this, and when I respond it uses their signature and my signature in combination to get the rest of the picture: Authorization and Encryption. If we can get to the point where the standard is to put messages in an envelope instead of postcards, it makes much better tools for eliminating spam available–don’t accept mail unless it is from a verifiable source. It’s not perfect, but a big improvement over the situation we have today.