Tab Dumping with AppleScript and back to Python

Rock

Goal: Iterate through all my (OS X Safari) browser windows and make a list of titles and urls which is then placed in the clipboard ready to be pasted into an email or blog post.

This is an update to Tab Dumping in Safari. That still works well as the basis for extending any Cocoa-based application at runtime, but it relies on SIMBL, which while it is a great bit of code, essentially is abusing the InputManager interface. Some developers and users shun such hacks, and at least one Apple application checks for them at startup and warns you from using them.

I have been running the WebKit nightlies, which are like Safari, but with newer code and features (most importantly to me right now, a Firebug-like developer toolkit). WebKit warns at startup that if you’re running extensions (such as SIMBL plugins) it may make the application less stable. I was running both Saft and my own tab dumping plugin, and WebKit was crashing a lot. So I removed those and the crashes went away. I miss a handful of the Saft extensions (but not having to update it for every Safari point release), and I found I really miss my little tab-dumping tool.

I toyed with the idea of rewriting it as a service, which would then be available from the services menu, but couldn’t figure out how to access the application’s windows and tabs from the service. So I tried looking at Safari’s scriptable dictionary, using the AppleScript Script Editor. Long ago, John Gruber had written about the frustration with Safari’s tabs not being scriptable, but a glance at the scripting dictionary showed me this was no longer the case (and probably hasn’t been for years, I haven’t kept track).

I am a complete n00b at AppleScript. I find the attempt at English-like syntax just confuses (and irritates) me no end. But what I wanted looked achievable with it, so I armed myself with some examples from Google searches, and Apple’s intro pages and managed to get what I wanted working. It may not be the best possible solution (in fact I suspect the string concatenation may be one of the most pessimal methods), but it Works For Me™.

In Script Editor, paste in the following:

set url_list to ""
-- change WebKit to Safari if you are not running nightlies
tell application "WebKit"
  set window_list to windows
  repeat with w in window_list
    try
      set tab_list to tabs of w
      repeat with t in tab_list
        set url_list to url_list & name of t & "\n"
        set url_list to url_list & URL of t & "\n\n"
      end repeat
    on error
      -- not all windows have tabs
    end try
  end repeat
  set the clipboard to url_list
end tell

I had to use AppleScript Utility to add the script menu to my menu bar. From there it was easy to create script folders that are specific to both WebKit and Safari and save a copy of the script (with the appropriate substitution, see comment in script) into each folder. Now I can copy the title and URL of all my open tabs onto the clipboard easily again, without any InputManager hacks.

I had some recollection that is a way to do this from Python, so I looked and found Appscript. I was able to install this with a simple easy_install appscript and quickly ported most of the applescript to Python. The only stumbling block was that I couldn’t find a way to access the clipboard with appscript, and I didn’t want to have to pull in the PyObjC framework just to write to the clipboard. So I used subprocess to call the command-line pbcopy utility.

#!/usr/local/bin/python
from appscript import app
import subprocess
tab_summaries = []
for window in app('WebKit').windows.get():
    try:
        for tab in window.tabs.get():
            name = tab.name.get().encode('utf-8')
            url = tab.URL.get().encode('utf-8')
            tab_summaries.append('%s\n%s' % (name, url))
    except:
        # not all windows have tabs
        pass
clipboard = subprocess.Popen('pbcopy', stdin=subprocess.PIPE)
clipboard.stdin.write('\n\n'.join(tab_summaries))

The remaining hurdle was simply to put the Python script I’d written into the same Scripting folder as my AppleScript version. For me this was ~/Library/Scripts/Applications/WebKit/. When run from the scripts folder, your usual environment is not inherited, so the #! line must point to the version of Python you are using (and which has Appscript installed). You should also make the script executable. Adding .py or any other extension is not necessary.

Overall, while I found AppleScript to be very powerful, and not quite as painful as I remembered, I found the Python version (warts and all) to be easier to work with. Combined with the fact that the script folder will run non-Applescript scripts, this opens up new worlds for me. I have hesitated in the past to write a lot of SIMBL-based plugins, tempting though it may be, because they are hacks, and they run in every Cocoa-based application. But adding application-specific (or universal) scripts, in Python, is pure, unadulterated goodness.

Mac OS X Software Favorites Part One: Basics

Yesterday was my last day at my job for the past five years, and with it I left my Macbook Pro (it was a nice tool, but they own it). Coincidentally, I’m now setting up my new Macbook. Since I know several other people who are setting up new Macs, I thought I’d give my thoughts on some of the best software available. Since that’s a big topic, I’m going to break it up into several posts, starting with the basics: Application Launcher, Text Editor, Web Browser, Newsreader

Macbook vs. Macbook Pro

Before diving into the software, just a note on the hardware. I’m going with a black Macbook for several reasons. First, it looks really cool, and I can’t deny that appeals to me. Second, it is somewhat smaller and lighter than the smallest Macbook Pros, and since I carry my laptop to and from work every day by foot or bicycle, that matters (I’m getting two power bricks so I can leave one in each location to save even more weight). Third, the wifi reception is much better. Encasing an antenna in plastic appears to work better than encasing it in aluminum, go figure. Macbook Pros are more powerful, but only a bit, and mainly for heavy-duty 3D, which isn’t really my thing. We’ll see how much this matters in practice. White vs. Black Macbook: the white ones are rumored to stain really easily, plus did I mention that the black ones look really freaking cool? The Macbook is cheaper than the Macbook Pro, but that’s not a big factor in why I’m choosing it (since once again, work will own it, just a different work–more on that later). The Macbook also gets better battery life, is slightly less crotch-scaldingly hot, and so far I’m liking the keyboard a lot.

Application Launcher: Quicksilver

I used to use TigerLaunch (free, open source), and still do on other Macs in the family, but once you’ve tried Quicksilver (free, beta) there’s no going back. I’m still working on becoming a Quicksilver power user, but just the app-launching and file-searching facility is worth using this tool for. TigerLaunch is great for its simplicity, and Quicksilver is great for making you wonder how you ever got along without it. I’ve heard wonderful things about PathFinder ($34.95USD) and keep meaning to try it, but haven’t gotten to it yet. First I need to learn how to write Quicksilver plugins in Python…

Editor: TextMate

I used to use Pico on Unix and BBEdit Lite on Mac OS 7, back in the day. I couldn’t stand either vi or emacs. I wanted a T-shirt that read, “I’d rather die than use vi.” Then I worked at a company where the only editor you could count on being installed was vi and I began to get its keystrokes etched into the memory of my fingers. I’m not even that good at it, but sometimes vi can be so damn fast. Of course, I don’t use classic vi, but a nice modern Vim (free, open source). I’ve tried to switch back to Mac editors a few times: TextWrangler (free, the OS X incarnation of BBEdit Lite), SubEthaEdit ($35USD, great for collaborative editing), Smultron (free, open source). These are all great editors, but each one lacked something that kept me from switching over completely. I had been hearing about TextMate (€39EUR) for some time, but had trouble getting excited over a commercial editor when there were so many free ones to choose from. I finally gave it a serious try and I’m hooked. It still isn’t as easy to search as vi is, but the way it can be expanded on with plugins and the general fit and feel are great.

Browser: Safari + Saft

Safari is shaping up to be a great browser, fast and powerful. There are a few details that are missing, but nearly all of them are satisfied with the Saft ($12USD) extension, which provides great ad filtering, improves the way windows and tabs are handled, and gives shortcuts for accessing search sites from the address bar, among many other features. Safari itself takes two kinds of plugins, Netscape-style plugins and WebKit plugins. Unfortunately, these are only triggered when their target mime types are loaded, so they can’t readily be used to extend the way the application itself works. So all of the Safari extensions that I’m aware of are implemented as InputManagers (see my post on TabDump for more on InputManagers). My TabDump extension is the other extension that I find indispensable, which is why I wrote it, but more important than TabDump itself is the example it gives for writing your own extensions to Safari or any other (Cocoa) application. That’s what I love about OS X, you can get into it and take control of your own machine, make it your own. You can do that with Linux, of course, but the high-level of Cocoa applications make a huge difference. Back to browsers, I also recommend the Flip4Mac plugin, to allow your browser to play Windows Media files directly. Finally, I also keep Firefox (free, open-source) around for its advanced features. It’s not my main browser, but it has features that show where the browser (and all applications) will be going in the future. I may have to do a separate post just about the cool features coming in Firefox 2 (currently in beta) and Firefox 3. Other good browsers to keep around for testing are Camino (the Firefox engine with a Cocoa UI) and Opera (free). For more Safari plugins and extensions than you could ever use, check the listings at Pimp My Safari.

Newsreader: NetNewsWire

I realize that Safari now supports RSS feeds in some fashion, but it really isn’t a newsreader. I’m not crazy about trying to put every possible application into the browser. If you have to work betwwen Windows, OS X, and Linux, or some combination thereof, then things like GMail and Bloglines can be a godsend, otherwise you’re probably better off with a dedicated desktop app. NetNewsWire ($29.95USD) is a desktop app that is so good, it was the most popular newsreader (by far) for a long time, even though it only ran on OS X. Since being acquired by NewsGator, it’s still as good, but now synchronizes your feeds with the NewsGator site, so you get the best of both worlds: a top-notch desktop app, and a webapp that stays sychronized with it. I have a feeling that in the near future, nearly all applications will work this way. And if you can’t afford the price, they still offer NetNewsWire Lite for free (and I happily used it for a couple of years before upgrading).

Mail: Mail.app

Apple gives you a decent email program with OS X, nothing too fancy. It keeps getting better (mostly) with each new release. There has been some complaints about Mail.app in OS 10.4 changing to a proprietary format (which they did in order to integrate with Spotlight) and I’ll have more to say about that in a future post. I have tried using Gmail (when I was mostly using Windows at work) and Thunderbird (free, open-source), and coming back to Mail.app is like a breath of fresh air. For nearly everything I do, Mail.app is better. It’s far from perfect, and sites like HawkWings specialize in plugins and extensions for Mail.app, but it sure works for me. I do wonder why email programs are still so hard to get right. Since email is basically the oldest use of the internet (and networks generally), and the original “killer app,” if we don’t know how to do email yet, what hope do we have for anything that’s actually complicated? Of course, perhaps because they have been around for so long, choice of email programs tends to be very personal, so all I can say is, Mail.app works for me. I also recommend the AttachmentScannerPlugin by James Eagan, who also provides a tutorial on writing plugins for Mail.app which is generally applicable to extending Cocoa programs and using their undocumented private APIs. Also, he uses Python and PyObjC to write the plugin, which makes me happy.

Regular reader(s) were probably wondering how I was going to get that plug for Python and PyObjC in there, eh?

Tab Dumping in Safari

The Problem

I first saw the term “tab dump” on Dori Smith’s blog, but I immediately recognized the concept. I keep Safari running all the time and with the help of Hao Li’s wonderful extension Saft I keep everything in tabs in one window. Among its many features, Saft will let you consolidate your windows into tabs of one window, and it can save the tabs you have open when you close (or crash) Safari, and re-open them automatically when you start Safari again. What it doesn’t do is give you a list of all the tabs you have open in text format, suitable for blog or email. I don’t currently put tag dumps on the blog because a) I’d feel guilty doing that without adding at least a short comment for each link, which would take too much time, and b) because this isn’t really a link blog, more a place for me to bash out example code and tutorials. At least, that’s how I think of it.

I do however, find Safari teetering on the brink of being unfunctionally slow because I have so many tabs open, and often they’re only open because I want to remember to do something with them later, or come back to them, or some other reminder-type function. So I send myself a tab dump on a more-or-less daily basis. Firefox has tools to help you do this, but I haven’t seen anything for Safari, possibly because you can’t really do it with a Safari plugin, but need to use an InputManager, which is fairly deep magic, and basically a hack, an abuse of the system.

On the other hand, I couldn’t keep using Safari if it wasn’t for Saft, and Saft is an InputManager. Another tool for blocking ads and such (which Saft also does) is PithHelmet, but the interesting thing to me about PithHelmet isn’t that it is a popular ad blocker, but that the Mike Solomon (who wrote PithHelmet) decided to not just make an InputManager, but to make the only InputManager you’ll ever need. You see, PithHelmet itself is not an InputManager, it is a plugin for SIMBL (also by Solomon), which is an InputManager that loads plugins based on the application(s) they claim to support. InputManagers get loaded by every application (Cocoa apps, at least), so you have to be careful you’re in the app you want to modify, and take steps not to break things. SIMBL takes care of the nasty business of being a well-behaved system hack, and your code can assume it is in the right app, because it doesn’t get loaded otherwise.

The Goal

Once I figured out that the only way I was going to get Tab Dumping behaviour into Safari (because Safari tabs don’t play well with Javascript, that turned out to be a dead-end), I decided to try writing an InputManager in Python. SIMBL is open-source, so at first I was looking at the code to see what I need to do to create an InputManager (remember, this is a hack, so Apple doesn’t document it very well). I also read Mike’s essay Armchair Guide To Cocoa Reverse Engineering. What I decided was that, rather than recreate the functionality in SIMBL using Python, I would just create a SIMBL plugin in Python.

Getting started wasn’t too bad, but I found one issue in the above essay that stumped me for awhile. Mike recommends you put your initialization code into a class method load() which gets called after your class is loaded. I don’t know if it is artifact of using PyObjC or what, but my load() method was never getting called. What I did instead was to run the command-line utility class-dump on another SIMBL plugin to see what they were doing. They were using the class method initialize() rather than load and when I switched to that things started working, where by “things” I mean, “I could print to the console to see that my class had loaded.”

The Solution

The next step was to actually do something once I had my code loading into Safari. The tab behaviour of Safari isn’t part of WebKit, so it isn’t documented anywhere. Once again, I used the handy class-dump utility. This is a fabulous tool which will read any Cocoa library, bundle, or application and produce a pseudo-header file showing all the objects and methods defined. I still had to try a few different paths to get to the tab information I wanted, but it was pretty easy, armed as I was with Python and the output of class-dump. Here is the result:

import objc
from Foundation import *
from AppKit import *
class TabDump(NSObject):
    # We will retain a pointer to the plugin to prevent it
    # being garbage-collected
    plugin = None
    @classmethod
    # the following is not strictly necessary, but we only
    # need one instance of our object
    def sharedInstance(cls):
        if not cls.plugin:
            cls.plugin = cls.alloc().init()
        return cls.plugin
    @classmethod
    def initialize(cls):
        app = NSApp()
        menu = app.windowsMenu()
        cls.item = NSMenuItem.alloc().initWithTitle_action_keyEquivalent_(
            'Dump tabs to clipboard',
             'tabdump:',
             '')
        # should be after "Previous Tab" and "Next Tab"
        menu.insertItem_atIndex_(cls.item, 6)
        cls.item.setTarget_(cls.sharedInstance())
    def tabdump_(self, source):
        output = []
        app = NSApp()
        for window in app.windows():
            if window.className() == 'BrowserWindow':
                controller = window.windowController()
                for browserWebView in controller.orderedTabs():
                    output.append(browserWebView.mainFrameTitle().encode('utf8'))
                    output.append(browserWebView.mainFrameURL().encode('utf8'))
                    output.append('')
        self.copyToPasteboard_('\n'.join(output))
    def copyToPasteboard_(self, string):
        pasteboard = NSPasteboard.generalPasteboard()
        pasteboard.declareTypes_owner_([NSStringPboardType], self)
        pasteboard.setString_forType_(string, NSStringPboardType)

As you can see, on my class being initialized, I create a new menu item and insert it into the Windows menu. This could be more robust, by testing menu item names to make sure I’m in the right place, but it works for me, and simple code is more maintainable code. I create an instance of my object and make it the target of the menu item. Pretty basic stuff.

When the tabdump method is called (by selecting the menu item in Safari), it walks through Safari’s window objects (of which there are many) until it finds browser windows, then it extracts the tabbed views from the browser windows to get the titles and URLs involved. When it has collected all the title/URL pairs, it turns it into a big string and puts the string on the pasteboard. Here is where we could be a lot fancier. I’m just putting title/URL pairs, separated by newlines in plain text, because that’s how I mail them to myself. You could easily create Markdown links or any other format here. You could turn them into HTML and put them on the Pasteboard that way. There’s a lot you can do, and the Firefox tool I used to use to do this offered so many options that I was never sure what most of them actually did. Here you can customize the code to do exactly what you need, and keep it simple.

Building the plugin

I haven’t tested this with multiple windows, or with a window with only one tab. It might work, might not. I don’t plan on using it that way, and if I do, it’s easy enough to fix. Now, there is one more thing you’ll need, which is the setup.py script to build it. Assuming you’ve saved the above code as TabDump.py, the following script should be what you need:

'''
    Minimalist build file for TabDump.py
    To build run 'python setup.py py2app' on the command line
'''
from distutils.core import setup
import py2app
plist = dict(
    NSPrincipalClass='TabDump',
    CFBundleName='TabDump',
    SIMBLTargetApplications=[
        dict(
            BundleIdentifier='com.apple.Safari',
            MinBundleVersion='312',
            MaxBundleVersion='420')],
)
setup(
    plugin=['TabDump.py'],
    options=dict(py2app=dict(
        extension='.bundle',
        plist=plist,
    )),
)

In the above file, MinBundleVersion and MaxBundleVersion can keep your code from being loaded if an untested version of the application is running. I have more-or-less dummy values there, don’t treat them as the right thing to do. The SIMBLTargetApplications key holds a list, so if you want your code to load in other applications, add more dictionaries to the list.

Also note that you can build your bundle with python setup.py py2app -A to create a development version (can’t ship it that way) that is all symlinks, so you can edit TabDump.py to make changes without having to rebuild the plugin. If you modify the MinBundleVersion or MaxBundleVersion you will have to rebuild to regenerate the property list (or move the property list to be an external file rather than generating it in setup.py), but that should be an infrequent operation. More importantly, you can put a symlink to your bundle in your ~/Library/Application Support/SIMBL/Plugins/ directory. Then you can make changes to the python code and test it by simply restarting Safari. WARNING: If you have a syntax error in your file, Safari will most likely hang on restart. Just force quit it and check your console for the error to fix.

The Promise

Now, if you’ve followed along with me so far, I’d like to point out a few things that are really freaking cool about this. Item the first: You now have Python running in Safari. Can you think of anything else you’d like it to do while you’re there? I bet you can. Item the second: You can do this in any Cocoa-based application just as easily. Problems in Mail.app? Frustrated by iChat? Just fix it. Take control of your own applications! Make the computer work for you, not the other way around. Item the third: dump-classes gives you the keys to the kingdom. Seriously, the combination of being able to embed Python and get a listing of the objects and methods at will is so powerful that when I got TabDump working late last night and realized what I’d just done (i.e., these three things), I was barely able to get to sleep after that. The possibilities are endless.

If you use this and do something cool with it, please drop me a line and tell me about it. I’m really looking forward to hearing about what kind of cool ways we can push our existing applications.

Correction [2006-08-30]

The class-dump utility rocks, and you should add it to your arsenal of Cocoa tools, along with Python and PyObjC. Since I’ve found it it has already become indispensable for examining existing applications that I want to, er, adjust. Here’s what I’ve learned so far.

First, I want to update my previous post to talk a little bit more about the command-line utility class-dump. This is a fine tool that lets you introspect a Cocoa bundle (plugin, library, or application) and prints out a header file describing all the objects and methods in that bundle. I didn’t mention where to get it, and at BarCamp this weekend I gave some mis-information by telling people it came with Apple’s developer tools, which is not true. I assumed that’s where it came from, because I didn’t remember hearing of it before reading Mike Solomon’s Armchair Guide to Cocoa Reverse Engineering, which refers to classdump without any explanation of where to get it. I tried it, found class-dump worked (tab-completion is your friend), and assumed it came with my system, when in fact I had installed it earlier after reading about it on another blog (I’m afraid I don’t remember where) meaning to try it out, then forgotten about it. So it was there, waiting for me, when I discovered a need for it.

So the truth is, class-dump is a utility written by Steve Nygard. He says it provides the same output as the developer tools command otool -ov, but formatted as a header file. Besides the basic output it can also do various kinds of filtering, sorting, and formatting.
So this is my Tool of the Week (and then some): class-dump. Use it, love it, thank Steve.

google

google

asus