Johns Blog

The short news is that Conduit now lives in GNOME SVN. The move took a little longer than I would have hoped, but thats largely my fault. I would like to express my thanks to Olav Vitters for importing the SVN repository, his excellent response time to my emails, and the awesome mango system he created.

Whats Good

Conduit trunk is now pretty much feature frozen. It feels really good to finally cross off the TODO list and fix some of the bugs that I created (due to some poor choices when I started the project). While I will blog again soon about the shiny new features for the moment I will talk about one useful piece of code that I am proud of, and that may be useful to others.

Initially I used an in-memory python dictionary based database to store the object mappings which represent a sync relationship between two pieces of data (the mapping DB). I pickled this to disk when conduit closed. Unfortunately this was not only horribly gross (memory usage was huge), it was embarrassingly short sighted, as the one place I care most about types is this code path. We use dictionaries and object hashes extensively in the sync logic and its really hard to debug sync problems when you (for example) don’t notice that a dataprovider returns the wrong type until you unpickle it a few days later.

To remedy this I needed (wanted) to use a database that had few or no external dependencies, that was a bit tougher on type safety, and that was usable from multiple threads. After a while I decided that because sqlite was now part of python2.5 this would be a good choice, and that I could get around the threading issues some other way.

Not wanting to re-invent the wheel completely I started with a few existing pieces of code

• Christian Hergert had written a nice Gtk.TreeModel wrapper around a sqlite DB
  http://vwdude.com/dropbox/pystore/
• Louis RIVIERE had written a threadsafe wrapper around sqlite that allows the DB to be accessed from more than one thread
  http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/526618

I glued these together and created a really nifty (yet another) DB abstraction layer that can be used in either single or multiple threaded applications, and then displayed in a Gtk.TreeView with NO additional effort. Whats more, the DB also has a LRU cache that dramatically reduces the number of calls into the DB when displayed in a Gtk.TreeView. For more information check out the following

• The Database abstraction (single and multiple thread compatible)
• The Gtk.TreeModel wrapper
• An Example of how to use all these parts together
  
  

All of this has been really heavily tested and everything seems to work as expected*…….. except for the LRU cache, which I broke somewhere - hmm. I should really fix that.