Johns Blog

End of an Era: PyGTK

I just released PyGTK 2.24, which will almost certainly be the last major PyGTK release. The future of Python on the GNOME platform is PyGObject + GObject Introspection. From my experience over the last few months porting a number of my projects, the future is bright.

In a cruel twist of irony, the state of PyGTK on Windows and Mac has never been better. The credit for the windows work (and some great documentation improvements this cycle) must go to Dieter Verfaillie

• Windows installer (homepage) (download)

• Mac improvements (all in one installer work) (bundling advice)

I hope that the new stack will reach the same level of capability on other platforms as GTK+ 2.24, but in the large scheme of things the renewed development excitement surrounding GTK+ 3.0 and GNOME 3 is excellent consolation.

As a user I would like to thank those developers before me for creating PyGTK. It was the first 'pythonic' UI toolkit for linux, and a pleasure to use. As the recent maintainer of PyGTK I would especially like to thank those recent developers who helped me, in particular Dieter Verfaillie who really pushed PyGTK over the line regarding Windows support, into the great state it is now.

I'll leave you with some graphical statistics (generated using pepper) for the 12 year history of PyGTK. If planet strips the wordpress gallery then please click here.

update: To clarify a point raised in the comments, PyGTK will be maintained in the exact same was as the GTK+-2.0 series will be maintained. Bug fix releases will be made if necessary, but no new features will be added. If you want the new GTK+-3.0 features then you should use PyGObject + GObject Introspection.

The PyGTK code will not disappear from any servers, it will continue to be shipped in all distributions for the forseeable future, it will continue to work very well on windows, and many applications will continue to use it.

Map Updates

I recently made a number of improvements to osm-gps-map, the easy to use mapping widget. The motivation for these came at the request of the foxtrotGPS developers (foxtrotGPS is a community developed fork of TangoGPS). These changes enhanced the API for adding images and tracks to the map, and in addition allowed me to clean up the basic API making it easier to use for the common case. But, there is more, especially relevant to Gtk+/GNOME 3.0.

• Introspection Support (JavaScript example)

• GTK+ 3.0 Compatibility

• API Documentation (finally...)

• Improved examples (C, Python, JavaScript)

Misc Hacking

I had two days off while I moved offices, so I got a chance to catch up on my backlog of random hacking.

osm-gps-map

I released osm-gps-map v0.5.0 which adds a few new features (such as keyboard navigation) but also contains many bugfixes and performance improvements. Check the release notes for more information. The next item on the TODO is merging the OSD/layers branch.

Conduit

I released Conduit 0.3.17 which was long overdue. Mostly a bugfix release and updating to new API. The Conduit homepage has also moved to live.gnome.org. Progress on Conduit is a bit slow at the moment, it does everything I want it to (I have a budget cellphone so phone synce does not interest me), and is pretty stable. I have some SOC work I would like to merge, but basically I am looking for developers and inspiration...

PyGTK for Windows

I finished off the fixes to build correct PyGTK+ installers on windows, hopefully closing bug #589671. I uploaded new installers with the fixes people have reported. I expect these installers to become the 'final' installers at some point. Feedback welcome.

• Stable (requires the Gtk+-2.16 bundle)

  • PyGObject 2.20 for Windows

  • PyGTK 2.16 (no Glade) for Windows

  • PyGTK 2.16 (with Glade) for Windows

• Update: I also built

  • PyCairo 1.8.6 for Windows

PyGTK Hacking

I wanted to play with the new client side windows work in Gtk+, so I ported the effects gtk-demo to Python. This required a bit of ctypes magic to access the new API (good), and some more ctypes magic to interact with new signals that appears to have unfriendly prototypes (not so good, bug filed here).

A Frustrating Week

It all started so well. While Conduit was not accepted into GNOME 2.24, it was blessed as an external dependency for this cycle. That is great news for online service integration, and synchronization on the GNOME desktop. Congratulations to everyone who has helped me work on Conduit over these years, and well done to those who made it into the release set.

I was also able to make a 0.3.13 release incorporating those fixes and UI improvements I discussed last time around. After that things started to get worse.

1. The disk holding my /home partition crashed. Taking a whole bunch of stuff with it. Like a good geek most of it was backed up or in a RCS, with the exception of my SSH key. FAIL.

2. This meant I was unable to make the now customary Conduit x.x.x.1 brown paper bag release in time for the GNOME release. I do owe many thanks to our amazing sysadmin, Olav, for getting my new key onto the GNOME servers in a matter of hours.

3. The disk crash took with it a week+ of my SOC work, making me very angry. To calm the tempers I thought I would work on the conduit-gio branch.

   1. Unfortunately I am not certain if this has much chance of success. I seem to have hit a regression/gvfs bug #547133.

4. Then Christian Schlotter contributed a really nice Conduit dataprovider which supports sync/backup to Amazon S3.

   1. But this too seems blocked by a gvfs bug #547020.

5. I then realized the implications of Webkit not being blessed as an external dependency. It means that I have two choices. Continue to drag around the pain that pthon-gtkmozembed brings, or move to using the unreleased (Py)WebkitGtk. I really encourage the WebkitGtk and PyWebkitGtk teams to make stable releases ASAP, to ensure they get into $DISTRO.

6. Then I found that I could not push to bzr-playground.gnome.org anymore. It appears that it did not pick up my new SSH key. And loggerhead source code browsing is broken there.

7. Completely fed up with software, I thought I would play with my new hardware toy. This is destined to replace the terrible, unsupported mess that was the Phytec phyCORE LPC3180 module for the purposes of my PhD work.

My brand new beagleboard (photo)

But then the FAIL continued. http://www.openembedded.org has been down for the last 5 days, making it impossible for me to check out the openembedded tree that supports this board. Does anyone know why it is down?

Conduit UI Experiments - Some Conclusions

Thanks once again to the many people who commented on my last two blog posts. All the feedback has been gratefully received. I just released Conduit 0.3.13 which, amongst other things, features some UI improvements based upon this feedback.

Improvements to the Conduit user interface.

Technically speaking, the release adds a knowledge framework to the application. This makes it easy to assist the user as they work through the interface. New features, utilizing this framework are;

• Conduit now ships with a few example synchronization groups. These groups are shown to the user based upon the devices connected, and the data providers loaded. When the user selects an example, it is added to the canvas automatically.

• I have added a message area to the bottom of the window. This provides a limited number of hints to the user depending on their interactions with the application.

Coming Up This release also lay a lot of the foundations for the few remaining big ticket items on the 0.4.0 TODO list. In the very near future we will see

• Merging GIO port

• Merging of Alexandre's SOC work, which will improve sound and video support in Conduit

• The first release supporting Windows Mobile synchronization

• Miscellaneous PIM improvements (Improved Google support, ability to merge attributes during conflict)

John Carr is currently working on the ATK accessibility implementation. As far as we can tell, there are no other PyGTK applications that have implemented accessibility in manner we wish to (manual implementation of the Atk interfaces from a objects not derived from GtkWidget). Lucly he works with Mark!

As far as using and distributing Conduit goes, it may get painful before it gets better. The next release will likely depend on PyGObject > 2.17 (for GIO support), and maybe PyWebkitGtk (SVN because of lack of recent, stable, compatible releases of this and WebkitGtk). My experience with (py)GIO so far has been exceptionally positive, and those out there who have used the train wreck that is gtkmozembed before will appreciate my desire to move to PyWebkitGtk.

New Conduit Features

Last week (or so..) I released Conduit 0.3.9, the notable features of this release were the addition of documentation, and dramatically improved support for removable devices, like USB keys and portable hard drives.

Conduit has always supported removable disks, but the UI for working with them has been dramatically improved. In the screenshot below you will see Conduit automatically suggesting pre-configured dataproviders to sync with folders on the removable device. The use case for this is

1. Joe synchronizes stuff from one PC to his USB key.

2. When Joe plugs that USB key into another computer also running Conduit he is presented with a pre-configured method to synchronize information from the usb key to a location on his local machine.

The functionality is not all that different to before (it shares all the same code as the traditional folder <--> folder sync, meaning you get conflict detection etc) but the way it is now presented is more sensible and discoverable.

In the screenshot you will also see the first commit of mobile phone support in Conduit. Phone discovery is done using bluetooth. Picture sync is implemented atop gnomevfs-obex-ftp. Data is currently fetched from the phone using gammu, but discussion are underway on how this might tie into gnome-phone-manager (which uses gnokii).

This work is (mildly) separate, but necessary for a sensible SyncML (using libsyncml) implementation. In order to provide smart suggestions on what sync options to offer for a particular phone, phones in range (via bluetooth or USB connection) should be enumerated and their capabilities queried. Currently I am using bluez for the former, and gammu for the latter, however, at least in the bluetooth case, the latter could probbably be accomplished using pure AT commands over a serial socket to get the manufacturer and model number and them comparing this with a database of capabilities.

If its not clear, this is still in the experimental stage, but the current approach seems to demonstrate that Conduits architecture, and the approach to implementation are approximately correct.

There have also been a few other fixes go into SVN, including

• The ability to create tags in F-spot from within the Conduit UI

• Improved configuration for picasa web galleries

Finally I will finish with a dear lazyweb.....

Dear Lazyweb; The current implementation of network sync doesnt really scale to pushing large files around. Its using the python pickle module, which has a tendency to read everything into memory, base64 encode it, run out of memory, and then explode. I have considered may ways to solve this

• Use sftp - but that requires the user setup a ssh keypair with the other computer. Seahorse, can you give me a dbus interface to make this happen, easily?

• Re-implement (because the license is CPL), some of the ingenious performance optimizations in conary WRT decreasing XMLRPC memory usage.

• Wait for the GNOME desktop to offer some sort of GiveFileToThisUser() dbus api call. Could perhaps be implemented atop Giver/Telepathy (if... it gets accepted to the desktop....)

• _Custom python module written in C to do this.... _

• Continue doing what I am doing, and wait for people to complain louder....

They come in pairs

Following the tradition of the last Conduit release, I realized that I shipped Conduit 0.3.7 with a stupid build issue that broke all the Gmail/Google/Picasa support for people. Following in that tradition here is Conduit 0.3.8 which fixes that, and also features updated icons contributed by mejogid.

For more information:

• Release Notes

• Download

Conduit developer John Carr has also setup a Ubuntu PPA for Conduit. Check it out if you are interested in running the Latest and Greatest for Ubuntu Gutsy and Hardy.

Conduit 0.3.7: Many Small Updates

I just released Conduit 0.3.7. This was another regular release which contains the usual array of bug fixes and a few small new features and tidy ups. This was also the first release made automatically using maintainer. Aside from a few quirks, the tool worked amazingly well.

The next release will feature rewritten user documentation, more conflict UI improvements, and fix whatever bugs crop up in testing its compatibility with new GNOME. Highlights of this release are;

• Update to use the newer Tomboy Set/GetNoteComplete API. This finally allows lossless sync of Tomboy notes including metadata such as tags and notebooks.

• Update to the latest versions of our shipped static libraries

  • flickrapi (fixes many Flickr bugs)

  • pyfacebook

  • python-gdata (and use this for Picasa support)

  • libgmail (make emails work again)

• The Eye of gnome plugin for uploading photos to multiple sites now shows status while synchronizing

• Add one-way support for syncing Gmail contacts (with evolution, ipod, etc)

• Add a 'developer' sub-menu to the help menu if the user is running a development version. This sub-menu contains items which link to the newly improved developer documentation. The linked pages are displayed in conduit's built in web browser.

• Make the Conflict preferences per Conduit add descriptive icons to the preferences menu

Update: I completely forgot. Thomas Van Machelen recently added C# bindings for our DBus API to SVN. If any of you enterprising Novell Hack Week hackers wants to play with them, I have some ideas (use Conduit to export photos from FSpot, use Conduit as the sync engine for Tomboy, for both synchronization of notes between two computers and/or export to ipod, box.net, etc). Please email me if you are interested. I know Conduit is ready, there are many examples of the Conduit DBus API in SVN.

Conduit 0.3.6: I can has sleep please?

Quote courtesy of John Carr, who I had cook me up a patch at 1:30AM UK time. Ahhh the wonder of a 12h time difference.

I just released Conduit 0.3.6, a quick follow up to the last release. It fixes a number of bugs which were preventing some common sync scenarios.

• Download

• Release Notes

• Report Bugs

• Mailing List

• Test Results

• Bugs Fixed

  • Fix two-way Tomboy <--> folder sync

  • Fix two-way GConf <--> folder sync

  • Dont recreate the backpack API everytime we refresh

  • Fix Shutterfly to support image uploads and delete again (Jeremy Slater)

  • Fix attribution of Banshee dataprovider. It was contributed by Don Smith

  • Many fixes to image dataproviders

OK I lied, there is one new feature..

Conduit 0.3.4 - 37% more integrated

Wahoo!, after mad busy weeks of development, I just released Conduit 0.3.4. Check out the screencast (download here). This release is fresh with new features, and heaps of polish and integration work. These new features include;

• Google calendar support (Paul Novotny).

• Youtube dataprovider added (Renato Araujo).

• Photo sites are now two way, you data is no longer locked with one provider. Also see #480754 for improvements to Fspots DBus interface allowing two way photo sync with FSpot.

• Nautilus extension. (disabled by default).

• EOG plugin added. (disabled by default).

• Conduit now supports resizing and converting photos before uploading them to photo sites. This infrastructure means improved support for other types of conversions, such as transcoding music and video, will be added in the future.

• Progress indicator shows percent complete during synchronization.

• Add support for Facebook photo upload.

• Add included web browser for authenticating against web sites.

Website Authentication

The last point deserves some explanation. Currently, to authenticate against some sites the user must log in either once (Flickr), or once per session (box.net). This is difficult when using the system default web browser, if the browser is currently running on a different virtual desktop to conduit, the user often doesnt notice the web browser go to the login website. This is compounded by the different APIs of some sites. Flickr is good, I can just call am_i_authenticated(token) in a loop, and wait for success, or time out. Facebook is not good. Any call involving the facebook token before it has been validated (i.e. user logs in) invalidates the pending login. Because there is no guarenteed way to block in all cases (i.e. until the user logs in and closes the window), things like gnome.open_url() or python.webbrowser() are unsuitable. This leaves me with two options. I can wait a random amount of time, and hope its long enough for the user to log in, or I can write my own web browser that will block until the user closes it. Suggestions for option 3 are appreciated....

I know this is not ideal. I hope something like web-login-browser can solve my needs. Also, I know my gtkmozembed based browser has some (crasher..) issues (maybe related to this). I hope to fix them and/or move to a pywebkitgtk based browser, depending on what GNOME favors.

Here is a screencast showing the latest version and its features. You may download the original here. Apologies for the google video quality.

[googlevideo]http://video.google.com/videoplay?docid=-262844619875208739&hl=en[/googlevideo]

Boston Summit

Unfortunately, owing to my proximity to the opposite side of the world as Boston, I will not be attending the summit. I understand that the GNOME online desktop is on the agenda, and will no doubt be discussed at some length. I hope those involved in the discussion try out conduit before the summit, as it would be a shame if the future of the GNOME online desktop was set without due consideration. This is not a criticism, sour grapes, or anything like that. Its just a reflection of what appeared to happen after GUADEC - which I also did not attend (New Zealand = isolated and distant).

Have fun with Conduit. Please report bugs, join the mailing list, and abuse me at will. We are currently moving to GNOME servers, so if my server goes down from all the traffic I promise it will be the last time.