Dec 16, 2007

Jhbuild Anything on Windows - In 12 Steps

Part 1: The Complete Gtk+ Stack from Source

Its been a long week spent watching things compile [0], but I'm happy to report that Jhbuild is now able to build the complete [1] stable Gtk+ stack on windows using the msys/mingw tool chain. That literally means that you can now build you Gtk app, or Gtk itself, for windows, in 12 steps. You still getting all the normal benefits that Jhbuild allows. So, the steps are;

  1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

  2. Untar msysCORE-1.0.11-2007.01.11 to C:\msys

  3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:\ming

  4. Install Python 2.5.1 to default location (C:\Python25)

  5. Install ActivePerl to C:\Perl. You can probably also install a MingW-provided perl package, but the latest version of this was some 5.6 at the time of this writing, while ActivePerl is 5.8.

  6. Install bzr for windows to C:\Bazaar

OK thats all the preparation steps. Launch C:\msys\msys.bat now to start a minimal unix-like shell. Now onto the meaty part;

  1. bzr branch http://www.gnome.org/~jstowers/bzr/jhbuild-win32

  2. cp jhbuild-win32/win32-hacks/* . && mv profile .profile && mv perl.sh /bin/perl

  3. edit pkg-config.sh to reflect the prefix you are going to install (or have installed) pkg-config to

  4. edit jhbuild-win32/configs/gtk-win32.jhbuild so that full unix paths reflect the correct location of pkg-config.sh. Make sure /bin/perl and pkg-config.sh are executable

  5. Close the shell and restart it so that it picks up your new .profile. Check that you can run all needed executables perl --version && python --version && jhbuild --help && ./pkg-config.sh --version

Oh, I seem to be forgetting one more thing.... The following command will build gtk+,atk,cairo,pango,zlib,libpng, and glib from source. No really, that is it [0].

  1. jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

Built Gtk+ On Windows

The Hacks (aka things I don't really understand)

There are a few parts of this setup that still confuse me.

  • /bin/perl is a thin wrapper around perl that quotes args and escapes path slashes. Its necessity is based around the interaction of the msys path escaping and what happens when autotools invokes glib-mkenums. It was based upon what I read here. It also writes a debug log to ~/perllog

  • ~/pkg-config.sh is a wrapper around pkg-config.sh that quotes args. It is as described here but with a full prefix to pkg-config (so that you may use one that is installed in your jhbuild prefix). It also writes a debug log to ~/pkglog

Jhbuild Core Modifications

I am currently hosting a bzr branch with my modifications to Jhbuild. I'm still cleaning some things up, but I hope that this is suitable for committing to SVN in the future. My modifications to Jhbuild are basically;

  • Cleaning up all path handling code. That means using os.path.abspath and os.path.join internally. The msys does shell does some shiny things to convert unix paths to windows paths but it works best if the path it is passed is coherent and not a mix of both systems path separators.

  • Some path and environment variable fudging that checks all paths that get passed out to a shell, and converts them to unix format. Python on windows calls the windows CreateProcess function (that expects a windows executable). Therefor we starts a new msys/bin/sh.exe with the first arg the script we originally wished to start, followed by all the inferior args.

  • Added a binary module type. This is basically like a tarball but instead of executing configure, make, make install you can define arbitary shell commands to execute post download and unpack. This is used to provide glib-bin and gtk-bin metapackages (which get binaries from Tor's gtk win32 site). You can depend on the metapackages if you dont feel like compiling the gtk+/glib stack yourself and just want to build your app. You can also insert these binary modules in the dependency chain and use them to execute arbitrary scripts to fix (read: hack) some aspects of packages.

  • New Gtk+ and Glib modules files for binary installation, and source installation.

  • Use pythons build in functions to download files (instead of wget) and to untar/unzip archives (instead of tar and unzip command line programs). Note that this is not strictly necessary as there are builds of tar/unzip and wget for msy - but its easier from a developer perspective.

Remaining Work, Caveats (aka the full story)

  • libjpeg builds ok but doesn't seem able to generate a shared library. Hence libtiff fails and gtk builds without jpeg support.

  • glib-mkenums seems to insert arbiraty data (a directory listing) at the end of .h files. This results in many compiler warning about data after #endif.

  • Read the gtk-win32.jhbuildrc file.

  • Gtk doesn't set the correct paths to the pixbuf loaders and input modules. You will need to run (from the prefix to which Jhbuild installed everything)

    • $JHBUILDPREFIX/bin/gtk-query-immodules-2.0.exe > $JHBUILDPREFIX/etc/gtk-2.0/gtk.immodules

    • $JHBUILDPREFIX/bin/gdk-pixbuf-query-loaders.exe > $JHBUILDPREFIX/etc/gtk-2.0/gdk-pixbuf.loaders

  • You can change gtk to use the default theme by doing

    • echo 'gtk-theme-name = "MS-Windows"' > $JHBUILDPREFIX/etc/gtk-2.0/gtkrc

  • I have not tested this with SVN modules on windows, only tarball releases. I wasn't prepared to discover the compatibility of the msys provided autotools stack (yet).

  • --disable-docs on which/all modules?

  • Hookup to, and submit build reports to JhAutobuild

  • Python or msys or something buffers all the shell output. This means that there can be long periods (minutes) of nothing getting printed to the shell. If it looks like something has failed to build (no CPU/disk activity) the push 1,3 or 4 to retry, cancel, or drop into a shell respectively (i.e. the normal Jhbuild menu presented upon failure).

Part 2: I Just Want to Build my Application (not Gtk)

It should be noted that it is not necessary to build Gtk (from source) if you just want to build a single application. Using the new binary module type, I have prepared a gtk-binary metamodule that downloads all the needed files from Tor's site, and installs them for you. That sounds confusing, so lets take a randomly chosen example program [2]. Lets install medit, from source, using Jhbuild.

Starting with the example configs/gtk-win32.jhbuild file change the module and modulesets lines to

moduleset = [
    os.path.join(thisdir,'gtk-win32-bin.modules'),
    os.path.join(thisdir,'meddit.modules')
]
modules = ['meddit']

Add a module_autogen line for meddit

#Module specific configure arguments passed in addition to autogenargs
module_autogenargs['meddit'] =  ' --without-xml --disable-help --with-python=no'

And the final step....

jhbuild -f jhbuild-win32/configs/gtk-win32.jhbuild build

Meddit running on windows

Conclusion

I got into this because I wanted to be able to build pygoocanvas on windows, so I could hack on Conduit there. I still have not done that, instead I have;

  • gotten royally distracted from the original goal,

  • hacked on Jhbuild,

  • learned a lot about msys/mingw and autotools on windows,

  • hopefully been able to contribute something useful back to the community that makes it easy for joe normal and jane developer to test their app on windows

  • Oh and its not yet finished (but I have tested it a few times on fresh installs, and it should just work) yet

  • Pretty normal summary for a FOSS hackers week, correct?

I would really like help from some people in testing this further, so go ahead, test it today. Ive made it 12 easy steps for you to follow.......

References (thanks)

[0] A full Gtk+ build from source takes about 6 hours on Windows XP (running in VMware getting both 2.4Ghz Pentium 4 cores)

[1] Ok, not the complete stack. To get around some circular dependency issues I start with binary versions of pkg-config iconv and gettext. With pkg-config present I then build Glib from source. I also use a binary version of poppler because it depends on Gdk while Gtk depends on poppler.

[2] Well not entirely random. It was already advertised as being able to builad on windows. Also it failed during make install, complaining about missing share/mime/packages/freedesktop.org.xml and bin/update-mime-database. I created both as empty files and it was successfully installed.

posted at 10:01 am · 
None jhbuild Nerd Planet GNOME

Dec 13, 2007

Jhbuild and Windows - We Meet Again

The Good News

Following on from my last post I have continued work on getting Jhbuild to run on windows (within msys). The basic problem which I identified in my last post was the struggle between Python (for Windows) expectations about paths, and the hoops that msys and cygwin jump through to allow unix style paths to be used. Basically os.path.join and friends do the correct thing providing the subsequent path is used within python alone. Things get all difficult when that path is then passed out to a configure script, or as an argument to a shell command. With that in mind however, a picture speaks a thousand words....

JHBuild on Windows

  1. It works...

  2. On windows...

  3. Through magic....

The Status

Jhbuild is successfully able to build a test moduleset, and even some real modules such as zlib and iconv on windows using the mingw compiler. Things are looking promising and I will continue to play with this tomorrow. Its a bit of a pain to debug because of the inconsistancies between how subprocess.Popen behaves on windows and on linux. So far my changes can be summarised as follows

  1. Add Makefile.win32 and patch install-check to just call /usr/bin/install

  2. Replace wget/curl with pythons built in urllib.urlretrieve

  3. Replace tar/unzip with pythons built in tarfile and zipfile modules

  4. Misc fixes to get Jhbuild to start including disabling the tray icon etc

  5. Refactor buildscript.execute (the subprocess.Popen) wrapper to clean it up and to special case local script commands from system commands (because of differences in how win32 treats executing executables in the current dir). Best summarised as the following

    def execute_script(self, command, args_str='', cwd=None, extra_env=None): # THIS IS THE MAIN WIN32 HACK if sys.platform.startswith('win') and command[0].startswith('./'): command = ['sh', command[0].replace('./','')] + command[1:]

Jhbuild still works on Linux just as it used to (more or less, I need to port over some more things to the new buildscript.execute() functions). Wget/tar/curl/unzip etc are still used on Linux. In general the changes so far have been self contained and should not break regular operation

The Install Procedure

How can you try this pre-alpha code out?. Follow these steps on windows and check out from my bzr branch (or use the patch against SVN)

  1. Install MinGW combined installer (MinGW-5.1.3) to C:\mingw

  2. Install msysCORE-1.0.11-2007.01.11 to C:\msys

  3. Run C:\msys\postinstall\pi.bat, answer yes when it asks whether mingw is already installed and point it to C:/mingw. Launching C:\msys\msys.bat now provides us with a minimal unix-like shell.

  4. Install Python 2.5.1 to default location (C:\Python25)

  5. Install bzr for windows to the (C:\Bazaar)

  6. Start msys.bat (in C:\msys)

  7. Install Jhbuild

    1. bzr co http://www.gnome.org/~jstowers/bzr/jhbuild-win32 jhbuild-win32

    2. cd jhbuild-win32

    3. make -f Makefile.win32 install

  8. Add python to path. Other path manipulation is done within JHBuild. Add the following line to ~/.profile

    export PATH=$PATH:/c/Python25:/c/Python25/bin

You should now have a minimal msys install with JHBUILD. Test by performing

  • cd ~/bin

  • python jhbuild --help

Finally, you can now install the test modules with

python jhbuild -f jhbuild-win32/test.jhbuildrc build test

Standard disclaimers apply, alpha quality code, help appreciated, yada yada yada. Now back to Conduit hacking....

posted at 1:56 am · 
jhbuild python windows Nerd Planet GNOME

Dec 11, 2007

Jhbuild Adventures on Windows

Background: I'm going home to my parents house for Christmas, and because I don't have a laptop, I thought it would be the perfect opportunity to use their computer to get Conduit working on windows. However I could not find windows builds of (py)goocanvas anywhere. Never minding a challenge I thought it would be a good experiment to see if I could build pygoocanvas for windows via cross-compilation or natively - both using jhbuild.

I like to think that F/OSS works because people keep re-inventing the wheel in new and obscure ways everytime they do something, so what follows is my rambling tale of cross-compiling-gtk-for-windows wheel reinvention.

Cross compiling Gtk+ for Windows using Jhbuild

Starting with the work of Alberto Ruiz and Piotr Gaczkowski I thought I would setup a jhbuild environment to cross compile the Gtk+ stack. After a bit of fiddling I managed to get everything to build fine using mingw32 and jhbuild. The following files work successfully on Ubuntu gutsy and can build the latest stable releases of Gtk+/cairo/glib/libglade.

At this point I moved onto trying to build pygobject. This is where things got difficult. I tried a few different ways to get the cross compilation environment to see python. With some inspiration from this page on compiling glom for windows, I copied across the entire Python25 directory from a windows box and added it to the appropriate LDFLAGS and CFLAGS in jhbuild. This allowed pygobject to pass configure. However it now failed to build;

In file included from /usr/include/python2.5/Python.h:57,
    from pygobject.h:5,
    from pygobject-private.h:9,
    from gobjectmodule.c:27:
    /usr/include/python2.5/pyport.h:246:24: sys/select.h: No such file or directory

I went back to the windows machine and checked to see if sys/select.h was present in mingw. No luck. OK at this point I was stuck. Compiling wasn't looking likely, and I still wasnt even confident that if it did compile it would manange to actually link and run (due to different compiler versions and platforms, etc). I googled for a bit and saw that its now possible to build python extensions using mingw32 and that people had managed to cross compile python using mingw. Hmm, I pondered..... 'lets cross compile Python25 using mingw and then build pygobject using that.' Furthur googling revealed this python bug+patch for cross compiling using mingw. This is going to be easy.......

I added a new section to my modules file that would apply the patch to the Python 2.5.1 before building it. YOU FAIL AGAIN JOHN.

checking for %zd printf() format support... configure: error: cannot run test program while cross compiling

Hmm. I tried several strategies to get around this (including supplying a config.cache file with ac_cv_printf_zd_format=yes) but with no luck. You can see my plea for help on the bug report.......

At this point an entire day of my life had been irrevocably taken from me. I retired to bed. I would not let windows beat me....

Native compilation of Gtk+ for Windows using Jhbuild

OK. At this point I thought I might as well start again, but compile natively on windows using mingw and msys. In hindsight it probbably would have been easier to follow most of the instructions for glom verbaim, but I decided to continue my quest at wheel re-invention.

I read that Alberto was thinking about ways to make Jhbuild work on windows - a good goal. Starting with his patch on that bug report I modified it slightly so that tar, gunzip and friends wouldn't need to be installed. Furthur to that I set about hacking up some patches to allow jhbuild to at least run on windows. A few hours later and I was sure that this was going to take a few more hours :-). I now had a working (ish) minimal easy to set up windows build environment according to these instructions..... or so I thought.

Jhbuild managed to successfully parse my minimal jhbuildrc and module files, download and untar iconv but then it failed. Basically there is considerable work and thought needed to see how this should operate in msys/cygwin. The problem is already explained in part here, but I will summarise.

Python's os.path and friends do a good job at converting paths to the native system format. Unfortunately the msys and cygwin shells also convert paths. For instance msys automagically converts /foo/bar to C:\msys\foo\bar. Basically it means that python converts paths to windows format, but msys likes them in unix format (because its a unix shell environment). This results in files not getting found, and gross composite paths like --prefix=C:\foo\bar/baz/bob getting passed to configure.

So, I had a bit of a look at how this could be solved in Jhbuild. I guess there are a few strategies

  1. Create a whole bunch of utility functions join_path(), prepend_path(), etc. And call these instead of the system ones and only escape to windows format if you are running in windows, and not running in windows just kidding you are really running in cygwin haha.

  2. Remove all os.path and friends calls. Be really naive and hard code all path operations to use unix semantics. Then explictly note that Jhbuild on windows must be run from within a unix shell like msys/bash or cygwin/bash.

  3. Magic. Come up with some sort of paranormal event that would win GNOME $1 million dollars which could then pay a developer to fix all this for me.

Conclusion

Well things didn't work out as well as I hoped. I will probbably go and compile everything without jhbuild and be happy. I did learn that;

  • Jhbuild on windows is a worthwhile cause. It would really help in automatically (jhautobuild) testing things like glib and GIO (at least giving devs a compile yes/no signal)

  • It could make generating the latest tarballs really easy, perhaps even sharing the same modulesets as those maintained by the release team

  • For extra bling it could even automatically create installers by tracking installed files using a strategy similar to checkinstall.

Help, thoughts or advice appreciated. For all the luck I have had the last few days I would not be surprised if I have missed something fundamentally obvious to this whole process. Heck if some fairy was to drop a pygoocanvas installer for windows in my mail box then I would be super happy. Oh and did I mention that im really looking forward to gio!

posted at 7:20 pm · 
conduit jhbuild mingw Nerd Planet GNOME