Static Libraries and Plugins: Global Pain

Years after the X-Plane plugin SDK was ported to operating systems that support Unix .a (archive) static libraries, I have finally come to understand what a mess global symbols can make with incorrect linker settings. The problem is that the incorrect linker settings are almost always the defaults. This blog post will explain what goes wrong with this kind of linker setup and how to fix it.  While this stuff might be obvious to those intimately familiar with Linux and Unix-style linking, it's a bit astonishing to anyone coming from the Windows and pre-OS X Mac world, where the assumptions about linkage are very different.

This post may also thoroughly slander Linux, and if I learn why I'm an idiot and the whole problem can be solved in a much better way, hey, that's great.  I'd much rather find out that there's a better way and I'm wrong than find out that things really are as broken as they seem.

Globally Symbols and Shared Libraries

Unix-style linkers (e.g. ld on both OS X and Linux) support a shared global namespace for symbols exported from shared libraries. Simply put, for any given symbol name, there can be only one 'real' implementation of that symbol, and the first dynamic library (or host app with dynamic linkage, which is basically all host apps these days) to introduce that symbol defines it for every dynamic library.

In other words, if you have five implementations of "void a()" in your dynamic libraries, the first one loaded is used by everyone.  It's a global namespace.

Note that if your symbol is not global, it will not be replaced by an earlier variant.  So if your symbol isn't global, other people having global symbols can't hose you.

The implications of this are clear: you should be very very careful and very very minimal about what gets exported into the global namespace, because of the risk of symbol collision.  I found a bug in an X-Plane plugin because the internal routine sasl_done (in a plugin called sasl) was global and the second instance loaded - sasl_done from libsasl2.dylib had already been loaded by the OS.  The results: a random call into a DLL when the plugin thought it was calling itself!

Unfortunately, the default for GCC is to put everything into the global namespace.  As gcc 3.x fades into history, more code is using -fvisibility=hidden and attributes more aggressively, but the defaults make it really easy to do the wrong thing and dump a whole lot of symbols into the flat namespace.

There is one exception to this global calling: if you use dlsym to resolve a symbol from a specific dynamic library (as returned by dlopen) finds it in that dynamic library, like you would expect.  Therefore if you have a plugin with an "official" entry point (like "PluginStart") you can load multiple plugins into the global namespace and find the "right" start function via dlsym.  (If a plugin called its own start routine, it might jump into the wrong plugin due to globla namespace issues.

What Am I Exporting?

On both OS X and Linux you can use "nm" to view your globally exported symbols:

nm my_plugin.dylib | grep "T "

The stuff with a capital T from nm are code symbols in the global namespace.  If you make a plugin DLL that has a lot of those, your code may not operate if there are other plugins already loaded.

Static Libraries: Not So Static

In the Unix world, the .a (static archive) format is basically a collection of .o files with some header info to optimize when the code is linked.  .o files retain the hidden/visible attribute information that is used by the linker to export symbols out of a dynamic library.

What this means is: under normal operation, the linker may export dynamic library symbols out of a static library you link against. In other words, if you link against libpng.a, you may end up having your DLL export all of the symbols of libpng!  If you aren't the first dynamic library to load, the version of libpng you get may not be the static one you asked for.

This behavior is astonishing at best, but unfortunately it is, again, the default: if the static library didn't specifically set its symbols to hidden, you get "leakage" of static library symbols out of the client shared library.  Unfortunately, from my experience this kind of leakage happens all of the time.  With X-Plane we statically link libcurl, libfreetype and libpng, and all three have their symbols marked globally by default.  These are ./configure based libraries and we don't want to start second-guessing their build decisions.  Unfortunately the code tends to be marked up to build the right API in "shared library" mode but not static mode.

You can see this behavior using nm -m on OS X or objdump -t on Linux.

Working Around Library Leak

Someday we may reach a point where all Unix static libraries keep their symbols "hidden" for dynamic library purposes, but until then there is something a DLL can do to work around this problem: use an explicit list of symbol exports.

Using an explicit list of symbol exports is often considered annoying when an API has a large set of public entry points; usually attributes marking specific functions are preferred.  The advantage of an "official list" at link time is that the linker hides everything except that list, and if any static libraries have globally visible symbols, their absence from the master fixes the problem.

(As an example of how to set this up for gcc on Linux and OS X, see here.)

Addendum: What About Namespacing?

Both Linux and OS X have over time developed ways to cope with the flat namespace problem.

On OS X, a dynamic library can be linked with a two-level namespace.  The symbol is resolved against both the name of the providing dylib and the symbol itself. The result is that symbols come only from the dylibs where you thought they would come from.  If at link time symbol A comes from library X, library X is the only place where it will be provided in the future.  (This is the semantics Windows developers are used to.)

On Linux, library APIs can contain version information; as far as I can tell this works by "decorating" symbols with a named library version (e.g. @@GLIBC_2.0).  When the ABI is changed, symbols cannot conflict between versions, and in theory this may also protect against cross-talk between libraries since the version symbol has some kind of short universal library identifier.  I have found almost no documentation on library versioning; if anyone has a good Linux link I'll add it to this post.

X-Plane's plugin system does not use either of these mechanisms because the plugin system is older than both of them. (Technically on OS X two-level namespaces are older than the plugin system, but the plugin system is older than @loader_path, which is a requirement for strict linking of a dylib in the SDK.)  Thus we are stuck with the global namespace and find ourselves trying to force people to keep their symbols to themselves.

Semaphore Follow-Up: NTPL

A quick follow-up from my previous post on condition variables, etc. With NTPL (the pthreads implementation on Linux) a lot of the original issues I was trying to cope with don't exist. Some things NTPL does:

• pthread mutexes are spin-sleep locks, so they can be used as short-term critical sections without too much trouble. Given a moderately contested but shortly held lock, this is a win.
• sem_t semaphores have an atomic counter to avoid system calls in the uncontested case. When inited privately (sem_init) they appear to be lean and mean.
• All synchronization is done around futexes, ensuring that uncontested cases can be manged with atomic operations. (The OS X pthreads library at least uses spin locks around user space book-keeping for the uncontested case, but I think the futex code path is faster.)

There is one case where using a condition variable really would be superior to a semaphore on Linux: if you really want a condition variable (and aren't just using it to build a semaphore). In particular, the futex system call that helps NTPL sleep threads as needed has special operations to move a thread from one queue to another while asleep. This fixes the thundering herd problem when every thread on a condition is woken up at once. This isn't something I use, but if you need it, NTPL makes it fast.

OpenAL on Three Platforms

OpenAL is a cross-platform 3-d sound API. It is not my favorite sound API, but it is cross-platform, which is pretty handy if you work on a cross-platform game. Keeping client code cross-platform with OpenAL is trivial (as long as you don't depend on particular vendor extensions) but actually getting an OpenAL runtime is a little bit trickier. This post describes a way to get OpenAL on three platforms without too much user hassle.

OS X

On OS X things are pretty easy: OpenAL ships with OS X as a framework dating back to, well, long enough that you don't have to worry about it. Link against the framework and go home happy.

Well, maybe you do have to worry. OpenAL started shipping with OS X with OS X 10.4. If you need to support 10.3.9, weak link against the framework and check an OpenAL symbol against null to handle a missing installation at run-time.

Linux

On Linux, OpenAL is typically in a shared object, e.g. libopenal.so. The problem is that the major version number of the .so changed from 0 to 1 when the reference implementation was replaced with OpenAL Soft. Since we were linking against libopenal.so.0, this broke X-Plane.

My first approach was to yell and complain in the general direction of the Linux community, but this didn't actually fix the problem. Instead, I wrote a wrapper around OpenAL, so that we could resolve function pointers from libraries opened with dlopen. Then I set X-Plane up to first try libopenal.so.1 and then libopenal.so.0.

(Why did the .so number change? The argument is that since the .0 version contained undocumented functions, technically the removal of those undocumented functions represents an ABI breakage. I don't quite buy this, as it punishes apps that follow the OpenAL spec to protect apps that didn't play by the rules. But again, my complaining has not changed the .so naming conventions.)

Windows

The Windows world is a bit more complicated because there are two separate things to consider:

• The implementation of OpenAL (e.g. who provided openal32.dll).
• The renderer (e.g. which code is actually producing audio).

Basically Creative Labs wanted to create an ecosystem like OpenGL where users would have drivers installed on their machine matching specialized hardware. So the Create implementation of openal32.dll searches for one or more renderers and can pass through OpenAL commands to any one of them. The standard OpenAL "redistributable" that Creative provides contains both this wrapper and a software-only renderer on top of DirectSound (the "generic software" renderer).

OpenAL Soft makes things interesting: you can install OpenAL soft into your system folder and it becomes yet another renderer. Or you can use it instead of any of the Creative components and then you get OpenAL soft and no possible extra renderers.

Now there's one other issue: what if there is no OpenAL runtime on the user's machine? DirectSound is pretty widely available, but OpenAL is not.

Here we take advantage of our DLL wrapper from the Linux case above: we package OpenAL Soft with the app as a DLL (it's LGPL). We first try to open openal32.dll in the system folder (the official way), but if that fails, we fall back and open our own copy of LibOpenAL Soft. Now we have sound everywhere and hardware acceleration if it's available.

One final note: in order to safely support third party windows renderers like Rapture3D, we need to give the user a way to pick among multiple devices, rather than always opening the default device (which is standard practice on Mac/Linux). This can be done with some settings UI or some heuristic to pick renderers.

OpenAL on Linux, Part 27

This bug has effected X-Plane 8 and 9; we were able to recut 9 to work around it, but X-Plane 8 is a closed product. Here's the short story:

• A while ago intrepid developers replaced the implementation of libopenal on Linux with a new complete rewrite.
• When they did so, they raised the major version number.

Huh??? This caused naive application developers like me to say things like "what the hell are you guys doing? The whole point of dynamic linking is that you can replace implementations without breaking my app. So why did you break my app?"

The change in major version breaks the link to X-Plane, and would be appropriate if the library wasn't compatible.

Yesterday someone finally posted a list of dropped ABI symbols in the new OpenAL implementation. They are all extension symbols except for alBufferAppendData. So I can't deny that symbols are dropped and that is an ABI breakage. The question is: should the soname be revised?

Extensions

Most of the symbols missing are _LOKI. For those not familiar with OpenGL extensions (from which the OpenAL extension concept is stolen^H^H^H^H^H^Hborrowed) the idea is this: an app initializes the library, queries some kind of string to see what additional non-core features the library supports, and then resolves function pointers at run-time, using function pointers only once the extension string is present.

Therefore it's really important that the major version of the shared object not change when an extension is removed; the extension is not part of the ABI, applications should not (and cannot) depend on it being present at link-time, and an extension may not function without specialized hardware.

alBufferAppendData

There is one mystery symbol: alBufferAppendData, which is present without a decoration. From what I can tell from the annotated OpenAL 1.0 specification, "append-data" was a proposed streaming scheme that was eventually moved to an extension when it was dropped from the core. It's not in the 1.0 spec and it's not in the 1.1 spec.

So this strikes me as a bug in the implementation of the original library: it exports a symbol that shouldn't be there. Does it make sense to raise the major version of the .so because the symbol has been dropped? I don't think so, but I can see how you could argue it both ways.

The argument for dropping it is this: if the major version is changed, then the old and new OpenAL implementations can live side by side, and all applications are happy. Since alBufferAppendData is not trivial functionality, this would be better than expecting the new implementation to support alBufferAppendData for historic reasons.

But this is not at all what is happening; instead distributions are purging libopenal.so.0 (the old implementation) when they bring in the new one, and then asking applications to recompile themselves.

In other words, because some number of applications may be using a function that is not in any OpenAL specification but is in the old implementation, they have renamed the shared library, forcing everyone to recompile. In other words, they have replaced the convenience of having some games be broken with the convenience of having all games be broken.

(In X-Plane we work around this by simply dlopening either libopenal.so.0 or libopenal.so.1, whichever one we can find. Since both implement the core spec symbols, this works fine.)

glXGetProcAddressARB Syntax

I was slightly astounded to read that glxGetProcAddressARB is declared like this:

void (*glXGetProcAddressARB(const GLubyte *procName))();

Wha? Well, fortunately when you read the spec you'll note that they're just being clever...that's very strange C for

typedef void (*GLfunction)();
extern GLfunction glXGetProcAddressARB(const GLubyte *procName);

In other words, unlike all other operating systems, which define the returned type of a proc query as a void *, GLX typedefs it as a pointer to a function taking no arguments and returning nothing.

Why this is useful is beyond me, but if you are like us and call one of wgl, AGL, or GLX, you may have to cast the return of glXGetProcAddressARB to (void *) to make it play nice with the other operating systems.

Bonjour Monsieur Linux

I was poking around with mDNSResponder the other day. mDNSResponder is the library code and daemon implementation for mDNS (multicast DNS, also known as "zero-conf") and DNS-SD (DNS service discovery), sometimes known as "Bonjour". I have no idea which of these terms is a trademark, a technology, an implementation, or a slur. I'll go with mDNS and DNS-SD.

If you use a Mac, you've seen mDNS in that your computers sometimes grow weird names like macbookpro.local. Surprisingly, these names work as well as DNS names like www.google.com. You can ping them or gethostbyname on m.

Those names come from mDNS - that is, they are part of a distributed naming domain "local" that is jointly managed by all mDNS-enabled computers on the local net. These domain names are only transparent to the unix shell because naming services on OS X knows about them.

If you have shared your music with iTunes, you're using DNS-SD - that is, iTunes publishes a "service" (music sharing) using DNS-SD and then searches for other music databases the same way.

The DNS-SD C API and code that run below it are all open source, so it is possible to create cross-platform service discovery code. If you try to do this, you will have to pick one of two models:

• Run the DNS-SD daemon (if it doesn't already run) and use the DNS-SD API that uses IPC to talk to the daemon. This configuration is what Apple ships for OS X - the efficiency comes from having only one copy of cached DNS-SD data.
• Compile the DNS-SD network implementation (which is normally in the daemon) into your app directly; Apple recommends this only for embedded systems.

Here are my two tips if you want to try to make this work - that is, this is what kept me up until 4 AM. (Hint: 4 am = stupid...I really solved the bugs the next day after having had a cup of coffee and some sleep.)

1. You can't use gethostbyname to resolve a host into an IP address, despite what Apple's docs say - this only works on platforms like OS X that support mDNS host names "natively".

   The normal flow of operations is to browse for services, "resolve" them to hosts. On a non-Mac platform you need to do a third mDNS operation: use mDNS to resolve the host to an IP address using something like DNSServiceQueryRecord (with a record type of kDNSServiceType_A to convert the host to an address).

2. If you are compiling the mDNS library into your app, you need to initialize it with mDNS_Init_AdvertiseLocalAddresses. This flag will tell mDNS to publish a mapping of your dynamic host name (my_machine.local or whatever) to your actual IP address for each interface.

   If you do not publish, other machines will be able to find and resolve your service, but will fail to then convert your host to an IP. You will be able to get their IPs though. This bug can be masked on OS X, which always publishes itself since you use DNS-SD in daemon form.

   
   

Threads: Who Am I?

I was curious as to what the cost is of retrieving "thread local storage" (per-thread variables). I am looking to move some variables from globals to per-thread, and I want to know how expensive access will be.

I went digging into the Linux thread library source. First: pthread_getspecific (which pulls out a specific variable) works by:

1. Finding the thread's "identity" as a pointer to a basic control block.
2. Thread-local storage is simply an array at the end of the control block.
3. The specific retrieval is just an array access.

So specific key retrieval isn't going to any worse than getting the thread itself. How does Linux do that?

In a way that I thought was quite clever: stacks are stored on a fixed granularity/page size. The calling thread has a stack pointer within its own stack. The thread control block is at the very beginning of the stack.

So all the calling code has to do is take a stack-local variable and round its address down to the thread stack granularity and there's the control block. That's pretty quick! No system calls needed.

Edit: here's an even cheaper way.

So Where Is That Fast Path?

Amongst the many rants I've read about the new OpenGL 3.0 spec, is the claim that the spec needs to be cleaned out and rebuilt so that people can "find the fast path".

What application developers are getting at with this is that OpenGL is a rich API, and not all cards do everything at top speed.  There is usually one fastest way to talk to OpenGL to get maximum throughput.

The problem is: this is ludicrous.  Case in point, the GeForce 8800 - in my Mac Pro, running OS X 10.5.4 and Ubuntu 8.  So what is the fast path?

If I draw my terrain using stream VBOs for geometry and indices that are not in a VBO, I get 105 fps on Linux.  If I then put the indices into a stream VBO, I get 135 fps.  The fast path!

Well, not quite.  The index-without-VBO case runs at 73 fps on OS X, but once those indices go into a VBO, I crash down to 25 fps.  Wrong fast path.

Simply put, you can't spec the fast path, so the spec doesn't matter.  You find the fast path by writing a lot of code and trying it on a lot of hardware.  I can't see there ever being another way, given how many different cards and drivers are out there.

Sound and the 24" iMac

I finally found info on how to make it happen here.

Here's the short version:

• I used Ubuntu 7.10, which has Alsa 1.14, which is a little different.
• Make sure

  options snd-hda-intel model=bbp3

  is somewhere in your modprobe.d files, like in alsa-base.
• Turn on a lot of goo in the mixer.
• Surprisingly the built-in Mic is the digital input!

I now know that XSquawkBox Linux sound does work (but only if X-pane itself is run with --no_sound to allow OSS to open).

Triple-Boot Mac Part 2: MBRs Are Still Hell

A while ago I blogged about the setup process to get OS X 10.4, Windows XP, and Ubuntu 6.06 onto my MacBook Pro. I tried the same trick today on a Mac Pro running OS X 10.5, Windows Vista, and Ubuntu 8.04. To further complicate things, I created a true swap partition.

Now in the old way, Windows had to be last on the disk...fortunately Vista is a little bit more savvy about partitioning (although it stil uses MBR and not GPT partitions). So the order of operations was:

1. Install rEFIt onto the Mac.
2. Use diskutil to partition the main drive, cutting off three new Fat-32 partitions, size 30G, 30G, and 10G, which will become Vista, Linux and a swap partition.
3. Install ext2FSX.
4. Reformat the second Fat-32 partition as ext2 format. (Once ext2FSX is installed, Disk Utility will let you do this.)
   
5. Install Ubuntu, using manual partitioning, and using the last Fat32 partition as a swap file.
6. On reboot, Use rEFITs's partition tool to fix the MBR partition table, which the Ubuntu installer will trash.
   
7. Install Vista onto the middle partition, which hasn't been used. Vista will want you to reformat the Fat-32 partition as NTFS, which is fine. One tricky part: since rEFIt is installed, when Vista reboots you have to pick the volume that the Vista installer thinks it is rebooting to. As far as I can tell, this is always the volume Vista is being installed on.
8. Reboot with the Ubuntu Live CD, and run a grub shell and use "setup" to put a boot sector back on the Linux partition. I don't know why it was missing - possibly an error in setting up Ubuntu the first time.

Some notes on the sketchier steps:

• Why ext2 on Mac? For some reason the 10.5 version of diskutil won't format Linux volume drives. The problem is that when the Ubuntu installer reformats the FAT-32 partition as ext2, OS X doesn't know about it. The next time you boot OS X, your Linux partition is mounted as a FAT-32 volume, which completely trashes it. It took me quite a few tries to figure this out, and I got myself into a state where I had to totally rebuild the startup drive from a full format and reinstall OS X itself. So converting the format to ext2 is a hack to keep the partition from getting smashed by OS X.
• rEFIt is great - given a valid GPT partition table and a screwed up MBR table, it'll usually do what you want. It will prioritize the first four partitions, hence the Linux swap is the last partition, not Vista. (BTW Vista's partition tool sees the swap partition as "unused space" - so do not repartition in Vista.)
• The goo to rebuild grub is more or less:

sudo grub
root (hd0,3)
setup (hd0,3)

Grub will complain a bit, but it will fix the MBR. Note that hd0,3 means the 4th partition on the first hard drive, so your mileage may vary. Tab completion in grub is a great way to scan the local files sytems, e.g. type root(hd to list drives and root(hd0, to list partitions.

Creating OpenGL Objects in a Second Thread - Mac, Linux, Windows

This blog entry explains how to set up OpenGL to create new OpenGL "objects" (vertex buffer objects, display lists, and textures) from a second thread. The benefits of this technique are:

• It moves CPU-intensive driver operations to a second core even if the driver is not multi-threaded internally*. CPU-based on-the-fly texture compression would be an example where you can get a win.
  
• It allows you to move the entire process of building up and loading a scene-graph to a second thread, regardless of the use of OpenGL loading commands (like building of VBOs).
• Because there is no use of the main thread, the main rendering loop can continue to run at full speed (assuming the user has a dual-core machine).

The basic strategy is to build an OpenGL context for each worker thread that will do loading such that the contexts share objects (display lists, etc.) with the main rendering thread's OpenGL context.

(This information comes from in-field tests; if you work on the GL implementation and spot something wrong, or if you find in deployment that this doesn't work, please let me know. Also a warning: the latest version of this code hasn't been in-field very wrong - in the past edge cases have appeared that have caused us to redesign threaded loading more times than I'd like to admit.)

A warning on thread safety: OpenGL's asynchronous nature (sequential deferred execution within a context) leads to some tricky threading bugs that you wouldn't normally see - see this blog post for examples of what can go wrong.

Terminology

While the structures differ between AGL/CGL (Macintosh), WGL (Windows) and GLX (Linux/X11) there are some common features in the way the window manager and the GL interface.

• "Pixel Format." All three operating systems define a pixel format as an opaque description of a series of properties for a rendering context. In my tests, I use the same pixel format for all contexts.
• "Drawable." Basically any window, full screen destination, or off-screen hardware accelerated buffer is a "drawable", and it defines where the result of GL commands to a context end up.

Macintosh

The Macintosh is probably the easiest OS to set this up on, because you don't need to use a drawable for the loader contexts.

1. Create new contexts for each worker thread, using the main rendering context as a source for shared objects.
2. In the start of the worker thread, set the worker thread's context to be current.
   

Windows

On Windows, you'll need to do all of the creation work inside the worker thread:

1. Get a DC from the main window. This should create a new DC that your worker thread can use. (There is apparently nothing wrong with requesting multiple DCs from a single window.)
2. Create a new context for the worker thread.
3. Set the new context to share objects with your main context.
4. Set the worker context to current using the DC you acquired.

Note that your main thread will have to temporarily drop the main context as its current context - otherwise you'll get a "resource busy" error when you try to share contexts - the main thread effectively has that context locked when in use, so you'll want to set the main thread's current context to null, then set it back when done.

Also note that you can't use window-owned or class-owned DCs in this design because either of those will cause you to get back one DC every time you call GetDC and you really need a unique DC for each thread. Be sure to release the DC on the same thread you use to acquire it!

Linux

Linux is similar to the Macintosh, except that the worker threads need a drawable. For this I used p-buffers.

1. The main thread creates a new context, shared with the main context.
2. The worker thread makes a new small pbuffer for itself.
3. The worker thread sets the worker context as current, using the pbuffer as its drawable.

I tried using the main window as the drawable for all threads, but this causes some kind of locking or thrashing between threads and hurts performance of the main renderer. Extra hidden windows were unreliable for reasons I was not able to determine.

Alternatives

The alternative to this design is to use an RPC-like mechanism to execute all OpenGL activity in the main thread, e.g. once per frame pick up any OpenGL work to be done from worker threads, run it, then let them continue. This has some advantages and disadvantages:

• On the pro side, it doesn't require multiple contexts - you may find that some drivers do better executing everything on one context.
  
• OpenGL work is done at the same time in every frame, possibly resulting in more controlled operation, particularly on single-threaded machines.
• On the con side, loader code that needs to create OpenGL objects before proceeding (to keep memory footprint down, for example) may operate slowly since it has limited access to the GL.
  

* This assumes that the driver is not multi-threaded internally, but is capable of performing these CPU-based operations without locking out the render thread via a mutex...a big assumption that I cannot validate or invalidate at this point.

popen, Windows, and quotes

Turns out that this won't work very well on Windows:

popen("\"C:\\some dir\\some_file.exe\" \"input file\"");

But this will:

popen("\"\"C:\\some dir\\some_file.exe\" \"input file\"\"");

Those coming from Unix-land would think the second example has one too many quotes; here's more info.

Triple-Boot Mac (and MBR hell)

It only took about 8 hours of total time, but my MacBook Pro can now triple-boot into OS X 10.4.7, Windows XP Home SP2, and Ubuntu Linux (6.06 I think).

The install is actually not that bad once you've been through it once...the process is basically:

• Use diskutil to dynamically build two new partitions. The Mac comes with one "EFI" partition for boot control and one HFS+ partition. MBR gives you four max, so you build a Linux one followed by a FAT 32 one.
• Install Windows XP. Tedious and annoying but not complicated. Windows takes longer to boot from CD than an entire OS X install from scratch.
• Install Linux. This is the slightly dangerous part, as I found out.

So two things took a long time:

First, for some reason my brand new, genuine Windows XP CD-ROM doesn't work very well. Setup had at least two hangs, a failure to init a disk, a BSOD, and an assertion failure. Maybe all of those holograms on the disk play havoc with the optical drive? So the biggest single item was a 5 hour Windows install that mostly involved trying to boot setup over and over.

Windows XP asks questions during the install, so you have to sit there and watch it. Furthermore, if you are using rEFIt like I am, then you'll have to manually select Windows each time the installer reboots itself. The installer normally expects to reincarnate itself, so if your machine mysteriously reboots without warning be sure to keep reselecting to boot from the Windows install CD-ROM.

The other problem was that, almost certainly due to my own sleepiness this morning, I managed to splat LILO onto the boot block of the Windows partitiont. Ooops. Most tutorials online state that the only fix for this is a complete reinstall of both Windows and Linux, but fortunately I discovered this is not true.

It turns out that you can very gently run fixmbr on your Windows partition without disturbing triple-boot goodness. It's pretty much that simple...just launch the windows setup CD, wait 30 minutes for it to load every driver ever written, launch the recovery tool and use fixmbr on the appropriate drive. Linux and Mac continue to boot via rEFIt.

Now the only issue is: Windows can't understand Mac or Linux partitions, Linux can't understand Mac partitions, and while the Mac will figure anything out, it's not automounting.