Static Libraries and Plugins: Global Pain

Years after the X-Plane plugin SDK was ported to operating systems that support Unix .a (archive) static libraries, I have finally come to understand what a mess global symbols can make with incorrect linker settings. The problem is that the incorrect linker settings are almost always the defaults. This blog post will explain what goes wrong with this kind of linker setup and how to fix it.  While this stuff might be obvious to those intimately familiar with Linux and Unix-style linking, it's a bit astonishing to anyone coming from the Windows and pre-OS X Mac world, where the assumptions about linkage are very different.

This post may also thoroughly slander Linux, and if I learn why I'm an idiot and the whole problem can be solved in a much better way, hey, that's great.  I'd much rather find out that there's a better way and I'm wrong than find out that things really are as broken as they seem.

Globally Symbols and Shared Libraries

Unix-style linkers (e.g. ld on both OS X and Linux) support a shared global namespace for symbols exported from shared libraries. Simply put, for any given symbol name, there can be only one 'real' implementation of that symbol, and the first dynamic library (or host app with dynamic linkage, which is basically all host apps these days) to introduce that symbol defines it for every dynamic library.

In other words, if you have five implementations of "void a()" in your dynamic libraries, the first one loaded is used by everyone.  It's a global namespace.

Note that if your symbol is not global, it will not be replaced by an earlier variant.  So if your symbol isn't global, other people having global symbols can't hose you.

The implications of this are clear: you should be very very careful and very very minimal about what gets exported into the global namespace, because of the risk of symbol collision.  I found a bug in an X-Plane plugin because the internal routine sasl_done (in a plugin called sasl) was global and the second instance loaded - sasl_done from libsasl2.dylib had already been loaded by the OS.  The results: a random call into a DLL when the plugin thought it was calling itself!

Unfortunately, the default for GCC is to put everything into the global namespace.  As gcc 3.x fades into history, more code is using -fvisibility=hidden and attributes more aggressively, but the defaults make it really easy to do the wrong thing and dump a whole lot of symbols into the flat namespace.

There is one exception to this global calling: if you use dlsym to resolve a symbol from a specific dynamic library (as returned by dlopen) finds it in that dynamic library, like you would expect.  Therefore if you have a plugin with an "official" entry point (like "PluginStart") you can load multiple plugins into the global namespace and find the "right" start function via dlsym.  (If a plugin called its own start routine, it might jump into the wrong plugin due to globla namespace issues.

What Am I Exporting?

On both OS X and Linux you can use "nm" to view your globally exported symbols:

nm my_plugin.dylib | grep "T "

The stuff with a capital T from nm are code symbols in the global namespace.  If you make a plugin DLL that has a lot of those, your code may not operate if there are other plugins already loaded.

Static Libraries: Not So Static

In the Unix world, the .a (static archive) format is basically a collection of .o files with some header info to optimize when the code is linked.  .o files retain the hidden/visible attribute information that is used by the linker to export symbols out of a dynamic library.

What this means is: under normal operation, the linker may export dynamic library symbols out of a static library you link against. In other words, if you link against libpng.a, you may end up having your DLL export all of the symbols of libpng!  If you aren't the first dynamic library to load, the version of libpng you get may not be the static one you asked for.

This behavior is astonishing at best, but unfortunately it is, again, the default: if the static library didn't specifically set its symbols to hidden, you get "leakage" of static library symbols out of the client shared library.  Unfortunately, from my experience this kind of leakage happens all of the time.  With X-Plane we statically link libcurl, libfreetype and libpng, and all three have their symbols marked globally by default.  These are ./configure based libraries and we don't want to start second-guessing their build decisions.  Unfortunately the code tends to be marked up to build the right API in "shared library" mode but not static mode.

You can see this behavior using nm -m on OS X or objdump -t on Linux.

Working Around Library Leak

Someday we may reach a point where all Unix static libraries keep their symbols "hidden" for dynamic library purposes, but until then there is something a DLL can do to work around this problem: use an explicit list of symbol exports.

Using an explicit list of symbol exports is often considered annoying when an API has a large set of public entry points; usually attributes marking specific functions are preferred.  The advantage of an "official list" at link time is that the linker hides everything except that list, and if any static libraries have globally visible symbols, their absence from the master fixes the problem.

(As an example of how to set this up for gcc on Linux and OS X, see here.)

Addendum: What About Namespacing?

Both Linux and OS X have over time developed ways to cope with the flat namespace problem.

On OS X, a dynamic library can be linked with a two-level namespace.  The symbol is resolved against both the name of the providing dylib and the symbol itself. The result is that symbols come only from the dylibs where you thought they would come from.  If at link time symbol A comes from library X, library X is the only place where it will be provided in the future.  (This is the semantics Windows developers are used to.)

On Linux, library APIs can contain version information; as far as I can tell this works by "decorating" symbols with a named library version (e.g. @@GLIBC_2.0).  When the ABI is changed, symbols cannot conflict between versions, and in theory this may also protect against cross-talk between libraries since the version symbol has some kind of short universal library identifier.  I have found almost no documentation on library versioning; if anyone has a good Linux link I'll add it to this post.

X-Plane's plugin system does not use either of these mechanisms because the plugin system is older than both of them. (Technically on OS X two-level namespaces are older than the plugin system, but the plugin system is older than @loader_path, which is a requirement for strict linking of a dylib in the SDK.)  Thus we are stuck with the global namespace and find ourselves trying to force people to keep their symbols to themselves.

SvnX on OS X 10.6? You Need a Key Pair

A few members of our art team use a mix of the command line and SvnX to move art asset packs around via SVN.

One minor hitch: SvnX can't log into a server that uses svn+ssh as its access method if ssh requires a manually typed password.

The work-around is to establish a private/public key pair for ssh. Once you do that, keychain will offer to store the password, and SvnX can function normally.

In theory sshkeychain should let the key chain remember plain passwords, but I couldn't get this to work on 10.6.

The keypair can be established as follows:

cd ~/.ssh
ssh-keygen -t rsa
(type desired password, accept default file name)
scp id_rsa.pub you@server.com:/home/you/.ssh/auhorized_keys
(where "you" is your unix login name. authorized_keys may need a different name for different servers.)

Gamma and Lighting Part 1: Color Sync

The topic of color spaces, gamma, and color correction gets complex and muddled quickly enough that I had to delete my post and start over. Fortunately a number of folks on the Mac OpenGL list set me straight. This first post will discuss how to control color when working with textures; the second one will address how to light those textures in 3-d.

This FAQ explains Gamma infinitely better than I possibly can. I suggest reading it about eight times.

24-bit Color Spaces Are Pretty Much Never Linear

A color space defines the meaning of RGB colors (that is, triplets of 8-bit or more codes). How red is red? That's what your color space tells you. And the first thing I must point out is: no color space you'd ever use on an 8-bit-per-channel (24-bit color) display is ever going to have a linear mapping between RGB values and luminance. Why not? This Wikipedia article on sRGB puts it best:

This nonlinear conversion means that sRGB is a reasonably efficient use of the values in an integer-based image file to display human-discernible light levels.

Here's what's going on:

• Our perception of color is non-linear. It's roughly logarithmic. This means we're more discerning of light level changes at low light levels. That's good - we can see in the dark, sort of, or at least this improves our dynamic range of light perception.
• 256 levels of gray isn't a lot, so it's important that the levels be spread out evenly in terms of what humans perceive. Otherwise we'd run out of distinct gray levels and see banding.
• Therefore, to make 24-bit images look good, pretty much every color space including sRGB is radically different from linear luminance levels.

When we discuss images with different gamma below, let's just keep this in the back of our mind: pretty much any sane color space is going to have some gamma if it will be used for 24-bit images, and that's good because it lets us avoid banding and other artifacts. It will also become important when we get to lighting.

Part of the confusion over gamma correction is that CRTs have non-linear response by nature of their electronics. People assume that this is bad and that gamma correction curves make the RGB->luminance response linear. In fact, that response is actually useful, and that's why the gamma correction curves on a computer typically undo some of the non-linear response. For example, Macs used to bring the gamma curve down to 1.8 (from 2.5 for a CRT) but not all the way down to 1.0. As mentioned above, if we really had linear response between our RGB colors and luminance, we would need more than 8 bits per channel.

Consistent Color

If we are going to develop a simulator that renders using OpenGL onto a PC or Mac, we're going to work in a color space, probably with 24-bit color most of the time. (We might use higher bit depths in advanced rendering modes, but we can't afford to store our entire texture set in a higher res. Heck, it's expensive to even turn off texture compression!)

So the important thing is that color is maintained consistently through the authoring pipeline to the end user. In practice that means:

1. Artists need to have their monitors correctly calibrated. If the artist's monitor isn't showing enough red, the artist paints everything to be too red, and the final results look Mars-tacular. So the artist needs good "monitoring."

2. The color space needs to be tracked through the entire production process. Whether this happens by simply declaring a global color space for the entire pipeline or by tagging assets, we need to make sure that red keeps meaning the same red, or that if we change it, we record that we've changed it and correctly adjust the image.

   When working with PNG, we try to record the gamma curve on the PNG file. We encourage our artists to work entirely in sRGB.

3. The graphics engine needs to not trash the color on read-in. Similar to the pipeline, we need to either not mess up the colors, or track any change in color space. In X-Plane's space, X-Plane will try to run in approximately the color space of the end user's machine, so X-Plane's texture loader will change the color space of textures at load time.

4. The graphics engine needs to cope with a user whose display is atypical (or not). Finally, if the user's display is not in the same color space as the production pipeline, the engine needs to adjust its output. (Or not adjust its output and expect the user to fix it locally.) In X-plane's case we do this by using the user's machine's gamma curve and correcting images at load time.

   This is not the best solution for color accuracy; it would be better to keep the images in their saved color space and convert color in-shader (where we have floating point instead of 8-bits per channel). Historically X-Plane does the cheap conversion to save fill rate. We may some day change this.

(Note: pre-converting color is particularly a poor idea for DXT-compressed textures, because you can adjust the pair of 565 key colors in each block but you can't adjust the weightings, which are clamped at fixed mix-points based on the DXT specification. So the mid-tone colors for each block will be distorted by trying to color correct a compressed image.)

If all of these steps are taken, the end user should see textures just as the art team authored them. Before we had step four in X-Plane, PC users used to report that X-Plane was "too dark" because their monitor's color response wasn't the same as our art team's (who were using Macs). Now they report that X-Plane is too dark because it's dark. :-)

In my next post I'll discuss the relationship between non-linear color spaces (that work well in 24-bit color) and 3-d lighting models.

OpenAL on Three Platforms

OpenAL is a cross-platform 3-d sound API. It is not my favorite sound API, but it is cross-platform, which is pretty handy if you work on a cross-platform game. Keeping client code cross-platform with OpenAL is trivial (as long as you don't depend on particular vendor extensions) but actually getting an OpenAL runtime is a little bit trickier. This post describes a way to get OpenAL on three platforms without too much user hassle.

OS X

On OS X things are pretty easy: OpenAL ships with OS X as a framework dating back to, well, long enough that you don't have to worry about it. Link against the framework and go home happy.

Well, maybe you do have to worry. OpenAL started shipping with OS X with OS X 10.4. If you need to support 10.3.9, weak link against the framework and check an OpenAL symbol against null to handle a missing installation at run-time.

Linux

On Linux, OpenAL is typically in a shared object, e.g. libopenal.so. The problem is that the major version number of the .so changed from 0 to 1 when the reference implementation was replaced with OpenAL Soft. Since we were linking against libopenal.so.0, this broke X-Plane.

My first approach was to yell and complain in the general direction of the Linux community, but this didn't actually fix the problem. Instead, I wrote a wrapper around OpenAL, so that we could resolve function pointers from libraries opened with dlopen. Then I set X-Plane up to first try libopenal.so.1 and then libopenal.so.0.

(Why did the .so number change? The argument is that since the .0 version contained undocumented functions, technically the removal of those undocumented functions represents an ABI breakage. I don't quite buy this, as it punishes apps that follow the OpenAL spec to protect apps that didn't play by the rules. But again, my complaining has not changed the .so naming conventions.)

Windows

The Windows world is a bit more complicated because there are two separate things to consider:

• The implementation of OpenAL (e.g. who provided openal32.dll).
• The renderer (e.g. which code is actually producing audio).

Basically Creative Labs wanted to create an ecosystem like OpenGL where users would have drivers installed on their machine matching specialized hardware. So the Create implementation of openal32.dll searches for one or more renderers and can pass through OpenAL commands to any one of them. The standard OpenAL "redistributable" that Creative provides contains both this wrapper and a software-only renderer on top of DirectSound (the "generic software" renderer).

OpenAL Soft makes things interesting: you can install OpenAL soft into your system folder and it becomes yet another renderer. Or you can use it instead of any of the Creative components and then you get OpenAL soft and no possible extra renderers.

Now there's one other issue: what if there is no OpenAL runtime on the user's machine? DirectSound is pretty widely available, but OpenAL is not.

Here we take advantage of our DLL wrapper from the Linux case above: we package OpenAL Soft with the app as a DLL (it's LGPL). We first try to open openal32.dll in the system folder (the official way), but if that fails, we fall back and open our own copy of LibOpenAL Soft. Now we have sound everywhere and hardware acceleration if it's available.

One final note: in order to safely support third party windows renderers like Rapture3D, we need to give the user a way to pick among multiple devices, rather than always opening the default device (which is standard practice on Mac/Linux). This can be done with some settings UI or some heuristic to pick renderers.

Restarting the OS X Window Server for Fun and Profit

Well, it's not very profitable. Hell, it's not even that fun. But let's just say, hypothetically, that you were working on a flight simulator with an OpenGL rendering engine. And let's just say, to make this interesting, that if you crank up all of the new rendering engine options, sometimes it causes the OpenGL stack to completely lose its meatballs, and the resulting carnage renders the entire computer unusable.

(If you are having trouble imagining this, close your eyes and visualize a desktop where nothing but the mouse moves, but as you drag what were your windows, small pieces of your scene graph flicker in and out of what used to be your open windows, as if you were just showing random parts of video memory. Okay, maybe it is a little bit fun.)

Here's what you need to get your life back:

1. Have remote ssh enabled in the sharing control panel. ssh into your machine. Odds are, the remote shell is perfectly happy, even if the desktop looks like you hired Picasso as your art lead and he was extra high that day.
2. Kill -9 pid will bring back the desktop some of the time. That is, sometimes just killing off your app is enough to get your desktop back. Typically this is a win in the case where the driver is constantly resetting and you just can't use the UI because the reset cycle is slow.
3. If that doesn't work, this will kill off the entire window manager (including, um, everything...the Finder, your app, X-Code, icanhazcheesburger): sudo killall -HUP WindowServer

It beats a full reboot (by some marginal amount).

Bonjour Monsieur Linux

I was poking around with mDNSResponder the other day. mDNSResponder is the library code and daemon implementation for mDNS (multicast DNS, also known as "zero-conf") and DNS-SD (DNS service discovery), sometimes known as "Bonjour". I have no idea which of these terms is a trademark, a technology, an implementation, or a slur. I'll go with mDNS and DNS-SD.

If you use a Mac, you've seen mDNS in that your computers sometimes grow weird names like macbookpro.local. Surprisingly, these names work as well as DNS names like www.google.com. You can ping them or gethostbyname on m.

Those names come from mDNS - that is, they are part of a distributed naming domain "local" that is jointly managed by all mDNS-enabled computers on the local net. These domain names are only transparent to the unix shell because naming services on OS X knows about them.

If you have shared your music with iTunes, you're using DNS-SD - that is, iTunes publishes a "service" (music sharing) using DNS-SD and then searches for other music databases the same way.

The DNS-SD C API and code that run below it are all open source, so it is possible to create cross-platform service discovery code. If you try to do this, you will have to pick one of two models:

• Run the DNS-SD daemon (if it doesn't already run) and use the DNS-SD API that uses IPC to talk to the daemon. This configuration is what Apple ships for OS X - the efficiency comes from having only one copy of cached DNS-SD data.
• Compile the DNS-SD network implementation (which is normally in the daemon) into your app directly; Apple recommends this only for embedded systems.

Here are my two tips if you want to try to make this work - that is, this is what kept me up until 4 AM. (Hint: 4 am = stupid...I really solved the bugs the next day after having had a cup of coffee and some sleep.)

1. You can't use gethostbyname to resolve a host into an IP address, despite what Apple's docs say - this only works on platforms like OS X that support mDNS host names "natively".

   The normal flow of operations is to browse for services, "resolve" them to hosts. On a non-Mac platform you need to do a third mDNS operation: use mDNS to resolve the host to an IP address using something like DNSServiceQueryRecord (with a record type of kDNSServiceType_A to convert the host to an address).

2. If you are compiling the mDNS library into your app, you need to initialize it with mDNS_Init_AdvertiseLocalAddresses. This flag will tell mDNS to publish a mapping of your dynamic host name (my_machine.local or whatever) to your actual IP address for each interface.

   If you do not publish, other machines will be able to find and resolve your service, but will fail to then convert your host to an IP. You will be able to get their IPs though. This bug can be masked on OS X, which always publishes itself since you use DNS-SD in daemon form.

   
   

Immediate Mode Instancing

Some definitions: instancing is drawing the same mesh over and over, but in different places/poses/transform/whatever. Instancing comes into play when, for example, you want to draw 8000 of the same runway light fixture or 5000 of the same car. Any time we have a truly huge number of, um, things, we can expect a larger number of instances than art assets.

The idea of instancing is to pay once in CPU time for the entire set of instances that share the same art assets, rather than paying once per instance. Since CPU time is often the limiting factor on rendering, this lets us get a lot closer to the geometry throughput the card is capable of.

Now the simplest way to draw a lot of stuff is something like this:

Set up all GL state first
For each instance
glPushMatrix();
glMultMatrix(...);
glDrawElements(...);
glPopMatrix();

This is instancing by matrix transforms, and it is surprisingly faster than you'd think. (That is, I am surprised that the matrix transforms aren't more expensive.) But we are still paying per instance.

Before continuing, I cannot say this strongly enough: no state change inside the for loop!!! I say this because if you allow state change but try to minimize it, it can sneak up on you. I went to inventory X-Plane's objects for "inner loop state change", expecting to find about 20% of our urban-area art assets doing this. As it turns out, it's more like 80% that are doing this. We're losing 30% or more fps due to this state change.

Immediate Mode Instancing

Immediate mode instancing goes something like this:

Set up all OpenGL state
For each instance
glVertexAttrib (to set matrix transform)
glDrawElements

In conjunction, your vertex shader decodes the vertex attributes that the matrix is being passed down in.

It turns out that this code is at least 30% faster than matrix-transform instancing on OS X. And in hindsight this shouldn't be surprising. I would expect the built in matrix stack to be uniform state, as it isn't expected to change per vertex. And I would expect the update of attribute state to be faster than the update of uniform state.

It may also be that the GL has to do legacy processing with matrices (such as computing inverse matrices) that can be avoided.

Why would you use immediate mode instancing instead of real hardware instancing? Well, if you are on an OS that, despite the availability of hardware instancing for years, doesn't provide the extension, immediate mode instancing provides a useful half-way point.

(In particular, you can pull your immediate mode instance values directly out of the instance array you would have used.)

Compiling a 10.4u Application with GCC 4.2 on OS X

It turns out you can build an OS X 10.4 compatible application using GCC 4.2.  The trick is: you need to install GCC 4.2 Developer Preview 1, available as a download from Apple via ADC.

The developer preview has the GCC 4.2 support files for Darwin 9 (that is, OS X 10.5) in the 10.4u SDK.

So Where Is That Fast Path?

Amongst the many rants I've read about the new OpenGL 3.0 spec, is the claim that the spec needs to be cleaned out and rebuilt so that people can "find the fast path".

What application developers are getting at with this is that OpenGL is a rich API, and not all cards do everything at top speed.  There is usually one fastest way to talk to OpenGL to get maximum throughput.

The problem is: this is ludicrous.  Case in point, the GeForce 8800 - in my Mac Pro, running OS X 10.5.4 and Ubuntu 8.  So what is the fast path?

If I draw my terrain using stream VBOs for geometry and indices that are not in a VBO, I get 105 fps on Linux.  If I then put the indices into a stream VBO, I get 135 fps.  The fast path!

Well, not quite.  The index-without-VBO case runs at 73 fps on OS X, but once those indices go into a VBO, I crash down to 25 fps.  Wrong fast path.

Simply put, you can't spec the fast path, so the spec doesn't matter.  You find the fast path by writing a lot of code and trying it on a lot of hardware.  I can't see there ever being another way, given how many different cards and drivers are out there.

Sound and the 24" iMac

I finally found info on how to make it happen here.

Here's the short version:

• I used Ubuntu 7.10, which has Alsa 1.14, which is a little different.
• Make sure

  options snd-hda-intel model=bbp3

  is somewhere in your modprobe.d files, like in alsa-base.
• Turn on a lot of goo in the mixer.
• Surprisingly the built-in Mic is the digital input!

I now know that XSquawkBox Linux sound does work (but only if X-pane itself is run with --no_sound to allow OSS to open).

Triple-Boot Mac Part 2: MBRs Are Still Hell

A while ago I blogged about the setup process to get OS X 10.4, Windows XP, and Ubuntu 6.06 onto my MacBook Pro. I tried the same trick today on a Mac Pro running OS X 10.5, Windows Vista, and Ubuntu 8.04. To further complicate things, I created a true swap partition.

Now in the old way, Windows had to be last on the disk...fortunately Vista is a little bit more savvy about partitioning (although it stil uses MBR and not GPT partitions). So the order of operations was:

1. Install rEFIt onto the Mac.
2. Use diskutil to partition the main drive, cutting off three new Fat-32 partitions, size 30G, 30G, and 10G, which will become Vista, Linux and a swap partition.
3. Install ext2FSX.
4. Reformat the second Fat-32 partition as ext2 format. (Once ext2FSX is installed, Disk Utility will let you do this.)
   
5. Install Ubuntu, using manual partitioning, and using the last Fat32 partition as a swap file.
6. On reboot, Use rEFITs's partition tool to fix the MBR partition table, which the Ubuntu installer will trash.
   
7. Install Vista onto the middle partition, which hasn't been used. Vista will want you to reformat the Fat-32 partition as NTFS, which is fine. One tricky part: since rEFIt is installed, when Vista reboots you have to pick the volume that the Vista installer thinks it is rebooting to. As far as I can tell, this is always the volume Vista is being installed on.
8. Reboot with the Ubuntu Live CD, and run a grub shell and use "setup" to put a boot sector back on the Linux partition. I don't know why it was missing - possibly an error in setting up Ubuntu the first time.

Some notes on the sketchier steps:

• Why ext2 on Mac? For some reason the 10.5 version of diskutil won't format Linux volume drives. The problem is that when the Ubuntu installer reformats the FAT-32 partition as ext2, OS X doesn't know about it. The next time you boot OS X, your Linux partition is mounted as a FAT-32 volume, which completely trashes it. It took me quite a few tries to figure this out, and I got myself into a state where I had to totally rebuild the startup drive from a full format and reinstall OS X itself. So converting the format to ext2 is a hack to keep the partition from getting smashed by OS X.
• rEFIt is great - given a valid GPT partition table and a screwed up MBR table, it'll usually do what you want. It will prioritize the first four partitions, hence the Linux swap is the last partition, not Vista. (BTW Vista's partition tool sees the swap partition as "unused space" - so do not repartition in Vista.)
• The goo to rebuild grub is more or less:

sudo grub
root (hd0,3)
setup (hd0,3)

Grub will complain a bit, but it will fix the MBR. Note that hd0,3 means the 4th partition on the first hard drive, so your mileage may vary. Tab completion in grub is a great way to scan the local files sytems, e.g. type root(hd to list drives and root(hd0, to list partitions.

Creating OpenGL Objects in a Second Thread - Mac, Linux, Windows

This blog entry explains how to set up OpenGL to create new OpenGL "objects" (vertex buffer objects, display lists, and textures) from a second thread. The benefits of this technique are:

• It moves CPU-intensive driver operations to a second core even if the driver is not multi-threaded internally*. CPU-based on-the-fly texture compression would be an example where you can get a win.
  
• It allows you to move the entire process of building up and loading a scene-graph to a second thread, regardless of the use of OpenGL loading commands (like building of VBOs).
• Because there is no use of the main thread, the main rendering loop can continue to run at full speed (assuming the user has a dual-core machine).

The basic strategy is to build an OpenGL context for each worker thread that will do loading such that the contexts share objects (display lists, etc.) with the main rendering thread's OpenGL context.

(This information comes from in-field tests; if you work on the GL implementation and spot something wrong, or if you find in deployment that this doesn't work, please let me know. Also a warning: the latest version of this code hasn't been in-field very wrong - in the past edge cases have appeared that have caused us to redesign threaded loading more times than I'd like to admit.)

A warning on thread safety: OpenGL's asynchronous nature (sequential deferred execution within a context) leads to some tricky threading bugs that you wouldn't normally see - see this blog post for examples of what can go wrong.

Terminology

While the structures differ between AGL/CGL (Macintosh), WGL (Windows) and GLX (Linux/X11) there are some common features in the way the window manager and the GL interface.

• "Pixel Format." All three operating systems define a pixel format as an opaque description of a series of properties for a rendering context. In my tests, I use the same pixel format for all contexts.
• "Drawable." Basically any window, full screen destination, or off-screen hardware accelerated buffer is a "drawable", and it defines where the result of GL commands to a context end up.

Macintosh

The Macintosh is probably the easiest OS to set this up on, because you don't need to use a drawable for the loader contexts.

1. Create new contexts for each worker thread, using the main rendering context as a source for shared objects.
2. In the start of the worker thread, set the worker thread's context to be current.
   

Windows

On Windows, you'll need to do all of the creation work inside the worker thread:

1. Get a DC from the main window. This should create a new DC that your worker thread can use. (There is apparently nothing wrong with requesting multiple DCs from a single window.)
2. Create a new context for the worker thread.
3. Set the new context to share objects with your main context.
4. Set the worker context to current using the DC you acquired.

Note that your main thread will have to temporarily drop the main context as its current context - otherwise you'll get a "resource busy" error when you try to share contexts - the main thread effectively has that context locked when in use, so you'll want to set the main thread's current context to null, then set it back when done.

Also note that you can't use window-owned or class-owned DCs in this design because either of those will cause you to get back one DC every time you call GetDC and you really need a unique DC for each thread. Be sure to release the DC on the same thread you use to acquire it!

Linux

Linux is similar to the Macintosh, except that the worker threads need a drawable. For this I used p-buffers.

1. The main thread creates a new context, shared with the main context.
2. The worker thread makes a new small pbuffer for itself.
3. The worker thread sets the worker context as current, using the pbuffer as its drawable.

I tried using the main window as the drawable for all threads, but this causes some kind of locking or thrashing between threads and hurts performance of the main renderer. Extra hidden windows were unreliable for reasons I was not able to determine.

Alternatives

The alternative to this design is to use an RPC-like mechanism to execute all OpenGL activity in the main thread, e.g. once per frame pick up any OpenGL work to be done from worker threads, run it, then let them continue. This has some advantages and disadvantages:

• On the pro side, it doesn't require multiple contexts - you may find that some drivers do better executing everything on one context.
  
• OpenGL work is done at the same time in every frame, possibly resulting in more controlled operation, particularly on single-threaded machines.
• On the con side, loader code that needs to create OpenGL objects before proceeding (to keep memory footprint down, for example) may operate slowly since it has limited access to the GL.
  

* This assumes that the driver is not multi-threaded internally, but is capable of performing these CPU-based operations without locking out the render thread via a mutex...a big assumption that I cannot validate or invalidate at this point.

Updated OpenGL Performance Lore

Performance Lore (e.g. "I did X on my app and it got faster so you should do Y") is inherently dangerous in that performance tuning without performance analysis is a bad idea. With that in mind, it does have a use, I think, in at least pointing out possible areas of exploration.

With that in mind, some notes on X-Plane 9 performance:

• It doesn't take a very complex pixel shader to become fill-rate bound! In version 8 (shaders were almost like the fixed function pipeline) X-Plane was almost always limited by the CPU and our ability to find and emit batches, even at high FSAA. With X-Plane 9, even on high end cards, even modest sized shaders are the limiting factor. Never underestimate the cost of doing something ten or twenty million times!
• Overdraw is bad! As soon as you get into shaders, you become fill-rate bound and then overdraw becomes so expensive. So using blending and glPolygonOffset to composite into the frame buffer becomes a bad idea.
• Geometry performance: putting index buffers into VBOs gained us some speed.
  
• Optimizing down the number of vertices by more aggressively using indexing gained some speed, and never hurt, even if the mesh was a bit out of order, even on older hardware.
• Using 16-bit indices on Mac made no difference - perhaps they're not natively supported?
• Using the "static" VBO hint instead of "stream" for non-repeating geometry gave a tiny framerate improvement in a few cases, but mostly caused massive performance failures.

A note on this last point: the driver essentially has to guess a resource usage pattern from your API calls. A lot of them will use a most-recently-used purge scheme, that is, once you run out of VRAM you throw out the most recent thing you used and replace it. If we used a real FIFO, we'd guarantee the worst case: we would flush and reload every resource into VRAM every frame, so MRU makes sense.

The stream vs. static VBO hint will usually get your VBO into VRAM (static) or simply use it by mapping via AGP memory (stream). If your total working set is larger than VRAM (something's going to go over the bus) and a VBO is used only once, stream is the best choice; save VRAM for things that are used repeatedly, like repeating meshes and textures.

So when I tried jamming everything into VRAM, we ran out of VRAM in a big way and started to thrash...MRU works well when we're over budget by a tiny amount, but for a huge shortfall we just end up with thrash.

None of this is really surprising; people have been telling us we'd be shader-bound for years...it just took us a while to get there.

Virtual Memory Dumping for Fun and Profit

We're trying to answer the question: why did X-Plane run out of memory? To answer that, we need to look at the contents of virtual memory. Now dumping virtual memory isn't so bad.

On Windows you call VirtualQuery with a base address. Start at 0 and increment by the returned RegionSize. When you get to dwTotalVirtual (as returned by GlobalMemoryStatus) that's a good time to stop.

On the Mac call vm_region, starting at 0 and incrementing by "size" until it returns an error code. One tricky thing: vm_region will skip over unused virtual memory. To "see" these holes (this is address space that can be used later) compare the address you pass in to the one that's returned. If you pass in a pointer that's in an unused region, the address will be advanced to the next region.

What if you want to show the DLLs/dylibs that are mapped into a given region? I make no promise for the performance of this method but...

On Windows, use GetMappedFileName. Pass in your process handle and a base address from VirtualQuery and it fills in a buffer with a DLL name if possible. This is in psapi.dll so it isn't available on non-NT-derived Windows.

On Mac, use dladdr, passing in the base address of the region. You'll get a dylib file path and a symbol name, although the symbol name tends to be junk.

Windows Vista - Stable API

I'm not a huge fan of Microsoft or Windows Vista, and I do all my primary development on a Mac. But...

The same X-Plane code that runs on Windows Vista will run on Windows 98 SE.

In that time Apple has changed the API (introducing Carbon), the ABI twice (moving from CFM to Mach-O, and GCC3 to 4*) and the instruction set (PPC to Intel), as well as the compiler twice (MPW to X-Code, or more likely Metrowerks to X-Code, and GCC 3 to 4).

I love Apple dearly, but if I write an app and burn the source code, I wouldn't be surprisde if the binaries run on Windows 5 years from now. I wouldn't make that prediction for the Mac.

*Is it fair to call the upgrade from GCC 3 to 4 an ABI change or compiler change? Not really, they're not full ones, but besides the usual slight changes, the real problem is that GCC 4 generates run-time dependencies on shared libraries that were not available in older Mac Operating Systems. This is why X-Plane is not alone in requiring OS X 10.3.9 or higher - 10.3.9 is the oldest operating system that has the runtime for GCC 4. Applications that can run all the way back to OS X 10.2 are actually building their PowerPC executable code against a different deployment target, which requires extra makefile gymnastics.

Triple-Boot Mac (and MBR hell)

It only took about 8 hours of total time, but my MacBook Pro can now triple-boot into OS X 10.4.7, Windows XP Home SP2, and Ubuntu Linux (6.06 I think).

The install is actually not that bad once you've been through it once...the process is basically:

• Use diskutil to dynamically build two new partitions. The Mac comes with one "EFI" partition for boot control and one HFS+ partition. MBR gives you four max, so you build a Linux one followed by a FAT 32 one.
• Install Windows XP. Tedious and annoying but not complicated. Windows takes longer to boot from CD than an entire OS X install from scratch.
• Install Linux. This is the slightly dangerous part, as I found out.

So two things took a long time:

First, for some reason my brand new, genuine Windows XP CD-ROM doesn't work very well. Setup had at least two hangs, a failure to init a disk, a BSOD, and an assertion failure. Maybe all of those holograms on the disk play havoc with the optical drive? So the biggest single item was a 5 hour Windows install that mostly involved trying to boot setup over and over.

Windows XP asks questions during the install, so you have to sit there and watch it. Furthermore, if you are using rEFIt like I am, then you'll have to manually select Windows each time the installer reboots itself. The installer normally expects to reincarnate itself, so if your machine mysteriously reboots without warning be sure to keep reselecting to boot from the Windows install CD-ROM.

The other problem was that, almost certainly due to my own sleepiness this morning, I managed to splat LILO onto the boot block of the Windows partitiont. Ooops. Most tutorials online state that the only fix for this is a complete reinstall of both Windows and Linux, but fortunately I discovered this is not true.

It turns out that you can very gently run fixmbr on your Windows partition without disturbing triple-boot goodness. It's pretty much that simple...just launch the windows setup CD, wait 30 minutes for it to load every driver ever written, launch the recovery tool and use fixmbr on the appropriate drive. Linux and Mac continue to boot via rEFIt.

Now the only issue is: Windows can't understand Mac or Linux partitions, Linux can't understand Mac partitions, and while the Mac will figure anything out, it's not automounting.

MacBook Pro

This is just me blogging about getting a new toy...I just upgraded my work Mac laptop to one of the new 15" MacBook Pros. Some random musings...

They're really, really nice machines. Apple does a great job on the "appliance" aspect of the machines. The power cable attaches magnetically. The machine comes with a remote control that integrates with every media app that ships with the system. (All of which can share media from a remote server right out of the box. Digital housem anyone?) As always with Apple power management is good - Apple was preaching power management 10 yearse ago when no one knew what that was about, so by this point it's down in all levels of the software.

The MacBook Pros cost a bloody fortune ... I wouldn't own one if it wasn't for work. But the pro models come with a Radeon X1600, a very nice graphics chip for flight sim. By comparison the regular Mac Books ship with Intel integrated graphics, which are just crap...I'll have to rant on that some other time. Bottom line is - the non-pro MacBooks really aren't usable for flight simming, let alone flight sim dev. But for non-flight-sim users, the regular MacBooks aren't a bad way to get a nice Apple media-friendly machine.

Performance is great. Compile and sim speed rival the G5, a full sized heavy-duty desktop, and should be even better when the second gig of mem comes in the mail.

One quirk that I've seen in web forums too: when the machine is on batteries the wireless network will hang up if left idle for too long. Leaving a ping running keeps the card alive, but email checks aren't frequent enough. I haven't found a setting to keep wireless alive all the time.

EDIT: there is one way the G5 hauls the MacBook Pro - obvious but - virtual memory performance from the 7200 RPM SATA drives in the G5 is light-years ahead of the 5400 drive in the laptop. This is one case where a desktop comes through; the G5 is a "no-wait" machine because the paging performance is so good. But that's a form-factor issue.

I/O Error (-36) when writing on Mac

For months X-Plane's web-based installer has been returning I/O errors randomly to users withithe new Intel Macs. The specific offending routine was FSWriteFork but we couldn't figure out why.

We eventually turned to DTS, Apple's third party developer support team - you can buy a tech support "incident" from them - that is, assistance with one bug. It was from them that I got the missing tidbit of information:

Usually ioErr (-36) means something has gone deeply wrong at the lowest levels. Here's what the actual header file says if you have Apple's Carbon SDK:

ioErr = -36, /*I/O error (bummers)*/

That is, generally speaking, application programs can't recover from this - it would usually mean the hard drive is going bad. By comparison paramErr (-50) means you passed something illegal to the function.

Well it turns out that if you pass a block of memory to FSWriteFork and part of that memory is not mapped (that is, trying to access that memory to do the disk write generates a segfault) then you get ioErr back, not paramErr. That tip led us to realize that what we really had was (blushes) bad endian-swapping code that (on little-endian Macs only) generated bogus pointers. Depending on luck and the stars, that bogus memory might be mapped (thus writing junk into the file) or unmapped (generating the -36 error), hence the unreliable behavior.

In hindsight this isn't surprising, once you realize that Mac Carbon file calls are written on top of BSD file manager calls. I don't know the actual implementation of FSWriteFork, but what if it uses BSD's "write"? Well, that can return EFAULT if you pass in unmapped memory.