Debugging OpenGL

There are only two debugging techniques in the universe:

1. printf.
2. /* */

(I would say a nice IDE that gives you stepping, logging, variable examination, etc. is just a really nice variant of these two techniques that avoids recompiles.)

OpenGL poses some particular challenges to debugging: often OpenGL’s failure mode will be to draw nothing, or fill the entire screen with solid black or white. A lot of OpenGL’s internal state is not easily printed by the application, and since OpenGL is state-based, removing code can have cascading effects.

You can still utilize these debugging techniques with OpenGL. Here are some things I’ve done when working on X-Plane to try to retain my sanity while debugging OpenGL code:

• Premature swapping: in a double-buffered context, you can swap the context and then wait for a mouse click. If you do this at intermediate points in your drawing, you can see your drawing in steps as it takes place. This is sort of like printf in that it lets you see internal program state that would otherwise be hidden.
• Removing drawing state: when something goes wrong you can start to remove the OpenGL tricks you do until you arrive at a simpler problem. Turn off texturing, turn off texture projection, turn off blending, etc. Lighting is a good one to turn off when things get strange, since bad normals can cause abnormal results.
• Use a different primitive. If your triangle fan looks funny, try drawing it as a line loop. Or set the polygon mode from fill to line - the wire frame may reveal something not obvious in a solid model.
• If you use display lists, set up your code to allow for immediate-mode execution as an option; this will let you catch display-list-related problems (I have seen drivers do things differently in display lists, and there are lots of ways to make coding mistakes with them) and also let you apply other techniques inside the list, like swapping to see work in progress.
• #define common modes: for all of the above tricks, if you’re going to be doing them a lot, design the code for debugging. I try to leave a series of #define switches at the top of my translation unit that apply debug-mode effects.

As a side thought, I have found Vertex Buffer Objects (VBOs) to be easier to debug than display lists for a few reasons: it’s pretty easy to revert VBOs to client vertex arrays (CVAs) at which point you can inspect the data in the debugger. Display lists can contain state commands as well as geometry, so it may not be obvious what the full effects of calling a display list is; since VBOs are just geometry you know what you are getting.

An example: in X-Plane we had some code that essentially did this:

static int has_dl = 0;
if (has_dl == 0)
{
// Build the car display list the firs time we need it.
has_dl = glGenLists(1);
glNewList(has_dl, GL_COMPILE);
draw_a_car(); glEndList();
}
glCallList(has_dl);

This would be okay except that everything draw_a_car does gets compiled into the list. Furthermore, any non-OpenGL side-effects of draw_a_car get run only once. Neither of these problems are made clear by the code.

IsWindowVisible can be deceiving!

Recently I had to debug a bit of code that was causing an Active-x control to be hidden. I found that if i commented out the two ShowWindow lines below, the problem went away.

bool bVisibile= mChild.wnd.IsWindowVisible();
mChild.wnd.ShowWindow(SW_HIDE);
mChild.wnd.MoveWindow(x, y, w, h);
mChild.wnd.ShowWindow(IsWindowVisible ? SW_SHOW : SW_HIDE);

That snippet of code basically saves the current state of the window, then hides it, then moves it, then sets the state back to what it was before. So it should be a completely safe statement right? WRONG! The concept seems to make sense but a careful look at the Microsoft documentation will reveal the problem. Look particularly at this sentance.

If the specified window, its parent window, its parent’s parent window, and so forth, have the WS_VISIBLE style, the return value is nonzero. Otherwise, the return value is zero.

So let’s assume that the child window (the one we’re trying to move) had its WS_VISIBLE bit set so that it can be visible. Let’s also assume that either its parent or its parent’s parent or both were NOT visible. Well then IsWindowVisible would indicate that the child window was in fact NOT visible. Hmm this makes sense because the user clearly cannot see the window if a parent is hidden, BUT the child window’s bit is still set to visible. Well this code snippet doesn’t realize that and sets the child window to hidden. That’s fine for now. We perform our window move but here’s where it dies. We go to set our child window back to its original state…which was what? It was set to visible but that’s not what that function told us. So we leave the child window as hidden. Now when it comes time for the parent that was hidden to be displayed and show the Active-X control, the child window was explicitly set to hidden and cannot be seen.

Bug in the code? Yeap! Bug in the documentation? Possibly. I think the biggest problem is that the naming of this function is a bit misleading. Developers who use this function should ask themselves this question from a users standpoint. “Is this window visible to the user?” That’s how IsWindowVisible will answer the question. Most developers just have the tendancy to become focused on the window they’re trying to manipulate and forget that it’s really a piece of a bigger picture.

X-Code fun with settings

More X-Code ranting and raving…

One thing that’s nice about X-Code is that it shows you the relative path fragments used to track each project item. In particular if you have a tree of files represented as a tree of items in X-Plane, make sure they’re set to relative-to-enclosed group! If you do this, then you can move whole source-trees around and reconnect them in X-Code with a single command. X-Code’s reconnecting to missing files is pretty good anyway, but setting everything to enclosed-group-relative up front can really save time later.

Last night Austin and I got X-Code into a state where the debugger was clearly out of sync with the source code. Scary! I’ll post more if we can figure out what was going on.

A trick from the Apple guys: X-Code uses environment variables to build up the build environment in a huge complex frightening recursive set of entries…it is very powerful and flexible, but it’s not always obvious what set of effects are combining to create the final project. But they pointed out that if you make a shell-script post-build step (in target-view pick Project->New build Phase–>New Run Script Build Phase) then you can enter a dummy shell script. When the script is executed, X-Code will first use 8 tons of “setenv” commands to dump the internal state of the configuration into the shell environment. The result is visible in the command-line view (normally hidden) in the build progress window. In other words, you can use this to easily dump every single configuration variable’s final value and understand what you’re really getting.

I’ve spent the morning really cleaning out the project - our target now has ZERO customized target settings! In other words, when we make a new target we will not have to change ANYTHING. Or so the theory goes.

More X-Code Ranting - and Tips

I have in the past been a big CodeWarrior fan…here are some thoughts on transitioning to X-Code…(most of the credit must go to the DTS and Apple tools team for these tips…)

- Editor and code browsing…you can get X-Code to do almost everything CodeWarrior can in terms of useful code management. Command-double-click rather than option-double-click to jump to a definition, and the syntax coloring can’t color keywords, but finding symbols and jumping between source files is there.

- There’s a preference to make the layout look like a CodeWarrior project. I am trying to get myself used to the default view because it seems like it conveys more information in the long term. X-Code has a VC.net-like mode too in case you like pain.

- Compile is slow, but it’s also per-CPU and networkable. If you have a dual-core or dual-CPU machine, that’s a nice win and starts to close the gap. If you have a lot of headers, compile is actually faster than CodeWarrior when you crank the optimizers on both sims. (Warning: networked compile does not work between machines with different endian-nesses.)

- Once you understand the concept, preferences management in X-Code is definitely superior to CodeWarrior. In X-Code, every setting is a key-value pair. The target preferences are layered on top of project preferences - any preference can be anywhere. This means that you can factor the vast majority of your preferences out to your project, which simplifies making global settings changes.

- X-Code has a real concept of debug and release builds…in CodeWarrior if you have five applications that need to be debug, inlined debug, release, optimized release, and optimized universal release then that’s 25 targets. In X-Code that’s 5 targets and 5 build modes; most of the target settings are factored into the project anyway. The end result is a lot less setting check-boxes.

I’ll try to post more as I get there. Basically it only took one day to convert X-Plane from CodeWarrior to X-Code, and one more day to make X-Plane run on Intel Mac hardware.

Rebate Scams

This is a very important article to read if you’re like me and you love purchasing items that come with rebates. It could save you some hassle and some cash!

http://www.slate.com/id/2084210/

X-Code Scorecard

I’m in the process of converting our app from CodeWarrior to X-Code. X-Code is Apple’s IDE based on the GNU toolset.

Good things about X-Code (most of these things are good in CodeWarrior already):

• Recursive include directories work.
• Precompiled headers work reasonably well.
• Manages dependencies, more or less.
• Easy to temporarily preprocess and view a file.
• Targets and build configurations are separate - this is an improvement over CodeWarrior.
• View-by-groups is a nice way to work.

Bad things about X-Code (mostly inheritted from GCC):

• Compile speed is significantly slower than CodeWarrior.
• Dependency checker sometimes goes haywire and recompiles the PCH - at that point go get a cup of coffee.
• GNU STL - enough said.
• No per-group compile settings - too bad given the flexibility of the rest of the system. You can’t do this in CodeWarrior but you can do this in VC.net.

If you tried X-Code a few years ago when it first came out, it’s definitely in better shape now. But I’d trade all of the features not found in CodeWarrior to get my compile speed back!

Fun with threads

When it came time to rewrite the X-Plane threading code (which has to run on three APIs) I did what any modern lazy programmer does: I typed “posix threads condition variable” into Google and got this helpful page.

http://www.cs.wustl.edu/~schmidt/win32-cv-1.html

I make no vouchers for how accurate its contents are, but I can tell you that the authors are a lot smarter than I am; I say that because I made the mistake of changing their code and induced a bunch of bugs. But I learned something in the process!

Posix provides condition variables. Basically one or more threads block on the condition variable, and when the condition variable is signaled, one or more threads wakes up and does something useful. When you block on a condition variable, you specify a mutex to release - that release is atomic, meaning no other thread can acquire that mutex you are dropping before you are asleep and blocking on the condition variable. When your thread wakes up from the condition variable, it then acquires the mutex. (This is not atomic - it’s very possible that when you wake up the mutex is locked by someone else and you’ll then go right back to sleep waiting for the mutex. This sounds to me like the recipe for a box-car problem, but that’s off-topic.)

The zen of the condition variable is this: when you have some kind of producer/consumer problem you use a mutex to protect the precious internals of your object and the condition variable to implement blocking operations. A blocking operation acquires the mutex, looks at the internals, decides it needs to sleep, and atomically drops the mutex and blocks on the condition variable. When it fully wakes up (having been signaled by the condition variable and reacquired the mutex) your thread now can re-examine the internals and decide whether it can proceed.

The pseudo-code looks something like this:

void blocking_op ()
{
lock_mutex(my mutex);
mark guts that I am sleeping
while (guts say I am not ready)
wait on condition (my condition, my mutex)
guts are ready - do something with them
mark guts that I am not sleeping
release mutex (my mutex)
}
void unblocking_op()
{
lock_mutex(my mutex);
if (someone is sleeping)
signal
do something with guts
unlock_mutex(my mutex);
}

My utilization of this was for a message queue, but the technique can work with any producer/consumer-type problem. Now being the rocket scientist I am, I looked at this code and said “while loop - who needs it? The condition variable is always signaled when the unblocking op has occurred - why would the guts not be ready?

Turns out there is a race condition, as the paper above describes. Here’s how it works:

First we must understand that the blocking operation only sleeps on the condition variable when it cannot immediately run. In other words, if our message queue has a message and the mutex is free, we just run straight through without any blocking.

Second we must understand that waking up from the condition variable and acquiring the mutex are not atomic. They cannot be - at the instant the condition is signaled the mutex is never free (because it is owned by the unblocking thread). Even if this was not true (I believe that signaling from outside the mutex would introduce a race condition - but this is not the race condition we are looking for) another thread could certainly own the mutex for random reasons. So by Murphy’s Law of threaded code, something is going to happen between when our signalled blocked thread wakes up and when it can actually look at the guts of our object and do something.

The problem is: what states is our object in during this time? In our message queue’s case, the message queue has a message in it. So if a second reader thread comes along and acquires the mutex before our first reader wakes up and acquires it, then the second reader will consume the message without ever sleeping and by the time our first reader actually wakes up, the message is gone!

This is why the while loop is important. The while loop allows the first reader who has had his message stolen to go back to sleep and wait for the next one.

(An implication of all this is that our message queue based on condition variables is not “fair” - if two readers both read the message queue, the earlier one before it is empty the second one after it is filled, the second one may win the message without blocking, while the first one may stay slept. This is probably good for certain applications in some ways, but the thrashing of the slept thread may be unacceptable for others.)

If there’es a lesson from all of this, I think it’s this: threads are fundamentally complex and non-trivial to code. Think carefully before using threads about what the real benefit is and be sure it outweighs the cost of more complex development, less debuggable code and increased maintenance costs.