Fixing Camera Shake on Single Precision GPUs

I've tried to write this post twice now, and I keep getting bogged down in the background information. In X-Plane 11.10 we fixed our long-standing problem of camera shake, caused by 32-bit floating point transforms in a very large world.

I did a literature search on this a few months ago and didn't find anything that met our requirements, namely:

• Support GPUs without 64-bit floating point (e.g. mobile GPUs).
• Keep our large (100 km x 100 km) mesh chunks.

I didn't find anything that met both of those requirements (the 32-bit-friendly solutions I found required major changes to how the engine deals with mesh chunks), so I want to write up what we did.

Background: Why We Jitter

X-Plane's world is large - scenery tiles are about 100 km x 100 km, so you can be up to 50 km from the origin before we "scroll" (e.g. change the relationship between the Earth and the primary rendering coordinate system so the user's aircraft is closer to the origin).  At these distances, we have about 1 cm of precision in our 32-bit coordinates, so any time we are close enough to the ground that 1 cm is larger than 1 pixel, meshes will "jump" by more than 1 pixel during camera movement due to rounding in the floating point transform stack.

It's not hard to have 1 pixel be larger than 1 cm. If you are looking at the ground on a 1920p monitor, you might have 1920 pixels covering 2 meters, for about 1 mm per pixel.  The ground is going to jitter like hell.

Engines that don't have huge offsets don't have these problems - if we were within 1 km of the origin, we'd have almost 100x more precision and the jitter might not be noticeable. Engines can solve this by having small worlds, or by scrolling the origin a lot more often.

Note that it's not good enough to just keep the main OpenGL origin near the user. If we have a large mesh (e.g. a mesh whose vertices get up into the 50 km magnitude) we're going to jitter, because at the time that we draw them our effective transform matrix is going to need an offset to bring the 50 km offset back to the camera.  (In other words, even if our main transform matrix doesn't have huge offsets that cause us to lose precision, we'll have to do a big translation to draw our big object.)

Fixing Instances With Large Offsets

The first thing we do is make our transform stack double precision on the CPU (but not the GPU). To be clear, we need double precision:

• In the internal transform matrices we keep on the CPU as we "accumulate" rotates, translates, etc.
• In the calculations where we modify this matrix (e.g. if we are going to transform, we have to up-res the incoming matrix, do the calculation in double, and save the results in double).
• We do not have to send the final transforms to the GPU in double - we can truncate the final model-view, etc.
• We can accept input transforms from client code in single or double precision.

This will fix all jitter caused by objects with small offset meshes that are positioned far from the origin.  Eg. if our code goes: push, translate (large offset), rotate (pose), draw, pop, then this fix alone gets rid of jitter on that model, and it doesn't require any changes to the engine or shader.

We do eat the cost of double precision in our CPU-side transforms - I don't have numbers yet for how much of a penalty on old mobile phones this is, but on desktop this is not a problem. (If you are beating the transform stack so badly that this matters, it's time to use hardware instancing.)

This hasn't fixed most of our jitter - large meshes and hardware instances are still jittering like crazy, but this is a necessary pre-requisite.

Fixing Large Meshes

The trick to fixing jitter on meshes with large vertex coordinates is understanding why we have precision problems.  The fundamental problem is this: transform matrices apply rotations first and translations second. Therefore in any model-view matrix that positions the world, the translations in the matrix have been mutated by the rotation basis vectors. (That's why your camera location is not just items 12,13, and 14 of your MV matrix.)

If the camera's location in the world is a very big number (necessary to get you "near" those huge-coordinate vertices so you can see them) then the precision at which they are transformed by the basis vectors is...not very good.

That's not actually the total problem. (If it was, preparing the camera transform matrix in double on the CPU would have gotten us out of jail.)

The problem is that we are counting on these calculations to cancel each other out:

vertex location * camera rotation + (camera rotation * camera location) = eye space vertex

The camera rotated location was calculated on the CPU ahead of time and baked into the translation component of your MV matrix ,but the vertex location is huge and is rotated by the camera rotation on the GPU in 32-bits.  So we have two huge offsets multiplied by very finicky rotations - we add them together and we are hoping that the result is pixel accurate, so that tiny movements of the camera are smooth.

They are not - it's the rounding error of the cancelation of these calculations that is our jitter.

The solution is to change the order of operations of our transform stack. We need to introduce a second translation step that (unlike a normal 4x4 matrix operation), happens before rotation, in world coordinates and not camera coordinates.  In other words, we want to do this:

(vertex location - offset) * camera rotation + (camera rotation * (camera location - offset)) = ...

Heres' why this can actually work: "offset" is going to be a number that brings our mesh roughly near the camera. Since it doesn't have to bring us to the camera, it can change infrequently and have very few low-place bits to get lost by rounding.  Since our vertex location and offset are not changing, this number is going to be stable across frames.

Our camera location minus this offset can be done on the CPU side in double precision, so the results of this will be both small (in magnitude) and high precision.

So now we have two small locations multiplied by the camera rotation that have to cancel out - this is what we would have had if our engine used only small meshes.

In other words, by applying a rounded, infrequently  changing static offset first, we can reduce the problem to what we would have had in a small-world engine, "just in time".

You might wonder what happens if the mesh vertex is no-where near our offset - my claim that the result will be really small is wrong. But that's okay - since the offset is near the camera, mesh vertices that don't cancel well are far from the camera and too small/far away to jitter. Jitter is a problem for close stuff.

The CPU-side math goes like this: given an affine model-view matrix in the form of R, T (where R is the 3x3 rotation and T is the translation vector), we do this:

// Calculate C, the camera's position, by reverse-

// rotating the translation

C = transpose(R) * T

// Grid-snap the camera position in world coordinates - I used 

// a 4 km grid. smaller grids mean more frequent jumps but 

// better precision.

C_snap = grid_round(C)

// Offset the matrix's translation by this snap (moved back 

// to post-rotation coordinates), to compensate for the pre-offset.

T -= R * C_snap

// Pre-offset is the opposite of the snap.

O = -C_snap

In our shader code, we transform like this:

v_eye = (v_world - O) * modelview_matrix

There's no reason why the sign has to be this way - O could have been C_snap and we could have added in the shader; I found it was easier to debug having the offset be actual locations in the world.

Fixing Hardware Instancing

There's one more case to fix. If your engine has hardware instancing, you may have code that takes the (small) model mesh vertices and applies an instancing transform first, then the main transform. In this case, the large vertex is the result of the instancing matrix, not the mesh itself.

This case is easily solved - we simply subtract our camera offset from the translation part of the hardware instance. This ensures that the instance, when transformed into our world, will be near the camera - no jitter.

One last note: on some drivers I found the driver was very finicky about order of operations - if the calculation is not done by applying the offset before the transform, the de-jitter totally fails. The precise and invariant qualifiers didn't seem to help, only getting the code "just right" did.

Importance Sampling: Look Mom, No Weights

For anyone doing serious graphics works, this post will be totally "duh", but it took me a few minutes to get my head straight, so I figure it might be worth a note.

Fair and Balanced or Biased?

The idea of importance sampling is to sample a function in a biased way, where you intentionally bias your samples around where most of the information is. The result is better leverage from your sampling budget.

As an example, imagine that we want to sample a lighting function integrated over a hemisphere, and we know that that lighting function has a cosine term (e.g. it is multiplied by the dot product of the light direction and the normal.)

What this means is that the contributing values of the integration will be largest in the direction of the normal and zero at 90 degrees.

We could sample equally all around the hemisphere to learn what this function does. But every sample around the the outer rim (90 degrees off) of the hemisphere is a total waste; the sampled function is multiplied by cos(90), in other words, zero, so we get no useful information. Spending a lot of our samples on this area is a real waste. Ideally we'd sample more where we know we'll get more information back (near the normal) and less at the base of the hemisphere.

One way we can do this is to produce a sample distribution over the hemisphere with weights. The weight will be inversely proportional to the sample density. We come up with a probability density function - that is, a function that tells us how likely it is that there is information in a given location, and we put more samples where it is high, but with lower weights.  In the high probability regions, we get the sum of lots of small-weight samples, for a really good, high quality sampling. In the low probability region, we put a few high weight samples, knowing that despite the high weight, the contribution will be small.

You can implement this by using a table of sample directions and weights and walking it, and you can get just about any sampling pattern you want.  Buuuuuut...

Lighting Functions - Kill the Middle Man

With this approach we end up with something slightly silly:

1. We sample a lighting equation at a high density region (e.g. in the middle of a specular highlight).
2. We end up with a "strong" lighting return, e.g. a high radiance value.
3. We multiply this by a small weight.
4. We do this a lot.

In the meantime:

1. We sample a lighting equation in a low density region.
2. We end up with a very low radiance value.
3. We multiply it by a heavy weight.
4. We do this once.

Note that the radiance result and the weight are always inverses, because the probability density function is designed to match the lighting function. The relative weight of the brightness thus comes from the number of samples (a lot at the specular highlight, very few elsewhere).

We can simplify this by (1) throwing out the weights completely and (2) removing from our lighting equation the math terms that are exactly the same as our probability density function.  Steps 2 and 3 go away, and we can sample a simpler equation with no weighting.

Here's the key point: when you find a probability density function for some part of a lighting equation on the interwebs, the author will have already done this.

An Example

For example, if you go look up the GGX distribution equation, you'll find something like this:

GGX distribution:

float den = NdotH * NdotH * (alpha2 - 1.0f) + 1.0f;
return alpha2 / (PI * den * den);

That's the actual math for the distribution, used for analytic lights (meaning, like, the sun).  The probability density function will be something like this:

float Phi = 2 * PI * Xi.x;
float CosTheta = sqrt( (1 - Xi.y) / ( 1 + (a*a - 1) * Xi.y ) );
float SinTheta = sqrt( 1 - CosTheta * CosTheta );

(In this form, theta of 90 points at your normal vector; Xi is a 2-d variable that uniformly samples from 0,0 to 1,1. The sample at y = 0 samples in the direction of your normal.)

Note that the probability density function contains no weights. That's because the sample density resulting from running this function over a hemisphere (you input a big pile of 0,0 to 1,1 and get out phi/theta for a hemisphere) replaces the distribution function itself.

Therefore you don't need to run that GGX distribution function at all when using this sampling. You simply sample your incoming irradiance at those locations, add them up, divide by the samples and you are done.

Doing It The Silly Way

As a final note, it is totally possible to sample using a probability density function that is not related to your actual lighting equation - you'll need to have sample weights and you'll need to run your full lighting equation at every point.

Doing so is, however, woefully inefficient. While it is better than uniform sampling, it's still miles away from importance sampling with the real probability density function replacing the distribution itself. 

Deferred Weirdness: What Have We Learned?

To paraphrase the famous Nike Tiger Woods ad, "...and, did you learn anything?"

X-Plane is more of a platform than a game - we support content that comes from third parties, is not updated in synchronization with the engine itself, is sometimes significantly older than the current engine code, and is typically created by authors who have limited or no access to the engine developers.

The easiest, quickest, most efficient way to make deferred rendering work is to set the rules first and customize the content creation process around them.  X-Plane's platform status makes this impossible, and most of the nuttiness in our deferred pipeline comes from bending the new engine to work with old content.

We have a number of design problems that are all "difficult" in a deferred engine:

• A wide Z buffer range means we need to render twice.
• Content is often permeated with alpha but still needs to be part of a deferred render.
• Some content must be linearly blended, some must be sRGB blended, and there isn't any wiggle room.

Coping with any one of these problems is quite doable within a deferred renderer, but coping with all three at once becomes quite chaotic.  Having to maintain two blending equations (sRGB and linear) is particularly painful.

(This sits on top of the typical problems of being geometry bound on solid stuff and fill-rate bound on particles.)

So if there's a "learn from our fail"* in this, it might be: try to limit your design problem to only one "rough edge" in the deferred renderer if possible - it's not hard to put a few hacks into a deferred renderer but keeping several of them going at once is juggling.

* I should say: I'm just being snarky with "learn from our fail" - I consider our deferred rendering solution a success in that it provides our users with the benefits of deferred rendering (huge numbers of lights, better geometric complexity, soft particles, and there are other techniques we're just starting to experiment with) while supporting legacy content with very little modification.

The code is under stress because it does a lot, and while it's cute for me as a graphics programmer to go "oh my life would be easier if I could ignore alpha, color space, and go drink margaritas on the beach", the truth is that the code adds a lot of value to the product by provingg a lot of features at once.

I just don't want to have to touch it ever again. :-)

Deferred Weirdness: When Not to be Linear

In my previous post, I described X-Plane 10's multipass scheme for deferred rendering.  Because we need to simultaneously operate in two depth domains with both deferred and forward rendered elements, the multi-pass approach gets very complicated, and much stenciling ensues.

But the real wrench in the machinery is sRGB blending.  It was such a pain in the ass to get right that I can safely say it wouldn't be in the product if there was any way to cheat around it with art.

Color Space Blending

First: when I say "sRGB" blending, what I mean is: blending two colors in the sRGB color space.  sRGB is a vaguely perceptual color space, meaning equi-distant numeric pixel values appear about equi-distant in brightness - if you saw a color ramp, it would look "even" to you.  It's not!  The sRGB stripe looks even to humans because we have increased visual sensitivity in the darker brightness ranges. (If we didn't, how could we stumble for the light switch and trip over the dog at night?)

In the sRGB world you mix red (255,0,0) and green (0,255,0) and get "puke" (128,128,0).

Linear color spaces refer to color spaces where the amount of photons or physical-whatever-thingies are linear.  Double the RGB and you get twice as many photons.  Linear spaces do not look "linear" to humans, see above about dark and dogs.

In linear space if you mix red (255,0,0) and green (0,255,0) you get yellow (186,186,0 if we translate back to sRGB space).

When to Be Linear

There are two cases where we found we just absolutely had to be linear:

1. When accumulating light from deferred lights, the addition must be linear.  Linear accumulated lights look realistic, sRGB accumulated lights look way too stark with no ambiance.
2. When accumulating light billboards, it turns out that linear also looks correct, and sRGB additive blending screws up the halos drawn into the billboards.

In both cases, linear addition of light gives us something really important: adding more light doesn't double the perceived brightness.  (Try it: go into a room, turn on one light, then a second.  Does the room's brightness appear to double?  No!)  For billboards, if a bunch of billboards "pile up" having the perceptual brightness curve taper off is a big win.

When to Be sRGB

The above "linear" blending cases are all additive.  Positive numbers in the framebuffer represent light, adding to them "makes more light", and we want to add some kind of linear unit for physically correct lighting.

But for more conventional "blending" (e.g. adding light on top takes away light underneath) linear isn't always good.

Traditional alpha blending is an easy for artists to create effects that would be too complex to simulate directly on the GPU.  For example, the windows of an airplane are a witches brew of specularity, reflection, refraction, absorbtion, and the BDRF changes based on the amount of grime on the various parts of the glass. Or you can just let your artist make a translucent texture and blend it.

But this 'photo composited' style blending that provides such a useful way to simulate effects also requires sRGB blending, not linear blending.  When an 'overlay' texture is blended linear the results are too bright, and it is hard to tell which layer is "on top".  The effect looks a little bit like variable transparency with brightness, but the amount of the effect depends on the background.  It's not what the art guys want.

Alpha When You Least Expect It

Besides the usual sources of "blended" alpha (blended meaning the background is darkened by the opacity of the foreground) - smoke particles, clouds, glass windows, and light billboards - X-Plane picks up another source of alpha: 3-d objects alpha fade with distance.  This is hugely problematic because it requires the alpha blending you get when you put alpha through the deferred renderer to be not-totally-broken; we can't just exclude all 3-d from the deferred render.

I mention this now because we have to ask the question: what blending mode are we trying to get in the deferred renderer (to the extent that we have any control over such things)?  The answer here is again sRGB blending.

Faking Alpha in a GBuffer

Can we fake alpha in our deferred renderer?  Sort of.  We can choose to blend (or not blend) various layers of the G Buffer by writing different alphas to each fragment data component; we can gain more flexibility by using pre-multiplied alpha, which lets us run half the blending equation in shader (and thus gives us separate coefficients for foreground and background if desired.

Fake alpha in a GBuffer has real problems.  We only get one 'sample' per G-Buffer, so instead of getting the blend of the lighting equation applied to a number of points in space, we get a single lighting equation applied to the blend of all of the properties of that point.  But blending is better than nothing.  We blend our emissive and albedo channels, our specular level, our AO, and our normal.  (Our normal map is Lambertian Azimuth equal area and it blends tolerably if you set your bar low.)

The only channel we don't blend for our blended fragments is eye position Z; even the slightest change to position reconstruction causes shadow maps to alias and fail - hell, shadow maps barely work on a good day.

The G-Buffer blending is all in sRGB - the albedo and emissive layers are 8-bit sRGB encoded.

Adding It All Up

The emission and albedo layers of the G-Buffer must be added in sRGB space.  This is not ideal (because emission layers contain light) but it is necessary.  Consider two layers of polygons being drawn into a G-Buffer.  The bottom is heavy on albedo, the top heavy on emissive texture.  As we "cross-fade" them with alpha, we are actually darkening the albedo and lightening the emission layer - two separate raster ops into two separate images.  This only produces a correct sRGB blend if we know that the two layers will later be added together in sRGB.  In other words:

blend(A_alb+A_lit,B_alb+B_lit,alpha)

is only equal to

blend(A_alb,B_alb,alpha)+blend(A_lit,B_lit,alpha)

if the blending and addition all happen in the same color space. The top equation is how "blended" geometry work in a forward renderer (albedo and emissive light summed in shader before being blended into the framebuffer) and the bottom equation is how a deferred renderer looks (blending done per layer and the light addition done later on the finished blend).

Once we add and blend in sRGB space in our deferred renderer a bunch of things do work 'right':

• Alpha textures that can't be excluded from the deferred renderer that need sRGB blending work, as do alpha fades with distance.
• We can mix & match our emissive and albedo channels the way we would in a forward renderer and not be surprised.
• Additive light from spills is still linear, since it is a separate accumulation into the HDR framebuffer.

There is one casualty:

• We cannot draw linear additive-blended "stuff" into the deferred renderer.

This last point is a sad casualty - looking at the 11-step train-wreck of MRT changes in the previous post, if we could draw linear-blended stuff into the deferred renderer (even if just by using framebuffer-sRGB) we could save a lot of extra steps.  But we would lose sRGB alpha blending for deferred drawing.

 

I will try to summarize this mess in another post.

Deferred Weirdness: Collapsing Two Passes

X-Plane's deferred pipeline changed a lot in our 10.10 patch, into a form that I hope is final for the version run, because I don't want to have to retest it again.  We had to fix a few fundamental problems.

Our first problem was to collapse two drawing passes.  X-Plane needs more precision than the Z buffer provides.  Consider the case where you are in an airplane in high orbit, in your 3-d cockpit.  The controls are a lot less than 1 meter away, but the far end of the planet below you might be millions of meters away.  With the near and far clip planes so far apart (and the near clip plane so close) there's no way we avoid Z thrash.

X-Plane traditionally solves this with two-pass rendering.  Because an airplane cockpit is sealed, we can draw the entire outside world in one coordinate space, blast the depth buffer, and then draw the interior cockpit with reset near/far clip planes.  The depth fragments of the cockpit are thus farther than parts of the scenery (in raw hardware depth buffer units) but the depth clear ensures clean ordering.

(This technique breaks down if something from the outside world needs to appear in the cockpit - we do some funny dances as the pilot exits the airplane and walks off around the airport to transition, and you can create rendering errors if you know what to look for and have nothing better to do.)

The Dumb Way

So when we first put in deferred rendering, we did the quickest thing that came to mind: two full deferred rendering passes.

(Driver writers at NVidia: please stop bashing your heads on your desks - we're trying to help you sell GPUs!  :-)

Suffice it to say, two full deferred passes was a bit of a show-stopper; deferred renderers tend to be bandwidth bound, and by consuming twice as much of it as a normal, sane game, we were destined to have half the framerates of what our users expected.

No Z Tricks

Unfortunately, I didn't find a non-linear Z buffer approach I liked.  Logarithmic Z in the vertex shader clips incorrectly, and any Z re-encoding in the fragment shader bypasses early-Z optimizations.  X-Plane has some meshes with significant over-draw so losing early Z isn't much fun.

Particle + HDR = Fail

There was a second performance problem that tied into the issue of bandwidth: X-Plane's cloud system is heavy on over-draw and really taxes fill rate and ROPs, and in the initial pipeline it went down into an HDR surface, costing 2x the memory bandwidth.  So we needed a solution that would put particle systems into an 8-bit surface if possible.

One Last Chainsaw

One last chainsaw to throw into the mix as we try to juggle them: our engine supports a "post-deferred" pass where alpha, lighting effects, particles, and other G-buffer-unfriendly stuff can live; these effects are forward rendered on top of the fully resolved deferred rendering.  We have these effects both outside of the airplane and inside the airplane!

Frankenstein is Born

The resulting pipeline goes something like this:

• We have a G-Buffer, HDR buffer, and LDR buffer all of the same size, all sharing a common Z buffer.  The G-Buffer stores depth in eye space in half-float meters, which means we can clear the depth buffer and not lose our G-Buffer resolve.
• Our interior and exterior coordinate systems are exactly the same except for the near/far clip planes of the projection matrix.  In particular, both the interior and exterior drawing phases are the same in eye space and world space.

1. We pre-fill our depth buffer with some simple parts of the cockpit, depth-only,with the depth range set to the near clip plane.  This is standard depth pre-fill for speed; because the particle systems in step 4 will be depth tested, this means we can pre-occlude a lot of cloud particles with our cockpit shell.
2. We render the outside solid world to the G-Buffer.
3. We draw the volumes of our volumetric heat blur effects, stencil only, to "remember" which pixels are actually exposed (because our depth buffer is going to get its ass kicked later).
4. We draw the blended post-gbuffer outside world into the LDR buffer, using correct alpha to get an "overlay" ready for later use.  (To do this, set the alpha blending to src_alpha,1-src_alpha,1,1-src_alpha.)  This drawing phase has to be early to get correct Z testing against the outside world, and has the side effect of getting our outside-world particles into an LDR surface for performance.
5. We draw our light billboards to our HDR buffer.
6. We clear the depth buffer and draw the inside-cockpit solid world over the G-Buffer.  We set stenciling to mark another bit ("inside the cockpit") in this pass.
7. We draw the heat markers again, using depth-fail to erase the stencil set, thus 'clipping out' heat blur around the solid interior.  This gets us a heat blur stencil mask for later that is correct for both depth buffers.  (Essentially we have used two stenciling paths to 'combine' two depth tests on two depth buffers that were never available at the same time.)
8.  We go back to our HDR buffer and blit a big black quad where the stencil marks us as "in-cockpit".  This masks out exterior light billboards from step 5 that should have been over-drawn by the solid cockpit (that went into the G-Buffer).  This could be done better with MRT, but would add a lot of complexity to already-complex configurable shaders.
9. We "mix down" our G-Buffer to our HDR buffer.  Since this is additive, light billboards add up the way we want, in linear space.
10. We draw another stenciled black quad on our LDR buffer to mask out the particles from step 4.
11. Finally, we render in-cockpit particles and lights directly into the LDR buffer.

Yeah.  I went through a lot of Scotch this last patch.

A few observations on the beast:

• That's a lot of MRT changes, which is by far the weakest aspect of the design.  We don't ever have to multi-pass over any surface except the depth buffer, but we're still jumping around a lot.
• The actual number of pixels filled is pretty tame.
• Night lighting is really sensitive to color space, and we picked up a few steps by insisting that we be in exactly the right color space at all times. Often the difference between a good and bad looking light is in the 0-5 range of 8-bit RGB values!  When lights are rendered to a layer and that layer is blended, we have to be blending in linear color space both when we draw our lights and when we composite the layer later!

In particular, there's one really weird bit of fine print: while spill lights accumulate in our HDR buffer linearly (which is a requirement for deferred lighting), pretty much every other blending equation in the deferred engine runs in sRGB space.  That's weird enough that it still surprises me, it makes everything way more complicated than it has to be, and I will describe why we need sRGB blending in the next post.

Deferred Lighting: Stenciling is not a Win

I've been meaning to write up a summary of the changes I made to X-Plane's deferred rendering pipeline for X-Plane 10.10, but each time I go to write up an epic mega-post, I lose steam and end up with another half-written draft, with no clue about what I meant to say.  So in the next few posts I'll try to cover the issues a little bit at a time.

One other note from the pipeline work we did: using the stencil buffer to reject screen space for deferred lights is not a win in X-Plane.

The technique is documented quite a bit in the deferred rendering powerpoints and PDFs.  Basically when drawing your deferred lights you:

1. Use real 3-d volumetric shapes like pyramids and cubes to bound the lights.
2. Use two-sided stenciling to mark only the area where there is content within the light volume.  A second pass over the volume then fills only this area.

The stenciling logic is exactly the same as stencil-shadow volumes, and the result is that only pixels that are within the light volume are lit; screen space both in front and behind the volume are both rejected.

For X-Plane, it's not worth it.  YMMV, but in the case of X-Plane, the cost of (1) using a lot more vertices per light volume and (2) doing two passes over the light volume far outweigh the saved screen space.

For a few very pathological cases, stenciling is a win, but I really found myself having to put the camera in ridiculous place with ridiculously lopsided rendering settings to see a stencil win, even on an older GPU. (I have a Radeon 4870 in my Mac - and if it's not a win there, it's not a win on a GeForce 680. :-)

The cost of volumes is even worse for dynamic lights - our car headlights all cast spill and the light volume transform is per-frame on the CPU.  Again, increasing vertex count isn't worth it.

For 10.10 we turned off the stencil optimization, cutting the vertex throughput of lights from two passes to one.

For a future version will probably switch from volumes to screen-space quads, for a nice big vertex-count win.

Finally, I have looked at using instancing to push light volumes/quads for dynamic objects.  In the case of our cars, we have a relatively small set of cars whose lights are transformed a large number of times.  We could cut eight vertices (two quads per car) down to a single 3x4 affine transform matrix.

Again, YMMV; X-Plane is a very geometry-heavy title with relatively stupid shaders.  If there's one lesson, it's this: it is a huge win to keep instrumentation code in place.  In our case, we had the option to toggle stenciling and view performance (and the effect on our stat counters at any time.

The Limits of 8-bit Normal Maps

It's safe to say that when one of the commenters points out something that will go wrong, it's only a matter of time before I find it myself. In this case the issue was running out of normal map precision and it was a matter of having an art asset sensitive to normal maps.

Well, our artists keep making newer and weirder art assets, and once again normal maps are problematic. In particular, when you really tighten up specular hilights, the angular precision per pixel of 8-bit normal maps makes it very difficult to create "small" effects without seeing the quantization of the map.

I made this graph to illustrate the problem:



So what is this? This equation (assuming I haven't screwed it up) shows the fall-off of specular light levels as a function of the "displacement" of the non-tangent channels of a normal map. Or more literally, for every pixel of red you add to move the normal map off to the right, how much less bright does the normal become?

In this case, our light source is hitting our surface dead on, we're in 8-bit, and I've ignored linear lighting (which would make the problems here worse in some cases, better in others). I've also ignored having specularity being "cranked" to HDR levels - since we do this in X-Plane the effects are probably 2x to 3x worse than shown. Units to the right is added dx and dy vectors, and each unit vertically is a loss of brightness value.

Three fall-off curves are shown based on exponents ^128, ^1024, and ^4096. (The steepest and thus most sensitive one is ^4096). You can think of your specular exponent as an "amplifier" that "zooms in" on the very top of the lighting curve, and thus amplifies errors.

So to read this: for the first minimal unit of offset we add to the normal map, we lose about two minimal units of brightness. In other words, even at the top of the curve, with an exponent of ^1024, specular hilights will have a "quantized" look, and a smooth ramp of color is not possible. It gets a lot worse - add that second unit of offset to the normal map and we lose eight units of color!

(By comparison, the more gentle 2^128 specular hilight) isn't as bad - we lose six units of brightness for five of offset, so subtle normal maps might not look too chewed up.)

This configuration could be worse - at least we have higher precision near zero offset. With tangent space normal maps, large areas of near constant normals tend to be not perturbed very much - because if there is a sustained area of the perceived surface being "perpendicular" to the actual 3-d mesh, the author probably should have built the feature in 3-d. (At least, this is true for an engine like X-Plane that doesn't have displacement mapping.)

What can we do?

• Use some form of normal map compression that takes advantage of the full bit-space of RGB.
• Throw more bits at the problem, e.g. use RG16. (This isn't much fun if you're using the A and B channels for other effects that only need 8 bits.)
• Use the blue channel as an exponent (effectively turning the normal map into some kind of freaky 8.8 floating point). This is something we're looking at now, so I'll have to post back as to whether it helps. The idea is that we can "recycle" the dynamic range of the RG channels when close to dark using the blue channel as a scalar. This does not provide good normal accuracy for highly perturbed normals; the assumption above is that really good precision is needed most with the least offset.
• Put some kind of global gamma curve on the RG channels. This would intentionally make highly perturbed normals worse to get better results at low perturbations. (I think we're unlikely to productize this, but it might in some cases provide better results using only 16 bits.)
  
• Tell our artists "don't do that". (They never like hearing that answer.)

I'll try to post some pictures once I am further along with this.

So Many AA Techniques, So Little Time

This is a short summary of FSAA techniques, both for the art team, and so I don't forget what I've read when I come back to this in 9 months. (No promise on accuracy here, these are short summaries, often with a bit of hand-waving, and some of the newer post-processing techniques are only out in paper form now.)

Where does aliasing come from? It comes from decisions that are made "per-pixel", in particular (1) whether a pixel is inside or outside a triangle and (2) whether a pixel meets or fails the alpha test.

Texture filtering will not alias if the texture is mip-mapped; since the texel is pulled out by going "back" from a screen pixel to the texture, as long as we have mip-mapping, we get smooth linear interpolation. (See Texture AA below.)

Universal Techniques

Super-Sampled Anti-Aliasing (SSAA). The oldest trick in the book - I list it as universal because you can use it pretty much anywhere: forward or deferred rendering, it also anti-aliases alpha cutouts, and it gives you better texture sampling at high anisotropy too. Basically, you render the image at a higher resolution and down-sample with a filter when done. Sharp edges become anti-aliased as they are down-sized.

Of course, there's a reason why people don't use SSAA: it costs a fortune. Whatever your fill rate bill, it's 4x for even minimal SSAA.

Hardware FSAA Techniques

These techniques cover the entire frame-buffer and are implemented in hardware. You just ask the driver for them and go home happy - easy!

Multi-Sampled Anti-Aliasing (MSAA). This is what you typically have in hardware on a modern graphics card. The graphics card renders to a surface that is larger than the final image, but in shading each "cluster" of samples (that will end up in a single pixel on the final screen) the pixel shader is run only once. We save a ton of fill rate, but we still burn memory bandwidth.

This technique does not anti-alias any effects coming out of the shader, because the shader runs at 1x, so alpha cutouts are jagged. This is the most common way to run a forward-rendering game. MSAA does not work for a deferred renderer because lighting decisions are made after the MSAA is "resolved" (down-sized) to its final image size.

Coverage Sample Anti-Aliasing (CSAA). A further optimization on MSAA from NVidia. Besides running the shader at 1x and the framebuffer at 4x, the GPU's rasterizer is run at 16x. So while the depth buffer produces better anti-aliasing, the intermediate shades of blending produced are even better.

2-d Techniques

The above techniques can be thought of as "3-d" because (1) they all play nicely with the depth buffer, allowing hidden surface removal and (2) they all run during rasterization, so the smoothing is correctly done between different parts of a 3-d model. But if we don't need the depth buffer to work, we have other options.

Antialiased Primitives. You can ask OpenGL to anti-alias your primitives as you draw them; the only problem is that it doesn't work. Real anti-aliased primitives aren't required by the spec, and modern hardware doesn't support them.

Texture Anti-Aliasing. You can create the appearance of an anti-aliased edge by using a textured quad and buffering your texture with at least one pixel of transparent alpha. The sampling back into your texture from the screen is done at sub-pixel resolution and is blended bilinearly; the result will be that the 'apparent' edge of your rendering (e.g. where inside your quad the opaque -> alpha edge appears) will look anti-aliased. Note that you must be alpha blending, not alpha testing.

If you're working in 2-d I strongly recommend this technique; this is how a lot of X-Plane's instruments work. It's cheap, it's fast, the anti-aliasing is the highest quality you'll see, and it works on all hardware. Of course, the limit is that this isn't compatible with the Z buffer. If you haven't designed for this solution a retro-fit could be expensive.

Post-Processing Techniques

There are a few techniques that attempt to fix aliasing as a post-processing step. These techniques don't depend on what was drawn - they just "work". The disadvantages of these techniques are the processing time to run the filter iself (e.g. they can be quite complex and expensive) and (because they don't use any of the real primitive rendering information) the anti-aliasing can be a bit of a loose cannon.

Morphological Anti-Aliasing (MLAA) and Fast Approximate Anti-Aliasing (FXAA). These techniques analyze the image after rendering and attempt to identify and blur out stair-stepped patterns. ATI is providing an MLAA post-process as a driver option, which is interesting because it moves us back to the traditional game ecosystem where full screen anti-aliasing just works without developer input.

Edit: See also Directionally Localized Anti-Aliasing (DLAA).

(From a hardware standpoint, full screen anti-aliasing burns GPU cycles and sells more expensive cards, so ATI and NVidia don't want gamers to not have the option of FSAA. But most new games are deferred now, making MSAA useless. By putting MLAA in the driver, ATI gets back to burning GPU to improve quality, even if individual game developers don't write their own post-processing shader.)

It is not clear to me what the difference is between MLAA and FXAA - I haven't taken the time to look at both algorithms in detail. They appear to be similar in general approach at least.

Temporal Anti-Aliasing (TAA). This is a post process filter that blends the frame with the previous frame. Rather than have more samples on the screen (e.g. a 2x bigger screen in all dimensions for SSAA) we use the past frame as a second set of samples. The camera is moved less than one pixel between frames to ensure that we get different samples between frames. When blending pixels, we look for major movement and try to avoid blending with a sample that wasn't based on the same object. (In other words, if the camera moves quickly, we don't want ghosting.)

Deferred-Rendering Techniques

This set of techniques are post-processing filters that specifically use the 3-d information saved in the G-Buffer of a deferred renderer. The idea is that with a G-Buffer we can do a better job of deciding when to resample/blur.

Edge Detection and Blur. These techniques locate the edge of polygons by looking for discontinuities in the depth or normal vector of a scene, and then blur those pixels a bit to soften jaggies. This is one of the older techniques for anti-aliasing a deferred renderer - I first read about it in GPU Gems 2. The main advantage is that this technique is dirt cheap.

Sub-pixel Reconstruction Anti-Aliasing (SRAA). This new technique (published by NVidia) uses an MSAA G-Buffer to reconstruct coverage information. The G-Buffer is MSAA; you resolve it and then do a deferred pass at 1x (saving lighting) but then go back to the original 4x MSAA G-Buffer to edge detect.

G-Buffer Normals, Revisited

A while ago I posted the G-buffer format for X-Plane 10, which, as of this writing is still in development. SebH brought up CryTek's normal map compression and I hand-waived a bit and wondered to myself whether some kind of normal map goblin was going to pop up later in the development cycle.

The short answer: yes.

I will try to write up a post later describing the precision problems with normal maps in more detail, but for now I'll post the problem and its partial solution, while I still have the debug code in my shaders.

This is a Baron 58 that Tom Kyler is working on for version 10. He is probably not very happy that I'm posting pictures of it, because it's still in progress, and while I think it looks pretty good, our art guys get a lot of, um, "artsy goodness" into the models in the last few passes. (If the lighting seems a little, um, bizarre, it probably is; lord knows what state of debug the sun shader was in when I took these pics.)

The left side image is the airplane, lit by an evening sun that has just barely set, directly behind us, the right image is the fully reconstructed per pixel eye space normals. The small icons show the rough contents of the four layers of our G-Buffer.

So far things seem reasonably sane - the engine nacelle is lit from the side but not the top. But here's where things go south:

This is a wing with a light on the leading edge. The surface normal of the wing is almost perpendicular to the light direction, which really stress-tests the quality of our normal vectors. The first picture is the 'classic' G-Buffer technique: 16-bit-float dx and dy eye-space vectors, with Z reconstructed in shader. As you can see, it develops banding at the very low edge of angle-based attenuation. (Note that this area would be super-dark if we weren't in linear space.) The second image shows the full XYZ normal (burning an extra G-Buffer 16-bit channel)...clearly this fixes the problem of reconstruction from low-precision sources, but channels are hard to come by in gbuffers.

Fortunately I found this totally awesome write-up of different normal compression schemes. The above picture on the right is a Lambert Azimuthal Equal-Area Projection using two channels.

Here are a few more pics of the gbuffer normal map, both projected and expanded:


Side benefit: Lambertian projection copes with negative eye space Z (but not 0,0,-1, which is unlikely even with tangent space normal maps on art assets) so no more hand-waving there.

One last thought for now: this entire post refers to the 'normal map' layer of a g-buffer, that is, the saved per-pixel normal information. Compression of 'normal map' textures for art assets is a bit of a different problem - the most immediate note is that they can be compressed off-line, so non-realtime compression techniques are fair game.

Derivatives III: I Ran Out of Rez

One more note on derivatives in GLSL shaders: derivatives can run into precision problems that the underlying expressions don't have.

Recall that derivatives are typically calculated by taking the actual difference of two values in two nearby pixels. This means that the derivative is subject to the precision limits between two pixels.

Consider a UV map spread over a really huge distance, say, 5 km in-game. The texture is 1024 x 1024 and therefore each texel is about 10m in size.

What happens if we zoom way the heck in so that one game pixel (5m) is covering nearly all of the screen?

We need about 10 bits of precision to select our texel; any remaining precision can be used to take an interpolated position between texels for filtering. If our monitor res is about 1024x768, we're using 20 bits of precision in our UV map, and we have only three bits left. If we zoom in any more, we may reach a point where our interpolated UV map doesn't have enough precision to provide a UV position for each pixel.

(In other words, if we have less than 10 bits of precision left between the left and right side of the screen, then some adjacent pixels will have the same UV coordinates!)

Now this generally doesn't matter for texture sampling. We're sampling 1024 unique mixes between two pixels, and we can only show 256 shades on the screen - in practice if two pixels have the same UV coordinates, it doesn't matter, because the amount of RGB change per pixel is not perceivable anyway.

But for our derivatives, it's a different story: some pairs of pixels will have a zero derivative, and some will have a non-zero derivative! Even if we don't run out res, we're very low on res and our derivatives may be 'chunky' or in other ways screwed up. If we need to reconstruct basis vectors from our derivatives, those basis vectors are going to be a train wreck.

The only solution I have found to this problem has been to replace GLSL built-in derivatives with an algorithmic derivative function. Fortunately the only cases where we have ridiculous UV mapping is when the texture coordinates are generated by formula, and thus a similar formula can be used to create the derivatives.

Derivatives II: Conditional Texture Fetches

In my previous post I described how OpenGL often calculates derivatives by differencing nearby pixels in a block. This can cause problems if our UV map has discontinuities.

Even weirder things happe if we use texture fetches inside an if statement. For example, this will produce some very weird results:

if(uv.x >= 0.0)
gl_FragColor = texture2D(my_sampler,uv);
else
gl_FragColor = vec4(0.0);

You might think that if you use a texture a ramp of black on left, white on right, you'd get a ramp of texture and then the black texture would seamlessly transition into the hard-coded black from the else statement.

If your GPU and GLSL compiler are in a forgiving mood, this may work; if they are not, you may get a set of mid-gray artifact pixels at the transition point. The problem is this bit of fine print (from the GLSL 1.20.8 spec, section 8.8):

The method may assume that the function evaluated is continuous. Therefore derivatives within the body of a non-uniform conditional are undefined.

You can't take a derivative inside an if statement. (But since the results are undefined, the GPU can make your life more difficult by sometimes giving you useful results anyway. ) Recall from my past post that a texure2D fetch is like a texture2DGrad with derivatives of the texture coordinate expression. Since the derivative functions are invalid inside if statements, the derivatives passed to texture2D may be junk. In other words, this is bad:

if(stuff)
gl_FragColor = texture2D(tex,uv,dFdx(uv),dFdy(uv));

but this is okay:

float dx = dFdx(uv);
float dy = dFdy(uv);
if(stuff)
gl_FragColor = texture2DGrad(tex,uv,dx,dy);

In other words, you have to use texture2DGrad to move the derivative calculation out of the if statement.

Why Can't the GPU Get This Right (Except When It Does)

Artifacts due to incorrect derivative calculations inside incoherent texture fetches (that is, some pixels texture fetch, nearby ones don't, the derivative is hosed, and our texture fetch is therefore hosed) are definitely sensitive to the hardware, GLSL compiler, and driver, and I ended up switching out my Radeon and GeForce about 30 times before I wrapped my head around this issue.

This doesn't surprise me. The spec allows undefined behavior. Recall that the derivative is based on differencing the value of an expression across a 2x2 pixel group. To understand why conditionals and derivatives don't mix, we have to understand how modern GPUs handle conditional rasterization.

(What follows is based on my reading some docs on R700 assembly; it is best to think of it as a model for how GPUs can work, more or less; I am sure there are lots of subtleties to the R700 that I don't understand.)

The GPU rasterizes pixels in 2x2 blocks, with the same shader executed on four execution units in lock-step. That is, each pixel has its own intermediate registers and state, but all four pixels run the same instructions.

When the shader hits an if statement, the hardware sets a mask for each pixel indicating which pixels are "in" the if statement and which are not. The entire if statement is run on all hardware, but the results for the pixels that are not in the if statement are thrown out due to the mask.

If all four pixels hit the if statement the same way, only then can the GPU jump over the if statement, saving actual work.

So what happens if the if statement is being evaluated for some pixels and not others and we take a derivative? The answer is: lord knows! The expression we are calculating may be only partly updated, incorrect, or totally unavailable for some of the pixels.

Branch Coherence

As a side note, the property of the GPU to run the entire shader on all pixels when only some of them are using the if statement is why the GPU manufacturers will tell you that a conditional is only a performance win if it is coherent - that is, if nearby pixels all branch in the same way. This is because when nearby pixels branch in different ways, the GPU must run all code and throw out some of the results.

Derivatives I: Discontinuities and Gradients

The short of it is this: if you see 2x2 pixel artifacts in your shader, you might need texture2DGrad. Now the long version.

How does OpenGL know what mipmap level to use when you sample a texture in your GLSL shader with texture2D? The answer is that this:

texture2D(my_texture,uv);

actually does something like this:

texture2DGrad(my_texture,uv,dFdx(uv),dFdy(uv));

In other words, texture2D takes the derivative of your input texture coordinates and uses those derivatives to decide which mipmap level to access. The larger the derivatives, the lower mipmap level. (The actual implementation is more complicated.)

Before continuing, a brief exercise in visualization. Imagine a cube with a single square face visible to us (parallel to the screen). The cube face is textured with a single 256x256 texture. If we zoom the camera so that the cube takes 256x256 screen pixesl, the derivative of the UV map between any two pixels on screen is about 1/256 in both directions, and we want the highest level mipmap. If we zoom out so that the cube takes up only 2x2 pixels, the derivative is about 1.0 in both directions - and we want the lowest mipmap level.

Where Do Derivatives Come From?

The GLSL derivative functions are usually implemented by differencing - that is, the GPU takes a block of 2x2 pixels and differences the variable or expression passed to dFdx and dFdy, to calculate an 'approximate' derivative. Many GPUs rasterize 2x2 clusters of pixels at a time, with the shader instructions for the four pixels run in lock-step, so the hardware can be set up to efficiently "cross" the four texels to find our derivatives.

This means that if there is a discontinuity between those pixels, the derivative may be, well, surprising. For example, consider something like this:

vec2 uv = gl_TexCoord[0].st;
if(uv.x > 0.5) uv.y += 0.25;
gl_FragColor = texture2D(my_sampler, uv);

What happens if two of the pixels in our 2x2 block have uv.x > 0.5 and the other two don't? well, the answer is that uv.y will be 0.25 bigger for some but not all textures, and the derivative of uv.y will be very big! This in turn will cause texture2D to fetch a low mipmap level, much lower than any other 2x2 pixels that are "coherent". (Coherent here means all 4 pixels have the same boolean answer to the if conditional.)

One way to think of this is: since the derivatives are found by looking at actual pixels on screen, a discontinuity is seen by the derivative function as a really low-res UV map, and thus a low mipmap level is selected.

Fixing The Derivative

So what can we do? We can provide OpenGL with an expression whose derivative is about the same as our real texture coordinates, but without discontinuities. For example, we can rewrite our above example like this:

vec2 uv = gl_TexCoord[0].st;
if(uv.x > 0.5) uv.y += 0.25;
gl_FragColor = texture2DGrad(my_sampler, uv,dFdx(gl_TexCoord[0].st),dFdy(gl_TexCoord[0].st));

Our actual texture samples come from a discontinuous UV map, but our derivative comes from the original continuous function.

Breaking Continuity

I first ran across this while working on the 'tile' shader for X-Plane 10. The tile shader breaks each texture into a sub-grid of tiles and then randomly swizzles the tiles, like a number puzzle that someone has been scrambled. The tile shader hides repetitions in the shader, and (because it runs in shader) it doesn't require additionally tessellating geometry, saving vertex count.

(Using fragment ops to save vertex count might seem strange, but in this case, our base mesh is already heavily cut up based on other criteria; having the texture swizzle run orthogonally lets us subdivide the mesh based on other, unrelated criteria.)

Without texture2DGrad, we would get a set of 2x2 pixel dark pixels at the edge of the tiles. The tiles are induced via some math that includes a floor() function to separate our tile number from our location within the tile. The floor function can induce discontinuities even without conditional logic, because floor is not a continuous function.

Fun With GLSL Compilers

I've been poking at GPU ShaderAnalyzer; this Windows performance tool from ATI gives you a simple environment: you enter GLSL, hit compile, and look at the assembly that pops out. Perfect! (It also shows you execution cycles, but for my purposes, namely understanding what the compiler does for me, the assembly is key.)

Here are some things I have learned:

• RV790 assembly is quite complex. (Thank goodness I don't have to code it myself.) ALU instructions consist of 5 scalar sub-instructions, only one of which can have transcendental opcodes. There's a bit of fine print; this and this make useful reading. One thing to note: the ALU has a number of 'small' tricks (absolute value, negation, clamping) 'for free'. Sometimes the compiler will use these tricks, sometimes not.

  Generally, if you write vectorized code (e.g. uniform work on vec4) the scheduling will work out nicely. But the units of execution really are scaler, so it doesn't make sense to write work that isn't needed.

• The compiler inlines pretty much everything, which is just fine by me. (I have no idea if recursion is legal in GLSL, I'd never use it in production, but when I wrote a recursive factorial function, the compiler simply inlined 127 iterations and called it a day. Awesome!)

• The compiler understands a reasonable amount of constant folding, including (most importantly) multiply by zero. For example: I write an expensive albedo function: pow(gl_Color,gl_Color.aaaa) and multiply it by a light function that returns vec4(0.0). The result: the compiler nukes the entire code sequence and simply loads 0.

  (BTW, pow is expensive - since only one of the five ALU slots can run log and exp, raising each color channel to a non-constant power takes eight instruction groups! Ouch.)

• The compiler will remove conditional code when the condition is fully known at compile time. So for example, an if statement where the comparison comes from functions that return constants will be nuked, and one of its two clauses is deleted.

• The compiler does not seem to do inference, at least in the one case I looked at. By inference I mean: if (max(0.6,gl_FragColor.r) > 0.3) will (ignoring NaN logic) always be true, regardless of gl_FragColor. But for the compiler to know this, it has to make an inference - that is, it has to compare the range [0.6..inf) with 0.3. My understanding is that LLVM can do this kind of thing, but when I tried it in shader I simply got the full, expensive, conditional code. Moral of the story: use and apply your human brain. :-)

Now...why do I care?

X-Plane's physical shader is based on conditional compilation - that is, for any given state vector of "tricks" we want to use, we recompile the shader with some #defines at the front which turn features on and off. The result is a large number of shaders, none of which need conditional logic in-shader. Fill rate isn't consumed by features we don't use. (This technique comes from our original use of GLSL to emulate and then improve on the fixed function pipeline. To match fixed-function performance, we had to 'compile out' anything we didn't use, particularly for first-gen DX9 hardware which doesn't give you conditional logic for free.

The problem with this technique (and you can see this in the X-Plane 9 shaders) is that it doesn't scale well with code size. For version 10 we've done a lot of shader work, and hand-optimizing the conditional logic is getting more and more difficult.

My conclusion from observing the compiler is that 99% of the time, I can relax a little bit and let the compiler take care of optimizing the shaders down. In particular, if I define functions for each stage of the shader and use conditional compilation to 'simplify' the rule, then the simple cases will boil down to very few instructions. For example:

float calc_spec()
{
#if has_spec
return pow(max(0.0,dot(eye_nrm,sun-vec)),128.0);
#else
return 0.0;
#endif
}
void main()
{
....
float s = calc_spec();
gl_FragColor = albedo * lighting * shadow + ambient + shadow * vec4(s,s,s,0.0);
}

In this mess, our specularity function is subject to conditional removal for non-shiny materials. When we do this, not only is the actual specularity calc removed, but the compiler will figure out that 's' will alwys be 0.0 and nuke the MAD of shadow * specularity into the final lighting sum.

That's a trivial example, but it shows the principle of structuring the components separately and letting the compiler put the mess together.

As a final note: the compiler's optimization is not perfect; I suspect the above technique will 'leak' a few instructions in the simple cases relative to a one-off carefully hand-coded GLSL shader, and the GLSL isn't going to be quite as tight in a few cases as actually writing assembly.

But I can live with that, most of the time. We can always go and hand tune the performance cases that absolutely matter most, and the time saved working on the huge mess that is the conditional shader gives me the time to do that hand optimization where it is most important.

FMTT, GLSL Edition

This is in the same vein as I Hate C -- all of its derivatives are contaminated with its brain damage.

gl_FragData[0] = vec4(tex_color.rgb * gl_Color.rgb*tex_color.a,clamp(tex_color.a + lit_color.a,0.0,1.0));
gl_FragData[1] = vec4(shiny_ao * cut_pos, cut_pos*position_eye.z/-1024.0, 0.0, cut_pos);
gl_FragData[2] = vec4(normal_eye_use.xyz*cut_pos, cut_pos);
gl_FragData[3] = vec4(lit_color.rgb + tex_color.rgb * gl_FrontLightModelProduct.sceneColor.rgb, (tex_color.a + lit_color.a,0.0,1.0));

Yet Another This-Is-Our-GBuffer-Format Post

If you read just about any presentation by a game studio (E.g. for GDC or Sigraph) on deferred rendering or deferred lighting, they'll probably discuss their G-Buffer format and how they tried to pack as much information into a tiny space as possible. X-Plane 10 will feature a deferred renderer (to back the global spill feature set). And...here is how we pack our G-Buffer.

The good news with X-Plane is that we don't have a complex legacy material system to support. If a game is well-batched, the forward renderer can associate separate shaders with batches by material, and each 'material' can thus be radically different in how it handles/transfers light. With a G-Buffer, the lighting equation must be unified and thus everything we need to know to support some common lighting format must go into the G-Buffer. (One thing you can do is pack a material index into the G-Buffer.) Fortuantely, since X-Plane doesn't have such a multi-shader beast we only had one property to save: a shininess ratio (0-1, about 8 bits of precision needed).

What we did have to support was both a full albedo and a full RGB emissive texture; additive emissive textures have been in the sim for over a decade, and authors use them heavily. (They can be modulated by datarefs within the sim, which makes them useful for animating and modulating baked lighting effects.) X-Plane 10 also has a static shadow/baked ambient occlusion type term on some of the new scenery assets that needs to be preserved, also with 8 bits of precision.

The other challenge is that X-Plane authors use alpha translucency fairly heavily; deferred renderers don't do this very well. One solution we have for airplanes is to pull out some translucent surfaces and render them post-deferred-renderer as a forward over-draw pass. But for surfaces that can't be pulled out, we need to do the least-bad thing.

The Format

Thus we pack our G-Buffer into 16 bytes:

1. RGBA8 albedo (with alpha)
2. RG16F normal (Z vector is reconstructed)
3. RG16F depth + shadow-merged-with-shininess
4. RGBA8 emissive (with alpha)

When drawing into the G-Buffer, we use full blending for the albedo and emissive layers, but we always set the alpha for the depth/normal layers to 0.0 or 1.0, thus using the alpha blend as a per-render-target "kill" switch. This is based on the level of translucency. Thus if something is highly transparent, we keep the physical position of what is behind it (light passes through) but if it is opaque enough, we over-write the position/normal (light bounces off of it). It's not perfect, but it's the least bad thing I could come up with.

(As a side note, if we had a different layout, we could blend the shininess ratio, for example, when we keep a physical position fragment, to try to limit shininess on translucent elements.)

Note that on OS X 10.5 red-green textures are not available, so we have to fall back to four RGBA_16F textures, doubling VRAM. This costs us at least 20% fill-rate on an 8800, but the quick sanity test I did wasn't very abusive; heavier cases are probably a bit worse.

So far we seem to be surviving with a 16-bit floating point eye-space depth coordinate. It does not hold up very well in a planet-wide render though, for rendering techniques where reconstructing the position is important (e.g. O'Neil-style atmospheric scattering). A simple workaround would be to calculate the intersection of the fragment ray with the planet directly by doing a transform of the planet sphere from model-view space. (E.g. if we know that our fragment came from a sphere, why not just work with the original mathematical sphere.)

16F depth does give us reasonably accurate shadows, at least up close, and far away the shadows are going to be limited in quality anyway. I tried logarithmic depth, but it made shadows worse.

Packing Shadowing and Shine

For the static shadow/AO ("shadow") term and the level of specularity ("shine") we have two parameters that need about 8 bits of precision, and we have a 16-bit channel. Perfect, right? Well, not so much.

• NVidia cards won't render to 16-bit integer channels on some platforms.
• ATI cards don't export a bit-cast from integer to floating point, making it hard to pack a real 16-bit int (or two 8-bit ints) into a float.
• If we change the channel to RGBA8 (and combine RG into a virtual 16F) we can't actually use the fourth byte (alpha) because the GL insists on dumping our alpha values. Extended blend would fix this but it's not supported on OS X and even on Windows you can't use it with multiple render targets.

So we can't actually get the bits we pay for and that sucks. But we can cheat. The trick is: the deferred renderer will cut out specular hilights that are in shadow. Thus as the shadow term becomes stronger, the shine term becomes unimportant.

So we simply encode 256.0 * shadow + shine. The resulting quantity gives shininess over 8 bits of precision without shadow, and reduces shininess to around 2 bits in full shadow. If you view the decoded channels separately you can see banding artifacts come in on the shine channel as the shadows kick in. But when you view the two together in the final shader, the artifacts aren't visible at all because the shadow masks out the banded shininess.

(What this trick has done is effectively recycle the exponent as a 'mixer', allocating bits between the two channels.)

Future Formats

A future extension would be to add a fifth 4-byte target. This would give us room to extend to full 32-bit floating point depth (should that prove to be useful), with shadow and shine in 8 bit channels with one new 8-bit channel left. Or alternatively we could:

• Keep Z at 16F and keep an extra channel.
• Technically if we can accept this 'lossy' packing we can get four components into an RG16F, while we can only get 3 pure 8-bit components. (This is due to how the GL manages alpha.)
• If we target hardware that provides float-to-int bit casts, we could have four 8-bit components in the new channel.

How To Change Your UV Map on the Fly

I've been playing with "stupid UV map tricks" lately - the basic idea is to (in the fragment shader) change the texture coordinates before fetch. For example, given a texture divided into equally useful grid squares, we can on a per-grid square basis change which square we're in, to make the texture repetition less obvious. Why do this?

• You can make your textures look less repetitive without making meshes more complex.
• Since the effect is in-shader, it can be turned off on lower end machines - scalability!

But there's this one bit of fine print, and it escaped me for about four months: if you want to "swizzle" the UV map in a discontinuous way (e.g. using "fract", "mod", etc.) you need to use the explicit gradient texture fetch functions! If you don't, you get artifacts at the discontinuities.

Huh?!?!

In order to understand why this is necessary, you first have to understand how the hardware selects a mipmap level, and to understand that you have to understand how OpenGL generates derivatives.

First the derivatives. Most of the video cards I know about generate derivatives of a shader variable by "cross-differencing" - that is, a 2x2 block of pixels is run using the same shader, and when the shader hardware gets to the derivative (dFdx, and dYdx) it simply subtracts the interim values from the four pixels to find how much they "change" in the box. In other words, the derivative function in GLSL works by discreet per-pixel sampling.

(BTW this is why when you screw up code that needs to treat derivatives carefully, often you'll get 2x2 pixel artifacts.)

These derivatives allow the graphics card to select a LOD. At the sight of a texture fetch, the card can do a derivative operation on the input texture coordinates and see how fast they change per pixel. The faster they change, the lower the effective texture res and the lower LOD mip-map we need. That is how the card "knows" to use the lower mip-maps even when you use expressions for your texture coordinates - the derivative is taken on the entire expression.

But...what happens when you have a discontinuity in your UV map? Take a simple case like "fract". If you "fract" a wrapping texture, you will quite possibly see an artifact at the edges. This is because, right at the edge, the rate of change of the UV map is much higher than before, as it "jumps" from one edge of the texture to the other. High rate of change = low LOD - the graphics card goes and selects the lowest level LOD it has!

(If you don't know what's in your lowest mip, you might not know where the color was coming from.)

The solution is here: texture2DGradARB. This function lets you separately specify the texture coordinates and the derivatives. Here's a simple example. Imagine you have this:

vec2 uv_swizzled = fract(uv);
vec4 rgba = texture2D(my_tex, uv_swizzled);

That example will create a few pixels of low-mipmap texture at the discontinuity (where the texture goes from 1 back to 0). To use texture2DGradARB, you do this:

vec2 uv_swizzled = fract(uv);
vec4 rgba = texture2DGradARB(my_tex,uv_swizzled,dFdx(uv),dFdy(uv));

By using the original (continuous) texture coordinates for the derivative, but the modified ones for the fetch, you can have discontinuous fetches with no LOD artifacts.

NVidia and ATI cards don't respond the same way to discontinuous coordinates, but both will produce artifacts, and both are right to do so.

One last note. From the shader texture LOD extension:

   Mipmap texture fetches and anisotropic texture fetches
  require an implicit derivatives to calculate rho, lambda
  and/or the line of anisotropy.  These implicit derivatives
  will be undefined for texture fetches occuring inside
  non-uniform control flow or for vertex shader texture
  fetches, resulting in undefined texels.

I can tell you from experience that a number of my artifacts have come from conditional code flow. I believe that by non-uniform control flow they mean the case where the shader branches are not all taken the same way for a 2x2 block, but I am not sure.

Running Out of Derivative Res

In a previous post I went over the math behind generating the coordinate system for normal mapping in a pixel shader, which allows you to use tangent space bump mapping without encoding coordinate axes on your vertex mesh. (In X-Plane we do this so that we can allow authors to add bump maps to "unmodified" meshes.)

One of the problems with writing shaders is that it can be write-once, debug everywhere. As it turns out, this technique has a problem that I can repro on a GF8800 but not HD4870. On the 8800, I run out of precision in my derivative (dFdx and dFdy) functions.

In the scene in question, the UV map is generated in the vertex shader via projection off the world-space input vertices and the input mesh is big - 300 x 300 km in fact. (It is of course the base terrain.)

This means that the UV coordinates are pretty big too, particularly for highly scaled up textures. And that means that the effective resolution limit of the texture coordinates may be larger than one pixel.

When this happens, the result is a derivative that will be inconsistent across pixels, and the basis for the bump map will be corrupted on a per-pixel level.

Work-arounds? I can think of two:

1. Modify the texture coordinate generation system to produce higher precision UV maps.
2. Modify the shader to generate basis vectors from the projection parameters (rather than by "sampling" via the UV map) in the texture coordinate generation case.
   

A Tile Too Far

I've been playing with shading algorithms lately. One such algorithm is the "number puzzle". The basic idea is to take a repeating texture that is divided into sub-tiles and randomly move the tiles around. This is implemented in-shader by separating the UV coordinates and randomizing the bits that represent the "tile". (This is usually all but the lowest N bits.) The tile choice is made by sampling a random noise map, and the UV input to that comes from the upper bits so that it is stable (e.g. so we only switch tiles at the tile boundary).

One nice property of the number puzzle is that if you don't have shaders, you simply get a repeating texture. This is handy because the art assets and code doesn't have to be cased out for a fixed function case - we end up with uglier, but valid output.

It occurred to me today that the number puzzle can be atlased - that is, the random tile we pick could be constrained by the upper bits of the UV map, so that (by using a broad "space" of UV coordinates) we can pick from a set of tiles within a larger texture. This is a win because it means we can texture atlas and thus merge a bunch of differently tiled surfaces into one batch.

There is just one problem with this technique, one that might be a deal breaker as long as fixed function is necessary: when the shader is off, the atlasing gets ignored and we end up with junk. There really isn't a good way around this..wrapping + atlasing are, as far as I know, incompatible in the fixed function pipeline.

Per-Pixel Tangent Space Normal Mapping

This post assumes you know what tangent space normal maps are in general, and roughly how they work geometrically. If you try to code this, you'll hit one tricky problem: how do you convert your normal map from tangent to eye space? You need the normals in eye space to do lighting calculations.

The short answer is: a few lines of GLSL goo do the trick. Before explaining why those lines do what they do, we need to understand basis vectors.

Basis Vectors

Basically you can convert a vector from a source to a destination coordinate system like this:

new_ = [ dot(old, X), dot(old, Y), dot (old, Z) ]

where X, Y and Z are basis vectors - that is, X,Y, and Z are the axes of the old coordinate system expressed in the units of the new coordinate system. If you know where the old coordinate system is in terms of the new coordinate system, you can convert.

(A side note: the basis vectors form a matrix that performs this transformation, e.g.

X_x Y_x Z_x
[ X_y Y_y Z_y ]
X_z Y_z Z_z

What does X_y mean? That is the y component of the X axis of the old coordinate system. Because the coordinate systems are not aligned, the old X axis may be a "mix" of the new coordinate system.)

I strongly recommend taking the time to deeply understand basis vectors. Once you understand them, OpenGL matrices start looking like a geometric shape instead of a blob of 16 random numbers.

So if we want to convert our normal map from tangent space to eye space, we need basis vectors - that is, we need to know where the S, T and N axes are in terms of eye space.

(A side notation note: typically these are known as the Tangent, Bitangent and Normal vectors for a TBN matrix. When I say "S" and "T" I mean the direction on your model that matches the horizontal axis of your texture and vertical axis of your texture.)

We already know "N" - it's our normal vector, and we usually have that per vertex. Where do we get S & T?

The Derivative Functions

GLSL provides derivative functions dFdx and dFdy. These functions return the change in value of an expression over one pixel on the screen horizontally and vertically. You can pass anything into them. If you pass in a texture coordinate, you can find out how much your UV texture will change as you move right or up one pixel. If you pass in a location, you can find out how much your location will change per pixel. If we pass in a vector, the "derivative" is performed on each component.

My understanding is that on some hardware, pixel shaders are run on a block of 2x2 pixels. The same instruction is run 4 times for 4 sets of input data. When the shader hits the "derivative" function, it simply takes the difference between the emerging values in each of the blocks to find the deltas.

The key points about the derivative functions: we can apply them to anything, and they take derivatives along the X and Y axis in screen space.

Solving Parallel Equations

From our discussion of basis vectors we know that we the S and T vectors basically do this:

dx = ds * Sx + dt * Tx
dy = ds * Sy + dt * Ty
dz = ds * sZ + dt * Tz

In other words, they map from UV coordinates to eye space.

Don't we need three basis vectors? Well, yes. The third component would be dn * Nx, etc. But our UV map is effectively flat - that is, it's "N" coordinate is always 0. So we will drop out the third basis for now. (This should be expected - the normal vector is defined as tangent to our mesh, and the UV map is entirely on our mesh.)

So...we have 3 equations of 2 unknowns. For each, if we only had 2 sets of values (e.g. 2 sets of dx, ds, dt) could solve for our basis vectors.

Well we can! If we take the X derivative (dFdx) of our texture coordinates and eye space mesh location, we would get: Q.x (the change of eye space position) ST.s (the change of the S coordinate of our UV map ) and ST.t (the change of our T coordinate in the UV map). And we could do this twice, using dFdx and dYdx to get two separate sets of points.

vec3 q0 = dFdx(position_eye.xyz);
vec3 q1 = dFdy(position_eye.xyz);
vec2 st0 = dFdx(gl_TexCoord[0].st);
vec2 st1 = dFdy(gl_TexCoord[0].st);

Now we have a set of 6 constants to plug in to solve our two unknowns:

dx = ds * Sx + dt * Tx

With the substitution (q0 vs q1 is our derivatives from dFdx and dFdy, etc.):

q0.x = st0.s * Sx + st0.t * Tx
q1.x = st1.s * Sx + st1.t * Tx

We can solve this to come up with:

Sx = ( q0.x * st1.t - q1.x * st0.t) / (st1.t * st0.s - st0.t * st1.s)
Tx = (-q0.x * st1.s + q1.x * st0.s) / (st1.t * st0.s - st0.t * st1.s)

The same trick can be pulled for the X and Y axes - we'll use the same input st0 and st1 derivatives but the y or z components of our object-space derivatives. The resulting six values Sx Sy Sz, Tx Ty Tz are our basis vectors in tangent space.

vec3 S = ((q0 * st1.t - q1 * st0.t) / (st1.t * st0.s - st0.t * st1.s));
vec3 T = ((-q0 * st1.s + q1 * st0.s) / st1.t * st0.s - st0.t * st1.s));

Cleaning It Up

What units are our newly formed basis vectors? Well, they're in the right unit to rescale from UV map units (1 unit = one repetition of the texture) to our eye space units (whatever the hell they are). The main thing to note here is that S and T are almost certainly not unit vectors, but if we we are going to convert a normal map from tangent to eye space, we need unit vectors. So we are going to normalize S and T.

Now we can make an observation: the denominator of S and T (st1.t * st0.s - st0.t * st1.s) is the same for S and T and more importantly the same for all three axes. In other words, that divisor is really a constant scaling term.

Well, if we are going to normalize anyway, we don't really need a constant scaling term at all. Nuke it!

vec3 S = normalize( q0 * st1.t - q1 * st0.t);
vec3 T = normalize(-q0 * st1.s + q1 * st0.s);

When Does This Break Down

You may know from experience that tangent space normal mapping will fail if the author's UV map is zero or one dimensional (that is, the author mapped a triangle in the mesh to a line or point on the UV map). From this equation we can at least start to see why: our denominator was

st1.t * st0.s - st0.t * st1.s

When is this zero? It is zero when:

st1.t / st1.s = st2.t / st2.s

Which is to say: if the direction of the UV map derivatives are the same if we go up the screen or to the right on the screen, our UV map is degenerate and we're not going to get a sane answer.