Who Needs the Inverse-Transpose?

This was pointed out to me by another OpenGL developer: if you don't have non-uniform scaling as part of your model view matrix, the upper 3x3 of the model-view matrix is just as good as the "normal matrix" (which is the transpose of the inverse of the model view matrix) for transforming normals. In fact, the model view matrix is better because it means you don't have to compute an inverse.

I hit this case while looking at a bug in X-Plane 930; for some reason the normal matrix produces junk results when used from a vertex (but not fragment) shader. Totally strange. The nice thing about using the model-view matrix is that it gives me a work-around that is theoretically a faster path. (The alternative of using the normal matrix in the fragment shader means slower per-pixel operations.)

(See FAQ 5.27 for a traditional explanation of why you'd use the inverse-transpose matrix on normals.)

We actually don't have scaling at all on our model view matrix for another reason: fast sphere culling. Basically to cull a sphere you need to:

1. Transform it from model-view to eye space.
2. Find the distance from the sphere to the six faces of the viewing volume.

If the distance is larger than the sphere radius and we're outside any of the viewing volume sides (the distance can be signed, so "outside" is part of the numeric test) then we are culled. The trick is: the model view matrix needs to not scale at all or our sphere radius might be wrong.

So in X-Plane, we don't ever scale the model view matrix. We don't offer it as a content option for authors (not that it would be useful anyway) and we do our zooming by manipulating FOV.

The side-effect of this is that we could use the transpose instead of inverse if we need to undo camera rotations, and we don't need to ue the inverse-transpose to transform normals.