There are two ways to transform a normal vector N given a matrix M:
- Compute the inverse of M, transpose it, and use that new M' to transform the normal vector.
- Transform M directly, and transpose the origin (0,0,0) then subtract the transformed origin form the transformed normal. This is like transforming the two end points of the normal.
- If you are transforming a lot of normals, calculate the inverse and transpose it once. Now you can invert normals directly.
- If you are transforming only one normal, it might be cheaper to transform two points rather than invert a 4x4 matrix.
If your matrix contains a large translation, then the result of transforming the normal and origin will be two points that are close together, but far the origin. This means that you have lost some precision, and subtracting won't get it back!
The inverse/transpose method does not have this problem; the transpose moves the translation part of the matrix off into the bottom row where we don't really care about it - the remaining matrix terms should have fairly small magnitudes and not lose precision.