Showing posts with label Graphics. Show all posts
Showing posts with label Graphics. Show all posts

Saturday, June 11, 2022

sRGB, Pre-Multiplied Alpha, and Compression

 How do sRGB color-space textures (especially compressed ones) interact with blending? First, let's define some terms:

An image is "sRGB encoded" if the linear colors have been remapped using the sRGB encoding. This encoding is meant for 8-bit storage of the textures and normalized colors from 0.0 to 1.0. The "win" of sRGB encoding is that it uses more of our precious 8 bits on the dark side of the 0..1 distribution, and humans have weird non-linear vision where we see more detail in that range.

Thus sRGB "spends our bits where they are needed" and limits banding artifacts.  If we just viewed an 8-bit color ramp from 0 to 1 linearly, we'd see stripes in the darks and perceive a smooth color wash in the hilights.

Alpha values are always considered linear, e.g. 50% is 50% and is stored as 127 (or maybe 128, consult your IHV?) in an 8-bit texture. But how that alpha is applied depends on the color space where blending is performed.

  • If we decode all of our sRGB textures to linear values (stored in more than 8 bits) and then blend in this linear space, we have "linear blending" (sometimes called "gamma-correct" blending - the terminology is confusing no matter how you slice it).  With linear blending, translucent surfaces blend the way light blends - a mix of red and green will make a nice bright yellow. Most game engines now work this way, it's pretty much mandatory for HDR (because we're not in a 0..1 range, we can't be in sRGB) and it makes lighting effects look good.
  • If we stay in the 8-bit sRGB color space and just blend there, we get "sRGB blending". Between red and green we'll see a sort of dark mustard color and perceive a loss of light energy.  Lighting effects blended in sRGB look terrible, but sometimes artists want sRGB blending. The two reasons I've heard from my art team are (1) "that's what Photoshop does" and (2) they are trying to simulate partial coverage (e.g. this surface has rusted paint and so half of the visible pixel area is not showing the paint) and blending in a perceptual space makes the coverage feel right.
A texture has pre-multiplied alpha if the colors in the texture have already been multiplied (in fractional space) by the alpha channel. In a texture like this, a 50% translucent area has RGB colors that are 50% darker. There are two wins to this:
  • You can save computing power - not exciting for today's GPUs, but back when NeXT first demonstrated workstations with real-time alpha compositing on the CPU (in integer of coarse), premultiplication was critical for cutting ALU by removing half the multiplies from the blending equation.
  • Premultiplied alpha can be filtered (e.g. two samples can be blended together) without any artifacts.  The black under clear pixels (because they are multiplied with 0) is the correct thing to blend into a neighboring opaque pixel to make it "more transparent" - the math Just Works™. With non-premultiplied textures, the color behind clear pixels appears in intermediate samples - tool chains must make sure to "stuff" those clear texels with nearby colors.

Pre-Multiplication and sRGB

Can we pre-multiply sRGB encoded textures that are intended to be decoded and used in a linear renderer?Yes! But, here's the catch: we can only do this right if we know the type of blending we will do.

Pre-multiplying is basically doing a little bit of the blending work ahead of time. Therefore it's actually not surprising that we need to know the exact math of the blend to do this work in advance. There are two cases:

If we are going to blend in sRGB (e.g. nothing is ever decoded) we can just pre-multiply our 8-bit colors with 8-bit alpha and go home happy.  This is what we did in 2005 and we liked it, because it's all there was.

If we are going to decode our sRGB to linear space, then blend, we have to do something more complex: when pre-multiplying our texture we need to:
  1. Decode our colors to linear (and use more than eight bits to do the following intermediate calculations).
  2. Multiply the alpha value by the linear color.
  3. Re-encode the resulting darkened color back to sRGB.
We have now baked the multiply that would have happened in normal linear blending. This should be okay in terms of precision - while we are likely to have darker results post-blending, sRGB is already using bits for those darker areas, and they're going to have less weight on screen.

The fine print here is that the "bake" we did to convert from non-premultiplied to pre-multiplied has different math based on the blending color space and that decision is permanent. While a non-premultiplied texture can be used for either blending, once you pre-multiply, you're committed.

Does Compression Change Anything?

What if our sRGB textures are going to be compressed? In theory, no - S3TC compression states that the block is decompressed into sRGB color space first, then "other stuff happens". And this is useful - with only 16 bits for block color end-points, S3TC blocks need to spread their bits as evenly as possible from a perceptual perspective.

In practice, I'd be very wary. DXT3/5 provide two separate compression blocks, one for color, and one for alpha. But with premultiplied alpha, the alpha mask has been "baked" into the color channel and are going to put pressure on end-point selection.

Consider the case of a 4x4 tile that goes from clear to opaque as we go left to right, and green to blue as we go from bottom to top.

Without pre-multiplication, we pick green and blue as color end points, clear and opaque as alpha end points, and we can get very accurate compression.

With pre-multiplication we're screwed. We either pick black as an end point (so we can have black in the color under the clear pixels) and our color end point has to be a single green/blue mash-up, destroying any color change, or we pick green and blue as our colors and we have "light leak" when our colors aren't dark under the alpha.

For this reason, I think pre-multiplying isn't appropriate for block compressors even if we get linear blending correct.

Wednesday, October 21, 2020

A Tip for HiZ SSR - Parametric 't' Tracing

HiZ tracing for screen space reflections is an optimization where the search is done using a hierarchial Z-Buffer (typically stashed in a mip-chain) - the search can take larger skips when the low-res Z buffer indicates that there could not possibly be an occluder. This changes each trace from O(N) to O(logN).

The technique was published in GPU Pro 5, but as best I can tell, the author found out after writing the article that he couldn't post working sample code. The result is a tough chapter to make heads or tales of, because some parts of the algorithm simply say "see the sample code".  This forum thread is actually pretty useful, as is this.

The article builds a ray that is described parametrically in "Z" space, starting at the near clip plane (0.0 screen-space Z, must be a DX programmer ;-) and going out to 1.0.

If your app runs using reverse-float Z and you reverse this (starting the march at 1 and going in the -Z direction), you're going to get a ton of precision artifacts.  The reason: our march has the lowest precision at the start of the march.  In reverse-float Z there's a lot of hyper-Z (screen-space) depth "wasted" around the near clip plane - that's okay because it's the 'good' part of our float range, which gets better with distance. But in our case, it's going to make our ray testing a mess.

The technique is also presented tracing only near occluders and tracing only away from the camera - this is good if you view a far away mountain off a lake but not good if you view a building through a puddle from nearly straight down.

As it turns out, all of these techniques can be addressed with one restructure: parametrically tracing the ray using a parametric variable "t" instead of the actual Z buffer.

In this modification, the beginning of the ray cast is always at t = 0.0, and the end of the ray-cast (a full Z unit away) is always at t = 1.0, regardless of whether that is in the positive or negative Z direction - our unit is normalized so that its Z magnitude is either +1.0 or -1.0, which saves us a divide when intersecting with Z planes.

What does this solve?  A few things:

  1. The algorithm has high precision at the beginning of the trace because t has small magnitude - we want this for accurate tracing of close occluders and odd angles.
  2. The algorithm is fully general whether the ray is cast toward or away from the camera - no conditional code inside the search loop.  Lower 't' is always "sooner" in the ray cast.
  3. We can now march around an occluder if we can attribute a min and max depth to it, by seeing if the range of t within a search cell overlaps the range of t in our min-max Z buffer.  This is fully general for "in front" and "behind" a cast and for "toward" and "away".
I didn't see any mention of changing the parameterization in my web searching, nor did I see anyone else complaining about the precision loss from reverse-Z (it took me a good day or two to realize why my floating point was underperforming on that one) so I figured I'd put it out there.

Thursday, April 16, 2020

Specular Hilites Have Their Own Depth

I just noticed this effect while I was brushing my teeth. The faucet in the sink is:

  • Chrome (or stainless steel) or some other reflective "metal" and
  • Reasonably glossy and
  • Kinda dirty.
As I looked down, I could see specular reflections from the lights above the mirror and I could see calcium deposits on the surface of the faucet itself.

And...then I noticed that they didn't have the same parallax.  Close one eye, then the other, and the relative placement of the specular hilites with regards to the calcium deposits changes.

In other words, the specular hilites are farther away than the surface they reflect in.

If you take a step back and think about this, it's completely logical. An image of yourself a mirror appears twice as far away as the mirror itself, and if you draw an optical diagram, this is not surprising.  Twice as much eye separation parallax is 'washed out' by distance on the full optical path as to the surface of the mirror.

Interestingly, X-Plane's VR system correctly simulates this. We do per-pixel lighting calculations separately on each eye using a camera origin (and thus transform stack and light vector) specific to each eye. When I first coded this, I thought it was odd that the lighting could be "inconsistent" between eyes, but it never looked bad or wrong.

Now I realized that that inconsistency is literally a depth queue.