3.1.8.1.3 Extra Quantization

When performing progressive encoding, an extra quantization step is performed on the data resulting from the Sub-Band Diffing Stage. Quantization is expressed in terms of the number of bits that are shifted. The number of bits is a function of:

  • The regular quality (low, medium-low, medium-high, high)

  • The color component (Y, Cb or Cr)

  • The band (HH1 to LL3)

  • The progressive chunk (0% to 100%)

Assume the following:

BitPos(quality, component, band, 0%) = 15

BitPos(quality, component, band, 100%) = 0

0 <= BitPos(quality, component, band, chunk) < 15; where (chunk > 0%)

If SB is the result of the sub-band diffing decision engine, the progressive quantized value is calculated as follows:

if (SB >= 0) then ProgQ(chunk) = SB >> BitPos(chunk)

if (SB < 0) then ProgQ(chunk) = -((-SB) >> BitPos(chunk))

This can also be expressed as:

ProgQ(chunk) = SB/PQF(chunk); division MUST round the result toward zero

Where PQF(chunk) is the Progressive Quantization Factor:

PQF(chunk) = 1 << BitPos(chunk)

However, the progressive quantized value of the LL3 band is calculated differently. The quantization of the elements is performed toward negative infinity, resulting in the following formula:

ProgQ-LL(chunk) = SB >> BitPos(chunk)