1605113 - Premultiplication methods produce slightly incorrect results

Status: ASSIGNED

(Core :: Graphics, defect, P2)

Description

[Andrew Osmond [:aosmond] (he/him)]

Currently to premultiply the alpha to the RB components, we use the following algorithm:

rb = rb*a + 255
rb += rb >> 8

This was intended to be distilled from GFX_DIVIDE_BY_255 and gfxPremultiply which maps to, where v = individual color component:

(va << 8 + va + 255) >> 16

This requires at least 24-bits for our integers which isn't directly workable with SSE2 given we are using 16-bit integers. However with slight manipulation, we go to:

(va + (va + 255) >> 8) >> 8

va maximum is 255 * 255, which fits in 16-bits.
va + 255 maximum is 255 * 255 + 255, or 256 * 255 which also fits in 16-bits.

Mapping this back to the accelerated premultiplication methods, I would expect to see:

rb = rb*a
rb += (rb + 255) >> 8

This produces the correct result for me, and allows us to remove all sorts of fuzz I added as well as produce the actual correct results :).

Comment 1

[Andrew Osmond [:aosmond] (he/him)]

Discussing further with Jeff, it appears neither is what we would expect. Things are written with the expectation of integer truncation, but Jeff's assertion is that we should be rounding. This boils down to just changing the 255 in the original algorithm to 128:

rb = rb*a + 128
rb += rb >> 8

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

Related tries I did last December:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=55c117d1dd3e07d8bf68b50dcdd090d6bd264478
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a52c1c02cc6992b6a3f7eb427cfcfd0ad55f1795