2046791 - Normalize CPU-architecture-dependent pixels (SSE2 vs scalar/NEON un-premultiply)

Status: RESOLVED FIXED

(Core :: Graphics: Canvas2D, defect)

Description

[Tom Ritter [:tjr]]

For a 2D canvas containing any non-opaque (alpha < 255) pixel, getImageData() returns different pixel bytes depending on the client's CPU architecture. The drawing operations, geometry, coverage (alpha channel), and fully-covered interior pixels are identical across architectures; the difference is confined to anti-aliased / alpha-blended fringe pixels, where each differing channel is off by exactly ±1 (one 8-bit LSB). The split is 100% clean by CPU architecture and was root-caused to the per-architecture un-premultiply implementation on the readback path, then confirmed on try by switching that implementation in isolation.

---

The rendering difference

Identical drawing operations, rendered on the software path and read back with getImageData(), hash differently by CPU architecture (SHA-1 of the readback bytes):

canvas content	x86 / x86-64	aarch64 / arm	interiors identical?
opaque fillRect (alpha=255 only)	7c3f7a6e…	7c3f7a6e…	yes — no difference
AA triangle + 5px stroke	23aa45bf…	57ed9aec…	yes except AA fringe
globalAlpha=0.5 rotated rects	edc3b8fc…	77b662c4…	yes except AA fringe

The split is exact: every x86/x86-64 client produces the first hash of each pair, every aarch64/arm client the second. At the pixel level the two outputs of a representative frame differ on ~0.76% of bytes, all on sub-255-alpha edge pixels, each by 1. Opaque canvases (every pixel alpha=255) are byte-identical across architectures — the control that localizes the cause.

Root cause

CanvasRenderingContext2D::GetImageDataArray (dom/canvas/CanvasRenderingContext2D.cpp:6731) converts the canvas's premultiplied-alpha surface to the un-premultiplied RGBA bytes handed to JS by calling mozilla::gfx::UnpremultiplyData (gfx/2d/Swizzle.cpp), which dispatches to a different un-premultiply implementation per architecture:

arch	implementation	reciprocal arithmetic
x86 / x86-64	Unpremultiply_SSE2 (gfx/2d/SwizzleSSE2.cpp)	16-bit scaled reciprocal (UNPREMULQ_SSE2 + _mm_mulhi_epu16) — loses precision, rounds down on some (channel, alpha) pairs
ARM	Unpremultiply_NEON (gfx/2d/SwizzleNEON.cpp)	full 24-bit reciprocal split hi/lo, rounded
other	scalar UnpremultiplyFallback (gfx/2d/Swizzle.cpp)	(channel * (0xFF00FF / a)) >> 16

The SSE2 path packs a 24-bit reciprocal-of-alpha into 16 bits (scaling by 8 or 256), which rounds down where the exact division rounds up, producing a +1 difference on the un-premultiplied channel for sub-255-alpha pixels. The NEON and scalar paths keep full precision and agree with each other. So x86 differs by one LSB from ARM (and from any scalar-fallback architecture) on exactly the anti-aliased / alpha pixels.

This is also why only some canvases split: un-premultiply of an opaque pixel (alpha=255) is the identity (no division), so canvases with only opaque pixels are architecture-stable; only canvases with AA edges or alpha blending exercise the divergent divide. The difference is therefore in Firefox's own gfx/2d un-premultiply code on the getImageData readback path — not in Skia rasterization and not in the blend/composite step.

How the root cause was confirmed

Four independent checks on try, all consistent:

1. Forcing the scalar path flips x86 to exactly the ARM bytes. A build that makes UnpremultiplyData/UnpremultiplyRow skip the SSE2/NEON paths and use the scalar fallback (8308c97e9967) changed the x86 output 23aa45bf→57ed9aec (AA triangle) and edc3b8fc→77b662c4 (alpha rects) — byte-identical to the native ARM hashes — while the opaque canvas was unchanged. Switching only the un-premultiply implementation reproduces the entire difference.

2. Native ARM is unchanged by the same switch. On native Apple Silicon (macOS aarch64) the same forced-scalar build left the canvases at 57ed9aec/77b662c4: the scalar fallback produces the same bytes as NEON. So x86 differs from ARM solely because it runs Unpremultiply_SSE2; NEON and scalar agree.

3. The blend/raster step is not involved. A build that de-fuses the Skia float raster-pipeline multiply-add on x86 (forcing a+f*m instead of _mm256_fmadd_ps; 981be1ed062e) produced byte-identical output to the unmodified build (still 23aa45bf/edc3b8fc). The divergence is not in floating-point blend rounding; it is downstream in the integer un-premultiply. (This rules out the fused-multiply-add hypothesis.)

4. Unit-level confirmation. A gtest (Moz2D.UnpremultiplyPerfSSE2VsScalar, e3f27aec8a62) runs the same premultiplied buffer through both the SIMD and scalar UnpremultiplyData and compares: on x86 the two outputs differ on a fixed ~0.76% of bytes (the AA/alpha pixels), each by 1; on ARM they are identical. This isolates the divergence to UnpremultiplyData, with no rendering involved.

Proposed and tested solution

As I understand it, there are two ways to fix this:

1. Use the scalar path for x86. This is predictably going to be slower.
2. Modify the x86 path to match NEON. This is tweaking the assembly to produce the same output, most likely making it slower but probably not as much as the scalar path. claude implemented this here: https://hg-edge.mozilla.org/try/rev/20e1aabd0ba875985cf0b92e7779b5b82aabaeec and confirmed it did work ane make this identical

Comment 1

[Tom Ritter [:tjr]]

Alright, so let's compare performance

• Base commit: https://treeherder.mozilla.org/jobs?repo=try&revision=90578ae32b540dd2346125f9b6bea99730ee425b

• Scalar Path: https://treeherder.mozilla.org/jobs?repo=try&revision=237f9778595ab8d36bf086b765af47cc428d2754

• Modifed x86: https://treeherder.mozilla.org/jobs?repo=try&revision=0af076a7b93ea3c2c4e5a47ece5f4c93d63dd3fb

• Base vs Scalar

• Base vs Modified x86

I was advised that the most relevant tests for comparison are

• talos-svgr (tsvgx / tsvgm / tsvg_static / tsvgr_opacity) — the best macro proxy. SVG filter rendering routes through FilterProcessing → UnpremultiplyData/PremultiplyData.
  • Talos graphics suites g1/g3/g4/g5 and talos-bcv — the general 2D-canvas/graphics bucket; include for completeness, but expected to be flat (mostly drawing/compositing, which stays
    premultiplied and never hits the readback un-premultiply).
  • MotionMark (browsertime-benchmark-firefox-motionmark-1-3 and -motionmark-htmlsuite-1-3 — the headline 2D-canvas throughput test. Worth running, but it's a draw benchmark (rAF rendering, no getImageData), so it likely won't move either

SVG — filter path, the real proxy (lower is better — ms)

suite	platform	Base	Scalar	Δ%	Mod-x86	Δ%
tsvgx	linux64	332.6	329.4	−1.0	333.5	+0.3
tsvgx	mac-x64	152.8	151.8	−0.6	150.9	−1.2
tsvgx	win64	127.5	130.6	+2.4	126.0	−1.1
tsvg_static	linux64	40.3	38.8	−3.8	39.6	−1.9
tsvg_static	mac-x64	48.2	48.1	−0.2	49.6	+2.8
tsvg_static	win64	36.5	36.2	−0.9	37.5	+2.5
tsvgr_opacity	linux64	104.3	104.0	−0.3	104.8	+0.4
tsvgr_opacity	mac-x64	89.0	89.4	+0.4	89.5	+0.5
tsvgr_opacity	win64	62.1	66.3	+6.8	61.0	−1.7
rasterflood_svg	linux64	9033	9039	+0.1	9081	+0.5
rasterflood_svg	mac-x64	8992	8993	+0.0	8998	+0.1
rasterflood_svg	win64	8850	8854	+0.0	8847	−0.0

• Windows tsvgx loses +2.4% on Scalar, but gets 1.1% faster on modified-x86.
• Windows tsvg_static gets .9% faster on scalar (?!) but 2.5% slower on modified-x86
• Windows tsvgr_opacity gets 6.8% slower on Scalar (good, expected) and 1.7% faster on modified-x86
• Mac gets 2.8% slower on Modified x86 but that's noise because that patch doesn't modify Mac.

tsvg_static uses opacity, but the opacity here is alpha compositing (group/element opacity blended to the screen in premultiplied space), which is not the same operation we touch. The patch is in the readback un-premultiply (UnpremultiplyData on the premultiplied→unpremultiplied path), which is hit by getImageData/toDataURL/createImageBitmap and SVG/CSS filters (FilterProcessing) — not by plain opacity. SVG opacity compositing stays premultiplied and presents to screen without round-tripping through the un-premultiply.

So tsvg_static exercising opacity doesn't make it a strong proxy for this change — it's mostly measuring SVG paint, which never calls the patched function. Of the relevant suites, the closest real proxy is still the filter content (and the filter-bearing SVGs in tsvgx/rasterflood_svg).

2D-canvas (lower is better — ms)

suite	platform	Base	Scalar	Δ%	Mod-x86	Δ%
canvas2dvideo	linux64	4.31	4.28	−0.8	4.81	+11.6
canvas2dvideo	mac-x64	48.8	49.6	+1.5	50.0	+2.5
canvas2dvideo	win64	18.0	17.9	−0.9	19.1	+5.9
offscreencanvas_2d¹	mac-x64	12.3	11.6	−5.9	10.5	−14.3
offscreencanvas_2d¹	linux64	9.8	9.8	±0.0	9.8	−0.2
offscreencanvas_2d¹	win64	22.0	22.1	+0.6	21.6	−1.4

¹ all four variants (main/worker × av1/vp9) behave identically; one row shown per platform.

• Again, that mac gets 2.5% lower and 14.3% faster on modified-x86 is noise because we didn't touch those paths!
• The 11.6% slowdown on linux looks concerning, but: the runs are [4.28, 4.29, 4.32, 4.35, 4.36, 4.38, 7.66]. Exclude the outlier and the mean is 4.33 which is right next to the base.

MotionMark (higher is better — score)

suite	platform	Base	Scalar	Δ%	Mod-x86	Δ%
motionmark-1-3	linux64	503.7	494.4	−1.9	477.5	−5.2
motionmark-1-3	mac-x64	898.5	929.4	+3.4	916.2	+2.0
motionmark-1-3	win64	1527	1521	−0.4	1525	−0.1
motionmark-htmlsuite	linux64	148.6	154.8	+4.1	137.4	−7.6
motionmark-htmlsuite	mac-x64	667.8	663.8	−0.6	640.3	−4.1
motionmark-htmlsuite	win64	1042	1039	−0.3	1051	+0.9

Conclusion

I don't think the modified x86 patch makes a significant difference in the benchmarks we have. On a microbenchmark it will certainly show up.

Comment 2

[Tom Ritter [:tjr]]

Attachment: Bug 2046791 - Make Unpremultiply_SSE2 bit-exact to the scalar/NEON un-premultiply r?#gfx-reviewers

getImageData() on a canvas with non-opaque pixels returned
CPU-architecture-dependent bytes: Unpremultiply_SSE2 used a 16-bit scaled
reciprocal (UNPREMULQ_SSE2 + _mm_mulhi_epu16) that rounds down by 1 LSB on a
subset of anti-aliased / alpha pixels, whereas the scalar fallback and NEON
paths use the full 8.16 reciprocal and agree with each other. The result was a
100%-clean x86-vs-ARM split in canvas readback.

Use the full 0xFF00FF/alpha reciprocal (same table as the scalar fallback) and
compute an exact (channel * reciprocal) >> 16 in 16-bit lanes by splitting the
reciprocal into high/low halves:
(channel*Q)>>16 = mullo(channel, Q>>16) + mulhi(channel, Q&0xFFFF).
This is bit-identical to the scalar/NEON output for all 256x256 (channel,alpha)
pairs (verified exhaustively) and keeps the SIMD path (two 16-bit multiplies +
an add per channel group). Both UnpremultiplyData and UnpremultiplyRow go
through UnpremultiplyVector_SSE2, so this fixes every getImageData caller.

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Lee Salzman [:lsalzman]]

Given how long ago I wrote the original code, and how comparatively marginal the performance loss might be here, it seems worth the potential slight performance sacrifice to improve the quality of the result and to reduce incidence of finger-printing as well. Thumbs-up.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/73160a2005d9
https://hg.mozilla.org/integration/autoland/rev/9a280a7bfcb2
Make Unpremultiply_SSE2 bit-exact to the scalar/NEON un-premultiply r=gfx-reviewers,lsalzman

Comment 6

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9a280a7bfcb2