2036881 - Reduce MathML fingerprintability due to FMA paths

Status: ASSIGNED

(Core :: MathML, enhancement)

Description

[Tom Ritter [:tjr]]

Telemetry-based analysis of MathML torture-test widths showed Android clients clustering into 4-5
distinct float32 ULP bands per element, with the same MATH-table-font binary producing bit-different
outputs depending on ARM core characteristics (FMA vs. non-FMA paths). Cross-OS bit-equivalence held
for desktop (WINNT and Darwin Cambria-Math cohorts produce identical torture-element values across
all 10 elements when the same font binary is used).

Comment 1

[Tom Ritter [:tjr]]

Attachment: Bug 2036881: Promote MathML stretchy-operator scale computations from float to double r?#layout-reviewers

Telemetry from the MathML metrics shows ARM-CPU-dependent ULP banding in
getBoundingClientRect().width on stretchy-operator-bearing formulas:
Some Linux sub-cohorts differ by 0.03-0.08 px (1-4 float32 ULPs at this
magnitude) on the same font binary, and Android shows 4-5 distinct
float32 bands per element on otherwise-identical fonts. The variance
correlates with ARM core class (FMA vs non-FMA divergence), not with font.

Tracing the getBoundingClientRect path:
Element::GetBoundingClientRect
-> DOMRect::SetLayoutRect (already double, quantizes to 1/65536 px)
-> frame's nsRect (nscoord/int32 - already perturbed by here)

So the variance enters before SetLayoutRect, in the layout production path
that mutates nscoord values. Audit of layout/mathml/ surfaces:

nsMathMLChar.cpp stretch-character scaling
- 4 `float scale = ...` declarations
- nscoord *= scale mutation sites for ascent/descent/leftBearing/
rightBearing/width
- mScaleX/mScaleY *= scale mutations
- Module-level scale-factor constants

nsMathMLmpaddedFrame.cpp
- 2 NSToCoordRound(float(int) * float_attr) sites for mpadded
Number/Percent attributes

Promote these to double precision. Mutations remain `nscoord *= scale`;
the implicit `int32 * double -> int32` truncation is preserved (this is
a precision change, not a rounding-mode change). gfxMathTable is already
double-precision throughout, so no change needed there.

mScaleX/mScaleY field types also become double. They feed
gfxContext::SetMatrixDouble's PreScale (which takes gfxFloat = double)
and gfxFloat r.width /= mScaleX, both strict-improvement consumers.

Comment 2

[Tom Ritter [:tjr]]

Try links:
https://treeherder.mozilla.org/jobs?repo=try&revision=9111c1c69d04faa1c6584e25aaebd302042f8b9f
https://treeherder.mozilla.org/jobs?repo=try&revision=bddb02ecaa977a3c37a63e6a0767c0f62552dd45