182670 - "Rounding" error in computed style values

Status: RESOLVED FIXED

(Core :: Layout, defect, P3)

Description

[Erich 'Ricky' Iseli]

I've got the default font-size of 16px (edit>preferences>appearance>fonts), and
an author stylesheet setting an element to 90%.

Normally, the computed value should be 16px*90% (mathematically that would be
16*0.9) which yelds 14.4px. However, the DOM-Inspector yelds 14.3529.

On IRC, I've been told this is because of a rounding error, however I don't see
where there is a rounding to be done. How does Mozilla calculate this value anyways?

And a suggestion: since there are rounding errors as we see here, wouldn't it be
more useful to round the result to one digit after the comma? Having 4 digits,
of which all 4 are wrong is really not useful, if you come to use the DOM-Inspector.

Comment 1

[James Ross]

This is very odd, I'm getting 14.3333px on Windows 2000, using Mozilla 20021128.

I'm just going to attach a simple test case.

Comment 2

[James Ross]

Attachment: Simple page with font-size of 90% on P tag.

Comment 3

[Christopher Aillon (sabbatical, not receiving bugmail)]

This is so not an inspector problem.  As you noted, inspector just uses
getComputedStyle() to obtain its info.

This is so not a getComputedStyle() problem.  It only gets the values from layout.

This is a layout problem caused partially because we do not store lengths in
pixels.  We store them in twips, which varies depending on your resolution.  Use
a standard resolution like 96dpi and you will get better results.

This is a dupe of bug 177805 and/or bug 63336 and various dependencies on the two.

Comment 4

[James Ross]

Dup or not, I'm using 96dpi, and getting results way off.

Comment 5

[Christopher Aillon (sabbatical, not receiving bugmail)]

Okay.  I thought 96dpi was more reliable but that appears to not be the case. 
Anywya, it's because of the other bugs I mentioned.  This is not an inspector
problem or DOM problem at all.

Comment 6

[Erich 'Ricky' Iseli]

Ok, dup of bug 63336. I'll post my comment #0 there.

*** This bug has been marked as a duplicate of 63336 ***

Comment 7

[Erich 'Ricky' Iseli]

By the way, the DOM-Inspector issue would be to just round to 1 digit instead of
4. What do you think?

Comment 8

[Erich 'Ricky' Iseli]

Un-duping as requested in bug 63336

Comment 9

[Boris Zbarsky [:bzbarsky]]

At 96 dpi, there are 15 twips per pixel.  So 0.4px == 4/10 * 15 == 6twips.  Thus
we should really not be rounding off 0.4px to anything at 96dpi... if we do, I
find that _very_ odd.

In any case, this is hardly DOM style.

Comment 10

[Boris Zbarsky [:bzbarsky]]

*** Bug 171761 has been marked as a duplicate of this bug. ***

Comment 11

[Eli Friedman]

The issue's at
http://lxr.mozilla.org/seamonkey/source/layout/style/nsRuleNode.cpp#1910.  I
think it should be rounding instead of truncating, although I'm not 100% sure. 
I'd make a patch, but it would be inconvenient because of other changes in my tree.

Comment 12

[Eli Friedman]

Attachment: Patch

This makes percentage font sizes round consistently; the old code was truncating, leading to odd results.

Comment 13

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Comment on attachment 206770 [details] [diff] [review]
Patch

>-    aFont->mSize = (nscoord)((float)(aParentFont->mSize) * aFontData.mSize.GetPercentValue());
>+    aFont->mSize = NSToCoordRound(aParentFont->mSize *
>+                                  aFontData.mSize.GetPercentValue());

sr=dbaron if you restore the explicit cast to float, preferably with a construction-style cast, i.e.,

NSToCoordRound(float(aParentFont->mSize) *
               aFontData.mSize.GetPercentValue())

Comment 14

[Boris Zbarsky [:bzbarsky]]

Attachment: Patch updated to dbaron's comment

Comment 15

[Boris Zbarsky [:bzbarsky]]

Fixed on trunk.  Eli, thanks for fixing this!