156594 - rounding error in format-number() function with "#0."

Status: VERIFIED FIXED

(Core :: XSLT, defect)

Description

[Jacob Kjome]

Using build 2002070913 on Win2k (SP2)

This bug is related to bug 155734 and its testcases...


Description:

It seems Mozilla also has a bug in its rounding.

For instance, given this input

<number>37425.6770833333</number>

and this xslt statment

<xsl:value-of select="format-number(number, '#0.')"/>

Mozilla outputs:  37425
     IE outputs:  37426

It seems to me that IE is right and Mozilla is wrong in this case since any
number 5 and above after the decimal point should force the function to round
up.  Mozilla doesn't round at all.


However, Mozilla seems to do the right thing in cases such as

<xsl:value-of select="format-number(number, '#0.#')"/>

where it outputs:  37425.7 just like IE


Expected:
The pattern of "#0." should cause 37425.6770833333 to round to 37426

Actual:
The pattern of "#0." causes 37425.6770833333 to round to 37425

A testcase is coming up...

Jake

Comment 1

[Jacob Kjome]

Attachment: XSLT stylesheet using format-number()

Comment 2

[Jacob Kjome]

Attachment: XML file with data to test xslt file

Comment 3

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Attachment: patch to fix

fixes this problem, bug 155734, a buffer overrun in standalone and a buffer
underrun in module

Comment 4

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

@@ -287,9 +286,15 @@:
fixes a buffer overrun problem where I had forgot to allocate enough buffersize
for the initial zeros in formats like '00000000.0'

@@ -262,7 +262,6 @@
@@ -346,6 +351,12 @@
@@ -446,6 +459,12 @@
fixes bug 155734 by not enforcing that minIntergerSize is > 0 (i.e. allow no
'0's before decimal separator). It also needs to insert a '0' if we havn't
outputted any numbers while creating the number. This happens when you use a
format like '##.##' to format the number 0.0001.

So we will now output 0.001 formatted with '##.##' as '0' and 0.01 formatted
with '##.##' as '.01'

@@ -373,7 +384,7 @@
This is needed to avoid a divide-by-zero if minIntegerSize == 0

@@ -316,10 +321,10 @@
@@ -330,7 +335,7 @@
Fixes some serious brokenness in standalone. It was pretty much not working before.

@@ -391,16 +402,18 @@
Fixes a buffer underrun bug in module.

@@ -430,7 +443,7 @@
Fixes bug 156594. I forgot to actually add +1 when detecting that a carry
existed during output of the integerpart

Comment 5

[Axel Hecht]

Are we two against foo that the 6 failing xalan tests in numberformat are both
# doesn't generate leading zeros, and there is no default - prepended to 
the second subpattern?

Comment 6

[Axel Hecht]

Comment on attachment 90758 [details] [diff] [review]
patch to fix

r=[e-mail]
(standalone has two more fails, but those are just output)
"Alles wird gut". Which is evil.

Comment 7

[Boris Zbarsky [:bzbarsky]]

Comment on attachment 90758 [details] [diff] [review]
patch to fix

sr=bzbarsky

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

checked in

Comment 9

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

fixed with the branchlanding

Comment 10

[Axel Hecht]

we didn't verify for a long time.
I really checked, so VERIFIED.