rounding error in format-number() function with "#0."

VERIFIED FIXED

Status

()

VERIFIED FIXED
16 years ago
16 years ago

People

(Reporter: hoju, Assigned: sicking)

Tracking

Trunk
x86
Windows 2000
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: TX_BRIDGE_1_1_FIXED)

Attachments

(3 attachments)

(Reporter)

Description

16 years ago
Using build 2002070913 on Win2k (SP2)

This bug is related to bug 155734 and its testcases...


Description:

It seems Mozilla also has a bug in its rounding.

For instance, given this input

<number>37425.6770833333</number>

and this xslt statment

<xsl:value-of select="format-number(number, '#0.')"/>

Mozilla outputs:  37425
     IE outputs:  37426

It seems to me that IE is right and Mozilla is wrong in this case since any
number 5 and above after the decimal point should force the function to round
up.  Mozilla doesn't round at all.


However, Mozilla seems to do the right thing in cases such as

<xsl:value-of select="format-number(number, '#0.#')"/>

where it outputs:  37425.7 just like IE


Expected:
The pattern of "#0." should cause 37425.6770833333 to round to 37426

Actual:
The pattern of "#0." causes 37425.6770833333 to round to 37425

A testcase is coming up...

Jake
(Reporter)

Comment 1

16 years ago
Created attachment 90729 [details]
XSLT stylesheet using format-number()
(Reporter)

Comment 2

16 years ago
Created attachment 90730 [details]
XML file with data to test xslt file
(Reporter)

Updated

16 years ago
Attachment #90729 - Attachment mime type: text/xml → text/xsl
(Reporter)

Updated

16 years ago
Attachment #90729 - Attachment mime type: text/xsl → text/xml
Created attachment 90758 [details] [diff] [review]
patch to fix

fixes this problem, bug 155734, a buffer overrun in standalone and a buffer
underrun in module
@@ -287,9 +286,15 @@:
fixes a buffer overrun problem where I had forgot to allocate enough buffersize
for the initial zeros in formats like '00000000.0'

@@ -262,7 +262,6 @@
@@ -346,6 +351,12 @@
@@ -446,6 +459,12 @@
fixes bug 155734 by not enforcing that minIntergerSize is > 0 (i.e. allow no
'0's before decimal separator). It also needs to insert a '0' if we havn't
outputted any numbers while creating the number. This happens when you use a
format like '##.##' to format the number 0.0001.

So we will now output 0.001 formatted with '##.##' as '0' and 0.01 formatted
with '##.##' as '.01'

@@ -373,7 +384,7 @@
This is needed to avoid a divide-by-zero if minIntegerSize == 0

@@ -316,10 +321,10 @@
@@ -330,7 +335,7 @@
Fixes some serious brokenness in standalone. It was pretty much not working before.

@@ -391,16 +402,18 @@
Fixes a buffer underrun bug in module.

@@ -430,7 +443,7 @@
Fixes bug 156594. I forgot to actually add +1 when detecting that a carry
existed during output of the integerpart

Comment 5

16 years ago
Are we two against foo that the 6 failing xalan tests in numberformat are both
# doesn't generate leading zeros, and there is no default - prepended to 
the second subpattern?
Assignee: peterv → sicking

Comment 6

16 years ago
Comment on attachment 90758 [details] [diff] [review]
patch to fix

r=axel@pike.org
(standalone has two more fails, but those are just output)
"Alles wird gut". Which is evil.
Attachment #90758 - Flags: review+
Comment on attachment 90758 [details] [diff] [review]
patch to fix

sr=bzbarsky
Attachment #90758 - Flags: superreview+
checked in
Status: NEW → ASSIGNED
Whiteboard: TX_BRIDGE_1_1_FIXED
fixed with the branchlanding
Status: ASSIGNED → RESOLVED
Last Resolved: 16 years ago
Resolution: --- → FIXED

Comment 10

16 years ago
we didn't verify for a long time.
I really checked, so VERIFIED.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.