186563 - Number.toFixed: loss of precision

Status: VERIFIED INVALID

(Core :: JavaScript Engine, defect)

Description

[石庭豐 (Seak, Teng-Fong)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3a) Gecko/20021201

In the following code, for 17 < f <= 20, toFixed loses precision:

var a = 0.827;
var b = 1.827;
var f = 17;

document.write(
  "a = " + a + " ==> a.toFixed(" + f + ") = " + a.toFixed(f) + "<br>" +
  "b = " + b + " ==> b.toFixed(" + f + ") = " + b.toFixed(f))


Reproducible: Always

Steps to Reproduce:

Comment 1

[Phil Schwartau]

(OUTPUT IN MOZILLA):
a = 0.827 ==> a.toFixed(17) = 0.82699999999999996
b = 1.827 ==> b.toFixed(17) = 1.82699999999999996


This type of behavior is seen with many types of numerical 
operations in JavaScript, for example with multiplication:

5.1 * 10
51

5.1 * 100
509.99999999999994


The reason for this is the following: computers do arithmetic
in base 2, not base 10. Certain numbers which have finite decimal
expansions in base 10 have infinite decimal expansions in base 2.
That, plus the fact that the computer must round off infinite 
expansions to finite ones, produces the results above.

The algorithm for Number.prototype.toFixed involves arithmetic
operations which are done in C (in the SpiderMonkey implementation
of JavaScript). The C language adheres to the ANSI/IEEE standard 754
for floating-point arithmetic, which requires this behavior.

That is why all the following bugs were marked invalid:

bug 20140   "Rounding Error?"
bug 42134   "JavaScript should be more lenient comparing (==) floats"
bug 129300  "Error in javascript internal routine : parseFloat"
bug 154103  "Arithmetic error in JavaScript Engine"
bug 160173  "Error de punto flotante al sumar en JavaScript"


Note that Netscape's web documentation is quite clear about loss of
precision that may be encountered when using Number.prototype.toFixed :

http://developer.netscape.com/docs/manuals/js/core/jsref15/number.html#1200964
contains an excellent explanation of some of the unexpected behavior.


However, I will defer to Roger or Waldemar on the current bug,
because IE6 seems to behave differently than we do in this case:


OUTPUT IN MOZILLA:
a = 0.827 ==> a.toFixed(17) = 0.82699999999999996
b = 1.827 ==> b.toFixed(17) = 1.82699999999999996

OUTPUT IN IE6:
a = 0.827 ==> a.toFixed(17) = 0.82700000000000000
b = 1.827 ==> b.toFixed(17) = 1.82700000000000000


My questions for Roger or Waldemar are:

1. Has SpiderMonkey (also Rhino) violated the spec in any way here?
2. Even if not, would we want to emulate IE in cases like this?

Comment 2

[Kenton Hanson (gone)]

Phil's analysis of the observed behavior is correct.  The difference seen in IE
and Mozilla is as follows. IE is storing 'a' as a string and Mozilla is storing
'a' as a value.  The spec doesn't nail down the storage format.  Thus when IE
does a.toFixed it starts out with a exact string representation while Mozilla
suffers the round trip conversions described by Phil.

Comment 3

[Phil Schwartau]

Kenton: thanks! I will go ahead and mark this invalid, then. 
If anyone objects, please reopen -

Comment 4

[Phil Schwartau]

Marking Verified.

[e-mail]: thank you for this report.
This is an interesting issue that often comes up -

Comment 5

[Garrett Smith]

Hi Phil,

Glad you find this interesting. I hope reopening is appropriate. I am very reluctant to reopen this. 

// This should be 1.126, not 1.125.
// Webkit, Spidermonkey, Futhark are all wrong.

(1.1255).toFixed(3); 

for convenience: 
javascript:alert((1.1255).toFixed(3))

http://bclary.com/2004/11/07/#a-15.7.4.5
Look at steps 10 and 11.  
10. Let n be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible. If there are two such n, pick the larger n.

11. If n =0, let m be the string "0". Otherwise, let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes).

I've tried to correctly implement the algorithm in JS.
========================================================
/**
 *   - Fix for rounding error in JScript.
 *   - Fix for rounding in all other engines - step 10, adjust precision.
 *   - Range for fractionDigits expanded to {-20...100}
 */

Number.prototype.toFixed = function(fractionDigits) {
    var f = parseInt(fractionDigits) || 0;
    if( f < -20 || f > 100 ) { 
        throw new RangeError("Precision of " + f + " fractional digits is out of range");
    }
    var x = Number(this);
    if( isNaN(x) ) {
        return "NaN";
    }
    var s = "";
    if(x <= 0) {
        s = "-";
        x = -x;
    }
    if( x >= Math.pow(10, 21) ) {
        return s + x.toString();
    }
    var m;
// 10. Let n be an integer for which the exact mathematical value of 
// n ÷ 10^f - x is as close to zero as possible. 
// If there are two such n, pick the larger n.
    n = Math.round(x * Math.pow(10, f) );

    if( n == 0 ) {
        m = "0";
  	}
    else {
        // let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes).
        m =  n.toString();
    }
    if( f == 0 ) {
        return s + m;
    }
    var k = m.length;
    if(k <= f) {
        var z = Math.pow(10, f+1-k).toString().substring(1);
        m = z + m;
        k = f+1;
    }
    if(f > 0) {
        var a = m.substring(0, k-f);
        var b = m.substring(k-f);
        m = a + "." + b;
    }
    return s + m;
};


I had to improvise for step 10. The algorithm in the spec omits the detail. The spec doesn't nail down the storage format, as Kentop pointed out, but it does say that the larger number should be taken. I provided no code there to "pick the larger of the two," as the spec dictates. 

The function above produces the result for things like:

(1.1255).toFixed(3); // 1.126 - correctly rounded.
(0.827 ).toFixed(17); // No loss of precision.

This behavior seem more practial and useful. 
=========================================================

Related bugs:
Bug 397880 – Incorrect rounding by toFixed() for some values when last decimal digit is 5 (edit)
Bug 226993 – Javascript toFixed rounds in unexpected manner (edit)

Where can I find Mozilla's source code for Number.prototype.toFixed(fractionDigits)?

Comment 6

[Garrett Smith]

Attachment: Test case showing original vs patched versions

Test case showing original vs patched versions of 
Number.prototype.toFixed

Allows for negative formatDigits, which was not properly supported in last post.

Comment 7

[Garrett Smith]

In the attachment,
(.07).toFixed() == 0
can be ignored. This will always be true.

Comment 8

[Garrett Smith]

Attachment: More Complete Test Case

Mozilla and Opera have rounding errors with (1.1255).toFixed(3) and (1.1255).toFixed(17), losing precision. 

Safari 2 got the wrong result with (1.1255).toFixed(3), but got the correct result with (1.1255).toFixed(17). 

Only Safari 3 got it right the first time. The RangeError it throws on a negative fractionDigits is perfectly valid. 

Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification.

Comment 9

[Garrett Smith]

Attachment: More Complete Test Case Description

Noticed that for .07, Safari 2 returns .1, but should instead return 0.1.

Only Safari 3 gets the correct result.

Comment 10

[Garrett Smith]

Attachment: Alternate "correct" Algorithm -- For Step 10.

Instead of using Math.round, it would be more true to the spec to implement step 10 with the following:

//----------------------------------------------------------------------------
// 10. Let n be an integer for which the exact mathematical value 
// of n ÷ 10^f - x is as close to zero as possible. 

    var fTo10 = Math.pow(10, f);
    
    n = Math.floor(x * fTo10 );
// If there are two such n, pick the larger n.    
    if (Math.abs(n / fTo10 - x) >= Math.abs((n + 1) / fTo10 - x)) {
        ++n;
    }
//----------------------------------------------------------------------------

Math.round works correctly.
http://bclary.com/2004/11/07/#a-15.8.2.15 

So should Number.prototype.toFixed.

It seems likely that the native code uses something like my first patch, but chooses error-prone rounding, and does not implement rounding as Math.round does. I can guess this to be the problem; where is the !#$@ing source code?

I have found no problems with the first patch I wrote using javascript's Math.round. 

The second approach I have written should be translated to native code.

I do, however, have a problem with the current implementation, which yeilds:
(1.1255).toFixed(3) // 1.125
(.1255).toFixed(3) // 0.126

Comment 11

[Garrett Smith]

toPrecision is also broken in all browsers but IE. (I did not test Opera 9.5b/Futhark Engine, but it is probably broken there, too)

javascript:alert(1.1255.toPrecision(4))

toExponential is broken in all browsers but IE and Safari 3/Webkit
javascript:alert(1.1255.toExponential(20))

The JS Guide documentation for toPrecision is wrong. Found with Google Search:   toPrecision documentation
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Predefined_Core_Objects:Number_Object

"Returns a string representing the number to a specified precision in fixed-point notation."

Incorrect and incomplete.

The reference documentation is correct, but avoids discussion of the rounding errors/bugs in Mozilla:
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Global_Objects:Number

------------------------------------------------------------------------------
We need a meta bug with dependencies of the following:
 toPrecision, toFixed, toExponential 

For each of the above, there should be a bug rep't tests of boundary and error-prone values.

* run the tests
* write patches/get review/checkin et c.

Comment 13

[Waldemar Horwat]

Per the ECMAScript standard, the correct answer for 1.1255.toPrecision(4) is "1.125".  This is what Mozilla produces.  This is not a bug.  The reason is that 1.1255 cannot be represented exactly as an IEEE double, so internally it is represented as 1.1254999999999999449329379785922355949878692626953125.

Comment 14

[石庭豐 (Seak, Teng-Fong)]

In reply to #13:
If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard.

Comment 15

[Waldemar Horwat]

We're doing that.  ES4 will also have a decimal number type.

Comment 16

[Garrett Smith]

The code example, follows the ECMA 262 algorithm and gets a different result than the native code. A more desirable one, I might add.

Does the example follow the algorithm incorrectly? If so, where. 

Where is the source code so we can have a look?

Comment 17

[Waldemar Horwat]

Yes, the example follows the ECMA algorithm incorrectly.  The mistake is at the very beginning -- it evaluates the token "1.1255" incorrectly.  This token evaluates to 1.1254999999999999449329379785922355949878692626953125.  This is required by ECMA 262 sections 7.8.3 and 8.5.  The rest of the example falls apart after this.

You also wrote "Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification."  How is this relevant here?  If you're referring to the note at the end of 15.7.4.5, then please re-read that note.  What it's stating is that instead of throwing RangeError, toFixed is allowed to, upon request, correctly produce more than 20 digits of the value it's given.  By ECMA 7.8.3 and 8.5, we've established that the value given is 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can see all of its digits in the Mozilla implementation of toFixed.

Comment 18

[Waldemar Horwat]

In the example the problem is in the emulation of step 10.  The spec requires you to pick n to be an integer for which the exact mathematical value of n ÷ 10^f - x is as close to zero as possible.  If there are two such n, pick the larger n.

This is not implemented by n = Math.round(x * Math.pow(10, f)) because that expression has double (or triple) rounding.  You round once in the * operator and again in Math.round; there may be a third rounding in Math.pow depending on the value of f.  The spec requires exact arithmetic, not rounded intermediate results.

Comment 19

[Garrett Smith]

(In reply to comment #17)
> Yes, the example follows the ECMA algorithm incorrectly.  The mistake is at the
> very beginning -- it evaluates the token "1.1255" incorrectly.  This token
> evaluates to 1.1254999999999999449329379785922355949878692626953125.  This is
> required by ECMA 262 sections 7.8.3 and 8.5.  The rest of the example falls
> apart after this.
> 
> You also wrote "Aside from fixing the bugs in the other browsers, this patch
> enhances toFixed with a greater precision. This is permitted by the
> specification."  How is this relevant here?  If you're referring to the note at
> the end of 15.7.4.5, then please re-read that note.  What it's stating is that
> instead of throwing RangeError, toFixed is allowed to, upon request, correctly
> produce more than 20 digits of the value it's given.  By ECMA 7.8.3 and 8.5,
> we've established that the value given is
> 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can
> see all of its digits in the Mozilla implementation of toFixed.
> 
Are you saying 1.1255.valueOf() should return 1.1254999999999999449329379785922355949878692626953125

Comment 20

[Waldemar Horwat]

Not only that, merely 1.1255 returns the exact mathematical value 1.1254999999999999449329379785922355949878692626953125.  It happens to print as "1.1255" if you convert it back to a string, but it's actually less than the exact mathematical value 1.1255.

Comment 21

[Brendan Eich [:brendan]]

Garrett: now you may appreciate why decimal is coming in JS2/ES4 ;-).

/be

Comment 22

[Brendan Eich [:brendan]]

(In reply to comment #19)
> (In reply to comment #17)
> > Yes, the example follows the ECMA algorithm incorrectly.  The mistake is at the
> > very beginning -- it evaluates the token "1.1255" incorrectly.  This token
> > evaluates to 1.1254999999999999449329379785922355949878692626953125.  This is
> > required by ECMA 262 sections 7.8.3 and 8.5.ntation of toFixed.
> > [snip]

> Are you saying 1.1255.valueOf() should return
> 1.1254999999999999449329379785922355949878692626953125

No, that's not what waldemar wrote. The point is that implementations must scan the 1.1255 token using extra internal precision, necessary for correct rounding to the binary precision limits of IEEE double.

Nothing in the spec requires or indeed allows visible results in the language to have greater than double precision. Which is why we have bugs on file about hardware such as modern Macs, which oddly use 80-bit precision in the x87 FPU but 64-bit in SSE (Mac OS X does not seem to allow you to override this strange FPU configuration; if anyone knows how, please mail me).

/be

Comment 23

[Garrett Smith]

Makes sense. Looking fwd to ES4.

Comment 28

[Alain Verleyen]

(In reply to comment #5)
Thank you Garrett for this FIX ! I need a method to obtain a mathematically  correct response. 
All this internal stuff on the specification is not a responsible response to a practical problem encountered by users.
I agree with comment #14 "If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the so-called standard".

Comment 30

[Jim Michaels]

here's another one:
565133671234567.89.toFixed(2)
565133671234567.88