Closed
Bug 186563
Opened 22 years ago
Closed 17 years ago
Number.toFixed: loss of precision
Categories
(Core :: JavaScript Engine, defect)
Tracking
()
VERIFIED
INVALID
People
(Reporter: lapsap7+mz, Unassigned)
References
Details
(Keywords: testcase)
Attachments
(2 files, 2 obsolete files)
UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.0; enUS; rv:1.3a) Gecko/20021201 Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.0; enUS; rv:1.3a) Gecko/20021201 In the following code, for 17 < f <= 20, toFixed loses precision: var a = 0.827; var b = 1.827; var f = 17; document.write( "a = " + a + " ==> a.toFixed(" + f + ") = " + a.toFixed(f) + "<br>" + "b = " + b + " ==> b.toFixed(" + f + ") = " + b.toFixed(f)) Reproducible: Always Steps to Reproduce:
Comment 1•22 years ago


(OUTPUT IN MOZILLA): a = 0.827 ==> a.toFixed(17) = 0.82699999999999996 b = 1.827 ==> b.toFixed(17) = 1.82699999999999996 This type of behavior is seen with many types of numerical operations in JavaScript, for example with multiplication: 5.1 * 10 51 5.1 * 100 509.99999999999994 The reason for this is the following: computers do arithmetic in base 2, not base 10. Certain numbers which have finite decimal expansions in base 10 have infinite decimal expansions in base 2. That, plus the fact that the computer must round off infinite expansions to finite ones, produces the results above. The algorithm for Number.prototype.toFixed involves arithmetic operations which are done in C (in the SpiderMonkey implementation of JavaScript). The C language adheres to the ANSI/IEEE standard 754 for floatingpoint arithmetic, which requires this behavior. That is why all the following bugs were marked invalid: bug 20140 "Rounding Error?" bug 42134 "JavaScript should be more lenient comparing (==) floats" bug 129300 "Error in javascript internal routine : parseFloat" bug 154103 "Arithmetic error in JavaScript Engine" bug 160173 "Error de punto flotante al sumar en JavaScript" Note that Netscape's web documentation is quite clear about loss of precision that may be encountered when using Number.prototype.toFixed : http://developer.netscape.com/docs/manuals/js/core/jsref15/number.html#1200964 contains an excellent explanation of some of the unexpected behavior. However, I will defer to Roger or Waldemar on the current bug, because IE6 seems to behave differently than we do in this case: OUTPUT IN MOZILLA: a = 0.827 ==> a.toFixed(17) = 0.82699999999999996 b = 1.827 ==> b.toFixed(17) = 1.82699999999999996 OUTPUT IN IE6: a = 0.827 ==> a.toFixed(17) = 0.82700000000000000 b = 1.827 ==> b.toFixed(17) = 1.82700000000000000 My questions for Roger or Waldemar are: 1. Has SpiderMonkey (also Rhino) violated the spec in any way here? 2. Even if not, would we want to emulate IE in cases like this?
Assignee: rogerl → khanson
Comment 2•22 years ago


Phil's analysis of the observed behavior is correct. The difference seen in IE and Mozilla is as follows. IE is storing 'a' as a string and Mozilla is storing 'a' as a value. The spec doesn't nail down the storage format. Thus when IE does a.toFixed it starts out with a exact string representation while Mozilla suffers the round trip conversions described by Phil.
Comment 3•22 years ago


Kenton: thanks! I will go ahead and mark this invalid, then. If anyone objects, please reopen 
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution:  → INVALID
Comment 4•22 years ago


Marking Verified. seak_tengfong@yahoo.com: thank you for this report. This is an interesting issue that often comes up 
Status: RESOLVED → VERIFIED
Comment 5•17 years ago


Hi Phil, Glad you find this interesting. I hope reopening is appropriate. I am very reluctant to reopen this. // This should be 1.126, not 1.125. // Webkit, Spidermonkey, Futhark are all wrong. (1.1255).toFixed(3); for convenience: javascript:alert((1.1255).toFixed(3)) http://bclary.com/2004/11/07/#a15.7.4.5 Look at steps 10 and 11. 10. Let n be an integer for which the exact mathematical value of n ÷ 10^f  x is as close to zero as possible. If there are two such n, pick the larger n. 11. If n =0, let m be the string "0". Otherwise, let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes). I've tried to correctly implement the algorithm in JS. ======================================================== /** *  Fix for rounding error in JScript. *  Fix for rounding in all other engines  step 10, adjust precision. *  Range for fractionDigits expanded to {20...100} */ Number.prototype.toFixed = function(fractionDigits) { var f = parseInt(fractionDigits)  0; if( f < 20  f > 100 ) { throw new RangeError("Precision of " + f + " fractional digits is out of range"); } var x = Number(this); if( isNaN(x) ) { return "NaN"; } var s = ""; if(x <= 0) { s = ""; x = x; } if( x >= Math.pow(10, 21) ) { return s + x.toString(); } var m; // 10. Let n be an integer for which the exact mathematical value of // n ÷ 10^f  x is as close to zero as possible. // If there are two such n, pick the larger n. n = Math.round(x * Math.pow(10, f) ); if( n == 0 ) { m = "0"; } else { // let m be the string consisting of the digits of the decimal representation of n (in order, with no leading zeroes). m = n.toString(); } if( f == 0 ) { return s + m; } var k = m.length; if(k <= f) { var z = Math.pow(10, f+1k).toString().substring(1); m = z + m; k = f+1; } if(f > 0) { var a = m.substring(0, kf); var b = m.substring(kf); m = a + "." + b; } return s + m; }; I had to improvise for step 10. The algorithm in the spec omits the detail. The spec doesn't nail down the storage format, as Kentop pointed out, but it does say that the larger number should be taken. I provided no code there to "pick the larger of the two," as the spec dictates. The function above produces the result for things like: (1.1255).toFixed(3); // 1.126  correctly rounded. (0.827 ).toFixed(17); // No loss of precision. This behavior seem more practial and useful. ========================================================= Related bugs: Bug 397880 – Incorrect rounding by toFixed() for some values when last decimal digit is 5 (edit) Bug 226993 – Javascript toFixed rounds in unexpected manner (edit) Where can I find Mozilla's source code for Number.prototype.toFixed(fractionDigits)?
Status: VERIFIED → REOPENED
Resolution: INVALID → 
Comment 6•17 years ago


Test case showing original vs patched versions of Number.prototype.toFixed Allows for negative formatDigits, which was not properly supported in last post.
Comment 7•17 years ago


In the attachment, (.07).toFixed() == 0 can be ignored. This will always be true.
Comment 8•17 years ago


Mozilla and Opera have rounding errors with (1.1255).toFixed(3) and (1.1255).toFixed(17), losing precision. Safari 2 got the wrong result with (1.1255).toFixed(3), but got the correct result with (1.1255).toFixed(17). Only Safari 3 got it right the first time. The RangeError it throws on a negative fractionDigits is perfectly valid. Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification.
Attachment #283513 
Attachment is obsolete: true
Comment 9•17 years ago


Noticed that for .07, Safari 2 returns .1, but should instead return 0.1. Only Safari 3 gets the correct result.
Attachment #283857 
Attachment is obsolete: true
Comment 10•17 years ago


Instead of using Math.round, it would be more true to the spec to implement step 10 with the following: // // 10. Let n be an integer for which the exact mathematical value // of n ÷ 10^f  x is as close to zero as possible. var fTo10 = Math.pow(10, f); n = Math.floor(x * fTo10 ); // If there are two such n, pick the larger n. if (Math.abs(n / fTo10  x) >= Math.abs((n + 1) / fTo10  x)) { ++n; } // Math.round works correctly. http://bclary.com/2004/11/07/#a15.8.2.15 So should Number.prototype.toFixed. It seems likely that the native code uses something like my first patch, but chooses errorprone rounding, and does not implement rounding as Math.round does. I can guess this to be the problem; where is the !#$@ing source code? I have found no problems with the first patch I wrote using javascript's Math.round. The second approach I have written should be translated to native code. I do, however, have a problem with the current implementation, which yeilds: (1.1255).toFixed(3) // 1.125 (.1255).toFixed(3) // 0.126
Comment 11•17 years ago


toPrecision is also broken in all browsers but IE. (I did not test Opera 9.5b/Futhark Engine, but it is probably broken there, too) javascript:alert(1.1255.toPrecision(4)) toExponential is broken in all browsers but IE and Safari 3/Webkit javascript:alert(1.1255.toExponential(20)) The JS Guide documentation for toPrecision is wrong. Found with Google Search: toPrecision documentation http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Guide:Predefined_Core_Objects:Number_Object "Returns a string representing the number to a specified precision in fixedpoint notation." Incorrect and incomplete. The reference documentation is correct, but avoids discussion of the rounding errors/bugs in Mozilla: http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Global_Objects:Number  We need a meta bug with dependencies of the following: toPrecision, toFixed, toExponential For each of the above, there should be a bug rep't tests of boundary and errorprone values. * run the tests * write patches/get review/checkin et c.
Comment 13•17 years ago


Per the ECMAScript standard, the correct answer for 1.1255.toPrecision(4) is "1.125". This is what Mozilla produces. This is not a bug. The reason is that 1.1255 cannot be represented exactly as an IEEE double, so internally it is represented as 1.1254999999999999449329379785922355949878692626953125.
Status: REOPENED → RESOLVED
Closed: 22 years ago → 17 years ago
Resolution:  → INVALID
Reporter  
Comment 14•17 years ago


In reply to #13: If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the socalled standard.
Comment 15•17 years ago


We're doing that. ES4 will also have a decimal number type.
Comment 16•17 years ago


The code example, follows the ECMA 262 algorithm and gets a different result than the native code. A more desirable one, I might add. Does the example follow the algorithm incorrectly? If so, where. Where is the source code so we can have a look?
Updated•17 years ago

Assignee: khanson → general
Status: REOPENED → NEW
QA Contact: pschwartau → general
Comment 17•17 years ago


Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the very beginning  it evaluates the token "1.1255" incorrectly. This token evaluates to 1.1254999999999999449329379785922355949878692626953125. This is required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls apart after this. You also wrote "Aside from fixing the bugs in the other browsers, this patch enhances toFixed with a greater precision. This is permitted by the specification." How is this relevant here? If you're referring to the note at the end of 15.7.4.5, then please reread that note. What it's stating is that instead of throwing RangeError, toFixed is allowed to, upon request, correctly produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5, we've established that the value given is 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can see all of its digits in the Mozilla implementation of toFixed.
Status: NEW → RESOLVED
Closed: 17 years ago → 17 years ago
Resolution:  → INVALID
Comment 18•17 years ago


In the example the problem is in the emulation of step 10. The spec requires you to pick n to be an integer for which the exact mathematical value of n ÷ 10^f  x is as close to zero as possible. If there are two such n, pick the larger n. This is not implemented by n = Math.round(x * Math.pow(10, f)) because that expression has double (or triple) rounding. You round once in the * operator and again in Math.round; there may be a third rounding in Math.pow depending on the value of f. The spec requires exact arithmetic, not rounded intermediate results.
Comment 19•17 years ago


(In reply to comment #17) > Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the > very beginning  it evaluates the token "1.1255" incorrectly. This token > evaluates to 1.1254999999999999449329379785922355949878692626953125. This is > required by ECMA 262 sections 7.8.3 and 8.5. The rest of the example falls > apart after this. > > You also wrote "Aside from fixing the bugs in the other browsers, this patch > enhances toFixed with a greater precision. This is permitted by the > specification." How is this relevant here? If you're referring to the note at > the end of 15.7.4.5, then please reread that note. What it's stating is that > instead of throwing RangeError, toFixed is allowed to, upon request, correctly > produce more than 20 digits of the value it's given. By ECMA 7.8.3 and 8.5, > we've established that the value given is > 1.1254999999999999449329379785922355949878692626953125, and, in fact, you can > see all of its digits in the Mozilla implementation of toFixed. > Are you saying 1.1255.valueOf() should return 1.1254999999999999449329379785922355949878692626953125
Comment 20•17 years ago


Not only that, merely 1.1255 returns the exact mathematical value 1.1254999999999999449329379785922355949878692626953125. It happens to print as "1.1255" if you convert it back to a string, but it's actually less than the exact mathematical value 1.1255.
Comment 21•17 years ago


Garrett: now you may appreciate why decimal is coming in JS2/ES4 ;). /be
Updated•17 years ago

Status: RESOLVED → VERIFIED
Comment 22•17 years ago


(In reply to comment #19) > (In reply to comment #17) > > Yes, the example follows the ECMA algorithm incorrectly. The mistake is at the > > very beginning  it evaluates the token "1.1255" incorrectly. This token > > evaluates to 1.1254999999999999449329379785922355949878692626953125. This is > > required by ECMA 262 sections 7.8.3 and 8.5.ntation of toFixed. > > [snip] > Are you saying 1.1255.valueOf() should return > 1.1254999999999999449329379785922355949878692626953125 No, that's not what waldemar wrote. The point is that implementations must scan the 1.1255 token using extra internal precision, necessary for correct rounding to the binary precision limits of IEEE double. Nothing in the spec requires or indeed allows visible results in the language to have greater than double precision. Which is why we have bugs on file about hardware such as modern Macs, which oddly use 80bit precision in the x87 FPU but 64bit in SSE (Mac OS X does not seem to allow you to override this strange FPU configuration; if anyone knows how, please mail me). /be
Comment 23•17 years ago


Makes sense. Looking fwd to ES4.
Comment 28•14 years ago


(In reply to comment #5) Thank you Garrett for this FIX ! I need a method to obtain a mathematically correct response. All this internal stuff on the specification is not a responsible response to a practical problem encountered by users. I agree with comment #14 "If a standard produces a result opposite to what people expected, it's time to review the standard instead of maintaining the socalled standard".
Comment 30•7 years ago


here's another one: 565133671234567.89.toFixed(2) 565133671234567.88
You need to log in
before you can comment on or make changes to this bug.
Description
•