478796 - Number.toFixed() returning incorrect values for certain large numbers

Reporter

Description

•

16 years ago

var x = 1000000000000000128; print(x); // 1000000000000000100 - strange but same as spidermonkey print(x.toFixed(0)); // 1000000000000000100 - sm returns: 1000000000000000128 print(x.toFixed(1)); // 1000000000000000130.0 - sm returns 1000000000000000128.0 Actual Results: see above Expected Results: same as spidermonkey Workaround: none tofixed works as expected for args 2-20 Transferred Comments: JIRA Migration Admin - Mon Jun 04 12:33:00 CDT 2007 Milestone ID = 1378 Milestone = FP 10 - Code Complete Build ID = 22846 Build = AVMPLUS_1_0_0_d686 Fix Build ID = null Fix Build = null JIRA Migration Admin - Mon Jun 04 12:33:00 CDT 2007 Move from BugDB issue number 188975 BugDB Migration - Mon Jun 04 12:33:00 CDT 2007 [cpeyer 10/20/06] Entered Bug. [cpeyer 10/20/06] Sending to QRB [cnuuja 10/20/06] This is probably not a compliance bug, but rather related to differences in how spidermonkey and (howler?) round numbers with more precision than can fit exactly into a double. Someone should see why our toFixed(1) output is different, but keep this in mind: First of all, this is a really huge number. Don't be decieved by the fact that its an integer. The number is just larger than can accurately fit in the 52 bits of full precision in a double (an excellent test case). It equivalent to the following in base 2: 110111100000101101101011001110100111011001000000000010000000 Only the first 52 digits fit exactly. 1101111000001011011010110011101001110110010000000000 10000000 that means we need to use some of the 11 bits of base 2 exponent in the double to fully represent this number. Note that there's a 1 just after that at the 53 bit, making the number an odd rather than an even number. Since the exponent is base 2, it can't represent odd numbers (it just multiplies the mantissa by a power of 2), so we're goint to have to round that 1 up or down. That's why the number displays different in Spidermonkey (and the same as Spidermonkey in Howler). Not sure why the toFixed representation is different, that may be a bug or just an o.k. (spec-wise) implementation difference. In any case, seems like a fp10 bug to me Mark DeMichele - Thu Sep 18 09:29:25 CDT 2008 This bug is also related to FP-557. Please see my last comment. It fails for any precision. If the first significant digit it > 5 and you are expecting to get a 0 as a result, you will get a 1 in the last precise spot. Here are some examples. This is VERY BAD. I CAN'T BELIEVE THIS HASN'T BEEN ADDRESSED YET. Numerically, it is absolutely wrong and since flash is no long used to make pretty animations, I think someone really needs to take this seriously and fix it. v=0.00005 v.toFixed(2)=0.01 v=0.00007 v.toFixed(2)=0.01 v=0.00009 v.toFixed(2)=0.01 v=5e-7 v.toFixed(2)=0.00 v=7e-7 v.toFixed(2)=0.01 v=9e-7 v.toFixed(2)=0.01 v=0.00005 v.toFixed(3)=0.001 v=0.00007 v.toFixed(3)=0.001 v=0.00009 v.toFixed(3)=0.001 v=5e-7 v.toFixed(3)=0.000 v=7e-7 v.toFixed(3)=0.001 v=9e-7 v.toFixed(3)=0.001 Trevor Baker - Tue Jan 27 16:41:36 CST 2009 send to internal review for prioritization This bug transferred from: http://bugs.adobe.com/jira/browse/ASC-2293