.toFixed() seems to have a problem with math precision (returning more digits than wanted)

RESOLVED INVALID

Status

()

--
major
RESOLVED INVALID
15 years ago
15 years ago

People

(Reporter: pepto, Unassigned)

Tracking

Trunk
PowerPC
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

15 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040628 Firefox/0.9.1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040628 Firefox/0.9.1

I just did a very simple thing:

   for (i=0; i<11; i++)
   {
      alert( i + " = " + ( i * ( 100 / 255 ).toFixed(2) ) );
   }

...and got the following result:

   0 = 0
   1 = 0.39
   2 = 0.78
   3 = 1.17
   4 = 1.56
   5 = 1.9500000000000002
   6 = 2.34
   7 = 2.73
   8 = 3.12
   9 = 3.5100000000000002
   10 = 3.9000000000000004

I'm surprised that the .toFixed(2) thing isn't just able to omit the last digits
in EVERY case and give me back the value it's supposed to. It should be more
powerful than what I get now. If I state to want back a .toFixed(2) number
SpiderMonkey should be able to return it to me in any case, right?


Reproducible: Always
Steps to Reproduce:
1.
2.
3.

Comment 1

15 years ago
The problem here is not with toFixed. You are multiplying the value returned by 
toFixed again, this is adding the digits:

( 100 / 255 ).toFixed(2) => 0.39
10 * 0.39 = > 3.9000000000000004

Btw., IE6 returns 3.9000000000000003. Don't know which result is correct, I 
guess Mozilla's.
(Reporter)

Comment 2

15 years ago
I really should have seen this myself... :-(

Sorry for wasting your time...

This bug is invalid...
Status: UNCONFIRMED → RESOLVED
Last Resolved: 15 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.