If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

.toFixed() seems to have a problem with math precision (returning more digits than wanted)

RESOLVED INVALID

Status

()

Core
JavaScript Engine
--
major
RESOLVED INVALID
13 years ago
13 years ago

People

(Reporter: Philip Timmermann, Unassigned)

Tracking

Trunk
PowerPC
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

13 years ago
User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040628 Firefox/0.9.1
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.7) Gecko/20040628 Firefox/0.9.1

I just did a very simple thing:

   for (i=0; i<11; i++)
   {
      alert( i + " = " + ( i * ( 100 / 255 ).toFixed(2) ) );
   }

...and got the following result:

   0 = 0
   1 = 0.39
   2 = 0.78
   3 = 1.17
   4 = 1.56
   5 = 1.9500000000000002
   6 = 2.34
   7 = 2.73
   8 = 3.12
   9 = 3.5100000000000002
   10 = 3.9000000000000004

I'm surprised that the .toFixed(2) thing isn't just able to omit the last digits
in EVERY case and give me back the value it's supposed to. It should be more
powerful than what I get now. If I state to want back a .toFixed(2) number
SpiderMonkey should be able to return it to me in any case, right?


Reproducible: Always
Steps to Reproduce:
1.
2.
3.

Comment 1

13 years ago
The problem here is not with toFixed. You are multiplying the value returned by 
toFixed again, this is adding the digits:

( 100 / 255 ).toFixed(2) => 0.39
10 * 0.39 = > 3.9000000000000004

Btw., IE6 returns 3.9000000000000003. Don't know which result is correct, I 
guess Mozilla's.
(Reporter)

Comment 2

13 years ago
I really should have seen this myself... :-(

Sorry for wasting your time...

This bug is invalid...
Status: UNCONFIRMED → RESOLVED
Last Resolved: 13 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.