Closed Bug 484293 Opened 12 years ago Closed 12 years ago

(1.95).toFixed(1) is 1.9, should be 2.0.

Categories

(Core :: JavaScript Engine, defect)

x86
macOS
defect
Not set
normal

Tracking

()

RESOLVED DUPLICATE of bug 186563

People

(Reporter: apeller, Unassigned)

Details

I see this in 3.0.7 on Windows and Mac, also 3.1b2 and Minefield.
Not a valid bug as this behavior is required by the spec. See the bug I dupe this bug to for more details.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 186563
Nope, this is a bug.  According to spec the result should be "2.0" if we have, in infinite-precision math:

|20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

where IEEEdouble() is the exact representation of the argument as a binary fraction.  On a test program gcc sez:

(gdb) pt d
type = union {
    double d;
    uint64_t u;
}
(gdb) p d
$1 = {
  d = 1.95, 
  u = 4611460838446019379
}
(gdb) p/x d.u
$2 = 0x3fff333333333333

We have 1 sign bit, 11 exponent bits, and 52 significand bits.  Decoding the above we have 1023 as exponent, so the value is the hex fraction 1.f333333333333.  Expanding that into a binary fraction yields:

 = 1                   (implicit)
 + (1/2+1/4+1/8+1/6)   0xf
 + (1/128+1/256)       0x3
 + (1/2048+1/4096)     0x3
 ...                   0x333333333
 + (3/256/(2**44))     0x3

The first value is:

 = 2 - IEEEdouble(1.95)
 = 2 - (1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)))
 = 1 - 15/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))
 = 1/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))

The second value is:

 = IEEEdouble(1.95) - 19/10
 = 1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10
 = 31/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10

For n and d in reduced form where n/d==sum(i from 0 to 12 exclusive, 2**(-4*i)) we have:

n=18764998447377
d=17592186044416

Then:

1/16 - 3/256*n/d      ?    31/16 + 3/256*n/d - 19/10
19/10 - 3/256*n/d     ?    15/8 + 3/256*n/d
19/10 - 15/8          ?    3/128*n/d
1/40                  ?    3/128*n/d
1/10                  ?    3/32*n/d
32/10                 ?    3*n/d
32*d                  ?    30*n

Which leaves us with:

562949953421312       ?    562949953421310

Comparing those vertically:

562949953421312 >
562949953421310

So therefore:

|20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

and 2.0 is the correct answer.

You'd better all have had as much fun reading this comment as I had running the calculations necessary to reproduce it.  :-P
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
> so the value is the hex fraction 1.f333333333333.

With you up to here. However the exact value of 1.95 in hex would have the 3 infinitely repeated; this is a truncation of that.  So this value is strictly less than 1.95.

> |20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

Agreed on that too.

Going back, though:

> According to spec the result should be "2.0" if we have, in infinite-
> precision math:
> |20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

That's wrong.  The inequality you give says that 1.95 is further from 2.0 than from 1.9; if that's the case the result should be 1.9.  The exact spec quote:

  Let n be an integer for which the exact mathematical value of n/(10^f) - x is
  as close to zero as possible. If there are two such n, pick the larger n.

Where x is IEEEdouble(1.95) and f == 1 in this case.  So we want n/10 to be as close to our number as possible.  The calculation above shows that 19/10-x is smaller than 20/10-x, so the correct value of n is 19, which gives 1.9 as the output.
Status: REOPENED → RESOLVED
Closed: 12 years ago12 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 186563
I suspected it might be a rounding problem also, but was thrown off by the fact that IE and Safari both come up with the "right" answer, at least in this case.  Any thoughts on the inconsistency there?
er, I meant a floating point problem.  sorry.
For what it's worth, Jeff filed a webkit bug on this.
You need to log in before you can comment on or make changes to this bug.