484293 - (1.95).toFixed(1) is 1.9, should be 2.0.

Status: RESOLVED DUPLICATE of bug 186563

(Core :: JavaScript Engine, defect)

Description

[Adam L. Peller]

I see this in 3.0.7 on Windows and Mac, also 3.1b2 and Minefield.

Comment 1

[Frank Wein [:mcsmurf]]

Not a valid bug as this behavior is required by the spec. See the bug I dupe this bug to for more details.

Comment 2

[Jeff Walden [:Waldo]]

Nope, this is a bug.  According to spec the result should be "2.0" if we have, in infinite-precision math:

|20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

where IEEEdouble() is the exact representation of the argument as a binary fraction.  On a test program gcc sez:

(gdb) pt d
type = union {
    double d;
    uint64_t u;
}
(gdb) p d
$1 = {
  d = 1.95, 
  u = 4611460838446019379
}
(gdb) p/x d.u
$2 = 0x3fff333333333333

We have 1 sign bit, 11 exponent bits, and 52 significand bits.  Decoding the above we have 1023 as exponent, so the value is the hex fraction 1.f333333333333.  Expanding that into a binary fraction yields:

 = 1                   (implicit)
 + (1/2+1/4+1/8+1/6)   0xf
 + (1/128+1/256)       0x3
 + (1/2048+1/4096)     0x3
 ...                   0x333333333
 + (3/256/(2**44))     0x3

The first value is:

 = 2 - IEEEdouble(1.95)
 = 2 - (1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)))
 = 1 - 15/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))
 = 1/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))

The second value is:

 = IEEEdouble(1.95) - 19/10
 = 1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10
 = 31/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10

For n and d in reduced form where n/d==sum(i from 0 to 12 exclusive, 2**(-4*i)) we have:

n=18764998447377
d=17592186044416

Then:

1/16 - 3/256*n/d      ?    31/16 + 3/256*n/d - 19/10
19/10 - 3/256*n/d     ?    15/8 + 3/256*n/d
19/10 - 15/8          ?    3/128*n/d
1/40                  ?    3/128*n/d
1/10                  ?    3/32*n/d
32/10                 ?    3*n/d
32*d                  ?    30*n

Which leaves us with:

562949953421312       ?    562949953421310

Comparing those vertically:

562949953421312 >
562949953421310

So therefore:

|20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

and 2.0 is the correct answer.

You'd better all have had as much fun reading this comment as I had running the calculations necessary to reproduce it.  :-P

Comment 3

[Boris Zbarsky [:bzbarsky]]

> so the value is the hex fraction 1.f333333333333.

With you up to here. However the exact value of 1.95 in hex would have the 3 infinitely repeated; this is a truncation of that.  So this value is strictly less than 1.95.

> |20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

Agreed on that too.

Going back, though:

> According to spec the result should be "2.0" if we have, in infinite-
> precision math:
> |20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

That's wrong.  The inequality you give says that 1.95 is further from 2.0 than from 1.9; if that's the case the result should be 1.9.  The exact spec quote:

  Let n be an integer for which the exact mathematical value of n/(10^f) - x is
  as close to zero as possible. If there are two such n, pick the larger n.

Where x is IEEEdouble(1.95) and f == 1 in this case.  So we want n/10 to be as close to our number as possible.  The calculation above shows that 19/10-x is smaller than 20/10-x, so the correct value of n is 19, which gives 1.9 as the output.

Comment 4

[Adam L. Peller]

I suspected it might be a rounding problem also, but was thrown off by the fact that IE and Safari both come up with the "right" answer, at least in this case.  Any thoughts on the inconsistency there?

Comment 5

[Adam L. Peller]

er, I meant a floating point problem.  sorry.

Comment 6

[Boris Zbarsky [:bzbarsky]]

For what it's worth, Jeff filed a webkit bug on this.

Comment 7

[Brendan Eich [:brendan]]

https://bugs.webkit.org/show_bug.cgi?id=24711

/be