(1.95).toFixed(1) is 1.9, should be 2.0.

RESOLVED DUPLICATE of bug 186563

Status

()

RESOLVED DUPLICATE of bug 186563
10 years ago
10 years ago

People

(Reporter: apeller, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

10 years ago
I see this in 3.0.7 on Windows and Mac, also 3.1b2 and Minefield.

Comment 1

10 years ago
Not a valid bug as this behavior is required by the spec. See the bug I dupe this bug to for more details.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 186563

Comment 2

10 years ago
Nope, this is a bug.  According to spec the result should be "2.0" if we have, in infinite-precision math:

|20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

where IEEEdouble() is the exact representation of the argument as a binary fraction.  On a test program gcc sez:

(gdb) pt d
type = union {
    double d;
    uint64_t u;
}
(gdb) p d
$1 = {
  d = 1.95, 
  u = 4611460838446019379
}
(gdb) p/x d.u
$2 = 0x3fff333333333333

We have 1 sign bit, 11 exponent bits, and 52 significand bits.  Decoding the above we have 1023 as exponent, so the value is the hex fraction 1.f333333333333.  Expanding that into a binary fraction yields:

 = 1                   (implicit)
 + (1/2+1/4+1/8+1/6)   0xf
 + (1/128+1/256)       0x3
 + (1/2048+1/4096)     0x3
 ...                   0x333333333
 + (3/256/(2**44))     0x3

The first value is:

 = 2 - IEEEdouble(1.95)
 = 2 - (1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)))
 = 1 - 15/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))
 = 1/16 - 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i))

The second value is:

 = IEEEdouble(1.95) - 19/10
 = 1 + 15/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10
 = 31/16 + 3/256*sum(i from 0 to 12 exclusive, 2**(-4*i)) - 19/10

For n and d in reduced form where n/d==sum(i from 0 to 12 exclusive, 2**(-4*i)) we have:

n=18764998447377
d=17592186044416

Then:

1/16 - 3/256*n/d      ?    31/16 + 3/256*n/d - 19/10
19/10 - 3/256*n/d     ?    15/8 + 3/256*n/d
19/10 - 15/8          ?    3/128*n/d
1/40                  ?    3/128*n/d
1/10                  ?    3/32*n/d
32/10                 ?    3*n/d
32*d                  ?    30*n

Which leaves us with:

562949953421312       ?    562949953421310

Comparing those vertically:

562949953421312 >
562949953421310

So therefore:

|20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

and 2.0 is the correct answer.

You'd better all have had as much fun reading this comment as I had running the calculations necessary to reproduce it.  :-P
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
> so the value is the hex fraction 1.f333333333333.

With you up to here. However the exact value of 1.95 in hex would have the 3 infinitely repeated; this is a truncation of that.  So this value is strictly less than 1.95.

> |20 / 10 - IEEEdouble(1.95)| > |19 / 10 - IEEEdouble(1.95)|

Agreed on that too.

Going back, though:

> According to spec the result should be "2.0" if we have, in infinite-
> precision math:
> |20 / 10 - IEEEdouble(1.95)| >= |19 / 10 - IEEEdouble(1.95)|

That's wrong.  The inequality you give says that 1.95 is further from 2.0 than from 1.9; if that's the case the result should be 1.9.  The exact spec quote:

  Let n be an integer for which the exact mathematical value of n/(10^f) - x is
  as close to zero as possible. If there are two such n, pick the larger n.

Where x is IEEEdouble(1.95) and f == 1 in this case.  So we want n/10 to be as close to our number as possible.  The calculation above shows that 19/10-x is smaller than 20/10-x, so the correct value of n is 19, which gives 1.9 as the output.
Status: REOPENED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 186563
(Reporter)

Comment 4

10 years ago
I suspected it might be a rounding problem also, but was thrown off by the fact that IE and Safari both come up with the "right" answer, at least in this case.  Any thoughts on the inconsistency there?
(Reporter)

Comment 5

10 years ago
er, I meant a floating point problem.  sorry.
For what it's worth, Jeff filed a webkit bug on this.
You need to log in before you can comment on or make changes to this bug.