669246 - toPrecision rounds ties to even, not away from zero

Status: NEW

(Core :: JavaScript Engine, defect)

Description

[Boris Zbarsky [:bzbarsky]]

(125).toPrecision(2) returns 120 whereas per spec it should be 130 (15.7.4.7 step 10.a says "If there are two such sets of e and n, pick the e and n for which n * 10^{e–p+1} is larger" where 'n' is in this case would be either 12 or 13).

See also http://stackoverflow.com/questions/6567690/firefox-and-javascript-rounding-rules : apparently everyone else gets this right.

Comment 1

[Jeff Walden [:Waldo]]

I filed a bug in the test262 bugzilla for this, so hopefully someone will write the test and add it.

Comment 2

[Andy Earnshaw]

This problem also occurs for floating point numbers, e.g.

    123.445 .toPrecision(5)
    //-> "123.44"

V8 also has this issue with toPrecision for floating point numbers, however, whereas it doesn't have the original issue as it is posted.  I don't think that it warrants a separate issue, though, and I realise there may be some argument against fixing it (e.g. the standard floating point arithmetic problem), but the specification requires the number to be rounded correctly.

Comment 3

[Vincent Lefevre]

(In reply to Andy Earnshaw from comment #2)
> This problem also occurs for floating point numbers, e.g.
> 
>     123.445 .toPrecision(5)
>     //-> "123.44"

JavaScript uses radix 2 internally for the Number object, and 123.445 is not exactly representable. So, it will first be rounded (to some number that is presumably below 123.445). Here you are not in a "tie" case, and the result just depends on the rounding of the decimal number to a binary double-precision number (53 bits). Thus on this example, this is not a bug.

Comment 4

[Andy Earnshaw]

(In reply to Vincent Lefevre from comment #3)
> Thus on this example, this is not a bug.

I'd argue that it is a bug, since it's off-spec.  It might not be a tie from the point of view of the internal representation of the number, but it is still a tie and produces an unexpected result.

Also, IE (out of all the mainstream browsers) has managed to produce the correct result since IE 6 and possibly older, indicating that it is absolutely achievable.

Comment 5

[Vincent Lefevre]

The spec http://www.ecma-international.org/ecma-262/5.1/ seems to be clear to me:

* Section 15.7.4.7 "Number.prototype.toPrecision (precision)" says that toPrecision is applied on a Number value ("Return a String containing this Number value...").

* Section 4.3.19 "Number value" says "primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value".

Here 123.445 is a numeric literal: Section 7.8.3 "Numeric Literals", which says "A numeric literal stands for a value of the Number type. This value is determined in two steps: first, a mathematical value (MV) is derived from the literal; second, this mathematical value is rounded as described below."

The mathematical value of 123.445 is 123.445. Concerning the rounding, since there are no more than 20 significant digits, it is "the Number value for the MV (as specified in 8.5)".

* Section 8.5 "The Number Type" says that "the Number value for x" corresponds to the IEEE 754 "round to nearest" mode (the paragraph in the spec gives details on what it is).

Here, the rounding to nearest of 123.445 is a value below 123.445; this can be seen with the following C program (run on a conforming IEEE 754 implementation with double = IEEE 754 double precision):

#include <stdio.h>
int main(void)
{
  double x = 123.445;
  printf ("%a %.20g\n", x, x);
  return 0;
}

which outputs: 0x1.edc7ae147ae14p+6 123.44499999999999318

(Here you have the exact binary value of the double-precision number in hex form, followed by a corresponding decimal approximation on 20 digits.)

The toPrecision method is applied to this double-precision number, which is a bit below 123.445, so that on 5 digits, it shall give "123.44".

Comment 6

[Andy Earnshaw]

(In reply to Vincent Lefevre from comment #5)
> The spec http://www.ecma-international.org/ecma-262/5.1/ seems to be clear
> to me:

No argument there.  However, my point is that the internal representation of a number is irrelevant.  Consider Number.prototype.toString(): it doesn't return "123.444999999999993179", which is the internal representation of the number, for 123.445 -- it returns "123.445".  What is relevant is the part of the specification for `Number.prototype.toPrecision()`.

Comment 7

[Vincent Lefevre]

(In reply to Andy Earnshaw from comment #6)
> However, my point is that the internal representation of a number is irrelevant.

The internal representation is irrelevant, but the *value set* is what must be considered. I repeat:

Section 4.3.19 "Number value" says "primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value".

See https://en.wikipedia.org/wiki/Double-precision_floating-point_format about this format.

> Consider Number.prototype.toString(): it doesn't return "123.444999999999993179",
> which is the internal representation of the number, for 123.445 -- it returns "123.445".

No, 123.444999999999993179 isn't the internal representation. As I've said, it's just a 20-digit approximation (sufficient to show that the Number value is strictly less than 123.445). Moreover toString() doesn't return the exact value of the Number object.

Note that 123.445 is mathematically equal to the irreducible fraction 24689/200, and since 200 is not a power of 2, it cannot be an exact double-precision number.

I'm wondering what MSIE is doing. Could you try (123.445).toPrecision(21)? Firefox under GNU/Linux (x86_64) gives 123.444999999999993179 as expected.

Comment 8

[Andy Earnshaw]

(In reply to Vincent Lefevre from comment #7)
> (In reply to Andy Earnshaw from comment #6)
> > However, my point is that the internal representation of a number is irrelevant.
> 
> The internal representation is irrelevant, but the *value set* is what must
> be considered. I repeat:
> 
> Section 4.3.19 "Number value" says "primitive value corresponding to a
> double-precision 64-bit binary format IEEE 754 value".

Yes, I understand that, but if the steps for Number.prototype.toPrecision() are followed to the letter, is the expected output still "123.44"?  I'm just a lowly front end developer, so my C++ isn't really good enough to know what's going on here (I think the code Firefox uses is at http://mxr.mozilla.org/comm-central/source/mozilla/js/src/jsdtoa.cpp#75).
 
> I'm wondering what MSIE is doing. Could you try (123.445).toPrecision(21)?
> Firefox under GNU/Linux (x86_64) gives 123.444999999999993179 as expected.

The output for IE 11 is "123.445000000000000000", didn't test older versions as I was cheekily using the company's Sauce Labs time to check the result and don't want to push my luck ;-).

Comment 9

[Boris Zbarsky [:bzbarsky]]

> but if the steps for Number.prototype.toPrecision() are followed to the letter, is the
> expected output still "123.44"? 

Yes.  Given that x has the exact decimal value 123.44499999999999317878973670303821563720703125 (which is the exact IEEE double value closest to 123.445), in step 12 of toPrecision(5) the only valid choices for n and e are 12344 and 2.

> The output for IE 11 is "123.445000000000000000"

Then IE11 is buggy here, as far as I can tell.  Every other JS engine I have on hand (V8, SpiderMonkey, JavaScriptCore, and Carakan) has (124.445).toPrecision(5) == 123.44 and (124.445).toPrecision(21) == 123.444999999999993179, as the spec requires.

Note that it's really easy to land here with a naive implementation of toPrecision that tries multiplying the given number by 10^n for various n and comparing to integers in the range (10^(p-1), 10^p) or something instead of doing what the spec actually says.  You can somewhat tell that IE has a buggy toPrecision by doing this test (though you do have to be a bit careful, since none of those three numbers is exactly representable as a Number in JavaScript, so they _all_ have some rounding issues):

  console.log(123.445 - 123.44);
  console.log(124.45 - 123.445);

In IE, as in every other browser, this shows:

  0.0049999999999954525
  0.005000000000009663

Now can we stop hijacking this bug, please?  There are now 10 comments here, of which only two (and the shortest ones at that) have to do with the bug I actually filed and that needs fixing.

Comment 10

[Vincent Lefevre]

To come back to the original bug, note that it seems to occur only on integers, such as 125, not on numbers like 10.5, 11.5, 12.5, etc. (which are exactly representable).

Comment 11

[Boris Zbarsky [:bzbarsky]]

Ah, good catch.  That suggests some sort of fast-path being taken in the integer case or something that doesn't do the right thing.

Comment 12

[Stefan]

toPrecision is a method that supposedly applies to decimals, and so should at least be consistent on decimals: 
However, for instance
   .00345.toPrecision(2) returns 0.0034 
whereas 
   .0345.toPrecision(2) returns 0.035
This looks buggy to me.

Comment 13

[Boris Zbarsky [:bzbarsky]]

> toPrecision is a method that supposedly applies to decimals

It applies to decimal representations of ES Number instances.

> However, for instance

Did you read comment 9?

>   .00345.toPrecision(2) returns 0.0034 

When you type ".00345", the Number that results has the exact decimal representation 0.00344999999999999994171329120717928162775933742523193359375.

>   .0345.toPrecision(2) returns 0.035

When you type ".0345", the Number that results has the exact decimal representation 0.0345000000000000028865798640254070051014423370361328125.

> This looks buggy to me.

Please do read up on how IEEE double-precision floating point numbers work...  I agree it's non-intuitive if you want to think of them as "real numbers", but that's because they're not "real numbers" in any meaningful sense of the word.

Comment 14

[Stefan]

Yes I did read point #9 but continue to disagree with the philosophy that a method supposedly about precision of *decimal places* does not work with decimal strings.

Note that (345/100000).toExponential() returns 3.45e-3  as expected (it does not return "3.44999999999999994171329120717928162775933742523193359375e-3 as the philosophy in #9 appears to suggest it should). Evidently, it gets represented in a rounded form somehow. 

So, then, why should (345/100000).toPrecision(2) not return the expected 0.0035 as well?

Anyway, rather than get into a philosophical argument about decimal manipulation, I have simply written my own simple JS routine that does what one expects using string manipulation of exponential form and the fat that toExponential() does what is it supposed to do.

Comment 15

[Boris Zbarsky [:bzbarsky]]

It's not a matter of philosophy.

You should try (345/100000).toExponential(20) and (345/10000).toExponential(20).

When toExponential is called with no arguments, I believe our behavior is actually buggy per spec, unless I'm totally misreading the spec.  Per spec at http://people.mozilla.org/~jorendorff/es6-draft.html#sec-number.prototype.toexponential the subtraction n*10^{e-f} - x in step 12 substep b should be happening exactly on arbitrary-precision representations, but we're presumably doing it on double-precision floating point values.  As a result, the "f is as small as possible" criterion leads us to pick the smallest f for which the difference is 0, and since we're doing the subtraction in limited precision the difference is 0 for n == 345, f = 2, e = -3.  But again, that's a bug due to the limited precision in the subtraction.

Comment 16

[André Bargull [:anba]]

This is a regression caused by http://hg.mozilla.org/mozilla-central/rev/222c29336422 (bug 384244), re-adding the biasUp parameter will fix this issue. I don't know if adding the biasUp parameter will introduce other bugs, though.