1648137 - Implement Intl.NumberFormat v3

Status: RESOLVED FIXED

(Core :: JavaScript: Internationalization API, enhancement, P3)

Description

[Yulia Startsev [:yulia]]

Intl.numberFormat v3 batch of features

Comment 2

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 1: Update estimated memory numbers. r=tcampbell!

Update the memory test script now that Intl.DisplayNames is enabled by
default and remove the Intl.PluralRules specific code, because the number
formatter is now always created. And finally update the memory numbers to
match the results from current ICU.

Comment 3

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 2: Add PluralRules::SelectRange(). r=#platform-i18n-reviewers!

Add intl::NumberRangeFormat based on the existing intl::NumberFormat class
and use it to implement intl::PluralRules::SelectRange().

Most ICU number range formatting functions are still draft APIs and therefore
are guared by #ifndef U_HIDE_DRAFT_API. Instead of requiring mozilla Intl
users to also use U_HIDE_DRAFT_API, add two additional preprocessor definitions
(MOZ_INTL_PLURAL_RULES_HAS_SELECT_RANGE and MOZ_INTL_HAS_NUMBER_RANGE_FORMAT).

When MOZ_INTL_HAS_NUMBER_RANGE_FORMAT isn't defined, intl::NumberRangeFormat
is an empty class. We could also hide the complete class if the preprocessor
guard isn't defined, but the other approach results in fewer #ifdef guards.

Depends on D119741

Comment 4

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 3: Add support for Intl.PluralRules.prototype.selectRange. r=tcampbell!

Delegate to intl::PluralRules::SelectRange to implement the new selectRange
function.

Depends on D119742

Comment 5

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 4: Fully clear the decimal quantity before assigning a new value. r=#platform-i18n-reviewers!

Depends on D119743

Comment 6

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 5: Add support for roundingIncrement, roundingMode, and trailingZeroDisplay options to intl::NumberFormat. r=#platform-i18n-reviewers!

First set of new options added in the "Intl.NumberFormat v3 API" proposal.

• All ICU rounding modes are added. (The Intl proposal doesn't support "HalfOdd".)
• The rounding mode names match the names in the Intl proposal, which are
  different compared to the ICU names.
• Significant digits options are ignored when rounding increment is used. This is
  an ICU limitation. The Intl proposal will throw an error if both options are
  used at the same time.

Depends on D119744

Comment 7

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 6: Add support for roundingIncrement, roundingMode, and trailingZeroDisplay options to Intl.NumberFormat. r=tcampbell!

Support the new options in the constructor and pass them through to intl::NumberFormat.

Depends on D119745

Comment 8

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 7: Add support for new useGrouping value options to intl::NumberFormat. r=#platform-i18n-reviewers!

Change mUseGrouping to mGrouping and use an enum to represent the two
new additional modes "Always" and "Min2".

Depends on D119746

Comment 9

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 8: Add support for new useGrouping value options to Intl.NumberFormat. r=tcampbell!

The proposal changes the existing "useGrouping" option from boolean to
string. It still needs to be verified if this change is web-compatible.

Depends on D119747

Comment 10

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 9: Add support for roundingPriority option to intl::NumberFormat. r=#platform-i18n-reviewers!

The rounding priority names match the names in the Intl proposal.

Depends on D119748

Comment 11

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 10: Add support for roundingPriority option to Intl.NumberFormat. r=tcampbell!

Depends on D119749

Comment 12

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 11: Add DecimalNumber class. r=tcampbell!

The next parts will need to work on arbitrary large decimal numbers. Add a
simple helper class for this purpose.

Depends on D119750

Comment 13

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 12: Add support for formatting number strings. r=tcampbell!

The "Intl.NumberFormat v3 API" proposal adds support to format arbitrarily large
decimal numbers. For example it's now possible to format the decimal number
1e1000 by passing the string "1e1000" to the format function.

ToIntlMathematicalValue() preprocesses the number as follows:

• Non-finite values (NaN and infinities) are always stored as double values.
• Positive and negative zero are also stored as double values.
• Non-decimal strings like "0x10" are stored as BigInts, because ICU only
  supports decimal number strings.
• An error is thrown if the exponent is too large to workaround ICU limitations.

Normalising non-finite values and ±0 is done in preparation for part 17.

Depends on D119751

Comment 14

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 13: Move GetPartTypeForNumberField into NumberFormatFields.h. r=#platform-i18n-reviewers!

Change GetPartTypeForNumberField to a free function so a later part in this
patch stack can call it when processing number ranges.

Depends on D119752

Comment 15

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 14: Make NumberPart a proper struct. r=#platform-i18n-reviewers!

In the next part of this patch stack, we want to record the "source" of a
number part. The source is used when formatting number ranges to record if the
formatted number part belongs to the start resp. end of the number range.

Depends on D119753

Comment 16

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 15: Record the source of a number part. r=#platform-i18n-reviewers!

Add NumberPartSource to NumberPart in preparation for part 15.

Also changes the underlying type of NumberPartType to int16_t to ensure the
size of NumberPart doesn't change on 32-bit platforms when adding
NumberPartSource. (Note: sizeof(NumberPart) is 16 on 64-platforms with or
without this change.)

Depends on D119754

Comment 17

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 16: Allow to format non-decimal number strings. r=#platform-i18n-reviewers!

Remove this Java compatibility restriction to be able to format ranges starting
resp. ending at infinity.

Depends on D119755

Comment 18

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 17: Add separate number range options struct. r=#platform-i18n-reviewers!

Depends on D119756

Comment 19

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 18: Add decimal number and formatToParts support to NumberRangeFormat. r=#platform-i18n-reviewers!

These new functions match the similarily named functions on intl::NumberFormat.

Also add some assertions to NumberFormat::formatToParts() when called with
decimal number strings to ensure non-finite number strings aren't used here,
because we always pass Nothing() for the number in this method.

Depends on D119757

Comment 20

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 19: Make NewNumberFormat a template function and process NumberPartSource. r=tcampbell!

Split these two changes from the next part, so that the next part will only add new functionality:

• Make NewNumberFormat a template function in preparation for the next part.
• Process NumberPartSource in FormattedNumberToParts.

Drive-by change:
Add trailing commas JS{Function,Property}Spec arrays to ensure clang-format
formats these more nicely, which will also give a better diff for the next
part.

Depends on D119758

Comment 21

[André Bargull [:anba]]

Attachment: Bug 1648137 - Part 20: Support number range formatting in Intl.NumberFormat. r=tcampbell!

The number range validation is a bit tricky, I've used multiple lambda functions
to closely mirror the specification (PartitionNumberRangePattern, steps 1-4), so
it should be easier to verify the implementation is correct.

One thing is still missing:
The specification requires to mark the approximation sign with the type
"approximatelySign" (cf. FormatApproximately, step 3). This isn't currently
implemented, because ICU doesn't support the necessary functionality.

Depends on D119759

Comment 22

[André Bargull [:anba]]

This is still missing more tests and I don't yet know how to implement the "approximatelySign" type. Probably we have to apply some heuristics to figure out which "literal" part is the approximation sign.

Comment 23

[Dan Minor [:dminor]]

:anba, thanks for working on this! Please flag platform-i18-reviewers for the reviews. I'm happy to review, but it would be good to give others an opportunity to have a look as well.

Comment 24

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/95b21af5cbb5
https://hg.mozilla.org/mozilla-central/rev/29b821e72d18
https://hg.mozilla.org/mozilla-central/rev/7378d1fa53d2
https://hg.mozilla.org/mozilla-central/rev/058d7ce44a39
https://hg.mozilla.org/mozilla-central/rev/3190562ddbf1
https://hg.mozilla.org/mozilla-central/rev/15c0ce41fb2b
https://hg.mozilla.org/mozilla-central/rev/75593dd7e670
https://hg.mozilla.org/mozilla-central/rev/1e59af459081
https://hg.mozilla.org/mozilla-central/rev/9f6017cf64dc
https://hg.mozilla.org/mozilla-central/rev/056920d890e1
https://hg.mozilla.org/mozilla-central/rev/65d646a0cd54
https://hg.mozilla.org/mozilla-central/rev/76c1abb8f043
https://hg.mozilla.org/mozilla-central/rev/a4a211c7aa69
https://hg.mozilla.org/mozilla-central/rev/03af8af6c44d
https://hg.mozilla.org/mozilla-central/rev/42dec0773d22
https://hg.mozilla.org/mozilla-central/rev/48d2eb0be8a6
https://hg.mozilla.org/mozilla-central/rev/e459419457d7
https://hg.mozilla.org/mozilla-central/rev/6da098e83be0
https://hg.mozilla.org/mozilla-central/rev/d4b1ac3710f2
https://hg.mozilla.org/mozilla-central/rev/7f44a968af7f

Comment 25

[Hamish Willee]

FYI Docs work for this can be tracked in https://github.com/mdn/content/issues/8617

A question:
https://github.com/tc39/proposal-intl-numberformat-v3#rounding-priority describes roundingPriority as a way to resolve rounding conflicts between the result when specifying fractional and significant digits. However I think it is perhaps more general than that.

When I run this code that sets roundingPriority: "morePrecision", it ignores the maximumSignificantDigits: 2 and the result is the original number with three fractional digits. If I set auto/less precision the specified digits are used. So we're not really rounding any more. Is this intended?

let nf = new Intl.NumberFormat("en", {
    maximumSignificantDigits: 2,
    roundingPriority: "morePrecision",
    //roundingPriority: "lessPrecision"
 
  });

console.log(nf.format(4.331));
console.log(nf.format(4.344));
console.log(nf.format(4.35));
console.log(nf.format(4.378));

Comment 26

[André Bargull [:anba]]

Based on this issue from the spec proposal repository and the corresponding meeting notes, the roundingPriority property will be slightly changed to be less confusing for certain cases.

But for your concrete example, it's also necessary to take the default value for maximumFractionDigits into account:

js> new Intl.NumberFormat("en", {maximumSignificantDigits: 2}).resolvedOptions()
({..., minimumSignificantDigits:1, maximumSignificantDigits:2, roundingPriority:"auto"})
js> new Intl.NumberFormat("en", {maximumSignificantDigits: 2, roundingPriority: "morePrecision"}).resolvedOptions()
({..., minimumFractionDigits:0, maximumFractionDigits:3, minimumSignificantDigits:1, maximumSignificantDigits:2, roundingPriority:"morePrecision"})

(I've removed all duplicate properties which are irrelevant for roundingPriority.)

When roundingPriority isn't equal to "auto", minimumFractionDigits and maximumFractionDigits will be set to their default values when they're absent. In your example, maximumFractionDigits is set to 3, which should explain why three fractional digits are used.

Comment 27

[Hamish Willee]

Thanks very much @Andre, that makes sense.

So my understanding on the issue is that previously the minimum values were being ignored for determining the digits. However now they will be used so setting minimum digits to 2 gives you a 1.00 - which is "higher precision" than 1.0. Right?

With respect to my question and your response. THanks! The fact that the default values apply in certain cases makes the behaviour pretty unintuitive when you look at code. I'll try to capture this in some examples.

Comment 28

[André Bargull [:anba]]

(In reply to Hamish Willee from comment #27)

So my understanding on the issue is that previously the minimum values were being ignored for determining the digits. However now they will be used so setting minimum digits to 2 gives you a 1.00 - which is "higher precision" than 1.0. Right?

Yes, exactly right!

With respect to my question and your response. THanks! The fact that the default values apply in certain cases makes the behaviour pretty unintuitive when you look at code. I'll try to capture this in some examples.

Great idea!

Comment 29

[Hamish Willee]

@Andre
Any idea when the change for precision will take place/go into firefox? My example code indicates that it doesn't work this way right now in FF (last two items seem to be ignoring the precision value).

const minFracNF = new Intl.NumberFormat("en", {minimumFractionDigits: 2} )
console.log(`minF2: ${minFracNF.format(1)}`);
// > "minF2: 1.00"
//console.log(minFracNF.resolvedOptions())

const minSigNS = new Intl.NumberFormat("en", {minimumSignificantDigits: 2} )
console.log(`minS2: ${minSigNS.format(1)}`);
// > "minS2: 1.0"
//console.log(minSigNS.resolvedOptions())

const minFracSigNS = new Intl.NumberFormat("en", {minimumFractionDigits: 2, minimumSignificantDigits: 2} )
console.log(`minF2S2auto: ${minFracSigNS.format(1)}`);
// > "minF2S2auto: 1.0"
// console.log(minFracSigNS.resolvedOptions())

const minFracSigNSless = new Intl.NumberFormat("en", {roundingPriority: 'lessPrecision', minimumFractionDigits: 2, minimumSignificantDigits: 2} )
console.log(`minF2S2less: ${minFracSigNSless.format(1)}`);
// > "minF2S2less: 1.00"
//console.log(minFracSigNSless.resolvedOptions())

const minFracSigNSmore = new Intl.NumberFormat("en", {roundingPriority: 'morePrecision', minimumFractionDigits: 2, minimumSignificantDigits: 2} )
console.log(`minF2S2more: ${minFracSigNSmore.format(1)}`);
// > "minF2S2more: 1.0"
//console.log(minFracSigNSmore.resolvedOptions())

If this is going to change, will there be releases that do both the old and new way?

Comment 30

[André Bargull [:anba]]

The proposal changes are currently Nightly-only. There aren't any concrete plans to ship them to Release until the proposal specification itself is finished. (Because we don't want to end up with a situation where users rely on specific behaviour which is known that it'll change.) Our implementation relies on ICU and this comment indicates that ICU internal changes are necessary. I don't if the ICU changes already happened resp. when they'll happen. ICU is released twice per year. The next release should be around October, so it seems likely that we have to wait until then before we can ship our implementation to Release.

Comment 31

[Hamish Willee]

Thanks André , but what I'm saying is that the text code I posted above shows that the proposal changes to take account of minimum values are NOT in Nightly 104.0a1 (2022-07-24) .

It actually looks like lessPrecision and morePrecision have been implemented wrong - i.e. less precision gives you more zeros.

> "minF2: 1.00"
> "minS2: 1.0"
> "minF2S2auto: 1.0"
> "minF2S2less: 1.00"
> "minF2S2more: 1.0"  => This is the "more precision" but as you can see it has lower precision than the one above, which is "lessPrecision".

And for the last one we see the object has this "moreResolution" resolution set - so we would expect the last two items to be reversed.

> Object { locale: "en", numberingSystem: "latn", style: "decimal", minimumIntegerDigits: 1, minimumFractionDigits: 2, maximumFractionDigits: 3, minimumSignificantDigits: 2, maximumSignificantDigits: 21, useGrouping: "auto", notation: "standard", signDisplay: "auto", roundingMode: "halfExpand", roundingIncrement: 1, trailingZeroDisplay: "auto", roundingPriority: "morePrecision" }

So either I'm not "getting it" or this isn't working. Would love your advice

Comment 32

[André Bargull [:anba]]

That example exactly matches what's discussed in https://github.com/tc39/proposal-intl-numberformat-v3/issues/96. According to the current spec text "lessPrecision" returning "1.00" and "morePrecision" returning "1.0" is correct. But because that's quite confusing, the spec will be changed to reflect the more natural behaviour of "lessPrecision" returning "1.0" and "morePrecision" returning "1.00". The spec changes for that change haven't yet been written and neither have there been any ICU changes.

Comment 33

[Hamish Willee]

Thanks! Clear now.