1466471 - Update our in-tree ICU to 62 and update to Unicode 11

Status: RESOLVED FIXED

(Core :: JavaScript: Internationalization API, enhancement)

Description

[André Bargull [:anba]]

From http://site.icu-project.org/download/62:
> ICU 62 upgrades to Unicode 11 and to CLDR 33.1 locale data.

That means we need to combine the ICU update with the Unicode update.

The ICU release is scheduled for June 13, 2018 and Unicode 11 is scheduled for June 5, 2018 according to https://www.unicode.org/versions/Unicode11.0.0/.


Try build with ICU 62 RC and Unicode 11 preview:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7015178ce7db30a0dc9654ecefd09bd6d1aa60af

Comment 1

[NVD]

http://site.icu-project.org/

"2018-06-20: ICU 62 (62.1) released. ICU 62 upgrades to Unicode 11 and CLDR 33.1. ICU 62 includes number format skeleton support in MessageFormat, ICU4C DecimalFormat wrapping the new NumberFormatter implementation for improved performance. For details, please see the ICU 62 download page."

Comment 2

[André Bargull [:anba]]

New try build with ICU 62.1: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3e5c0dc29ea1009218bbb794b95090c038a37bd0

Comment 3

[André Bargull [:anba]]

Attachment: bug1466471-part1-icu-patches.patch

Updates one of our ICU patches to apply cleanly on ICU 62.

Comment 4

[André Bargull [:anba]]

Attachment: bug1466471-part2-icu62-update.zip

The actual update to ICU 62. (As always too large to attach as a plain diff file.)

Comment 5

[André Bargull [:anba]]

Attachment: bug1466471-part3-update-test-expectations.patch

Updates the expected values for currency and percent formatting after the CLDR update in ICU 62. I've also added tests for percent formatting with "de" for the differences between "en" and "de": "de" (among other locales) adds U+00A0 (NO-BREAK SPACE) before the percent sign, whereas "en" doesn't.

Comment 6

[André Bargull [:anba]]

Attachment: bug1466471-part4-unicode11-spidermonkey.patch

Update SpiderMonkey to Unicode 11 to stay consistent with ICU 62, which internally uses Unicode 11, too.

Comment 7

[André Bargull [:anba]]

Attachment: bug1466471-part5-unicode11-gecko.patch

Update Gecko to Unicode 11 to stay consistent with ICU 62, which internally uses Unicode 11, too.

I've also updated "genUnicodePropertyData.pl" to use "https" instead of "http" links where possible.

Comment 8

[André Bargull [:anba]]

Attachment: bug1466471-part6-clobber.patch

Clobber for ICU update.

Comment 9

[Jeff Walden [:Waldo]]

Comment on attachment 8987033 [details] [diff] [review]
bug1466471-part3-update-test-expectations.patch

Review of attachment 8987033 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/tests/non262/Intl/NumberFormat/formatToParts.js
@@ +64,5 @@
>    };
>  var usDollarsCode = new Intl.NumberFormat("en-US", usdCodeOptions);
>  
>  assertParts(usDollarsCode, 25,
> +            [Currency("USD"), Literal("\xA0"), Integer("25")]);

This is a curious change in CLDR.  Hope we don't break anyone's over-expectations.

Comment 10

[Jeff Walden [:Waldo]]

Comment on attachment 8987031 [details]
bug1466471-part2-icu62-update.zip

rs=me

Comment 11

[André Bargull [:anba]]

The V8 team found two ICU62 regressions [1,2] which probably affect us, too. I don't yet know if ICU will push a dot-release or if we'll need to apply the patches ourselves.

[1] https://ssl.icu-project.org/trac/ticket/13851
[2] https://ssl.icu-project.org/trac/ticket/13852

Comment 12

[Jonathan Kew [:jfkthame]]

AFAICS, the https://ssl.icu-project.org/trac/ticket/13851 bug won't affect case conversions used by Gecko layout (e.g. CSS text-transform:lowercase), as we don't use the ICU u_strToLower API to implement these; but it does look like it will affect toLocaleLowerCase() in JS, so we'll still need the fix for that.

Comment 13

[André Bargull [:anba]]

https://ssl.icu-project.org/trac/ticket/13852 doesn't seem to affect us, because we always set the UNUM_SIGNIFICANT_DIGITS_USED flag before modifying UNUM_MIN_SIGNIFICANT_DIGITS/UNUM_MAX_SIGNIFICANT_DIGITS. So we can proceed with the update after pulling the patch for https://ssl.icu-project.org/trac/ticket/13851.

Comment 14

[André Bargull [:anba]]

Attachment: bug1466471-part6-case-conversion-bugfix.patch

Imports the case conversion patch from ICU.

Comment 15

[André Bargull [:anba]]

Attachment: bug1466471-part7-clobber.patch

Comment 16

[André Bargull [:anba]]

Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0a16d7a82a7fdfb729ff64cd5374e812b236fc5f

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/289ff944eadc
Part 1: Update patches for ICU 62. r=Waldo
https://hg.mozilla.org/integration/mozilla-inbound/rev/2ba22d8e422a
Part 2: Update in-tree ICU to release 62.1. rs=Waldo
https://hg.mozilla.org/integration/mozilla-inbound/rev/35887b27bae5
Part 3: Update Intl tests after ICU update. r=Waldo
https://hg.mozilla.org/integration/mozilla-inbound/rev/68eca3cfb965
Part 4: Update to Unicode 11 for SpiderMonkey. r=Waldo
https://hg.mozilla.org/integration/mozilla-inbound/rev/dce28e8cfcd6
Part 5: Regenerate Unicode files after ICU update. r=jfkthame
https://hg.mozilla.org/integration/mozilla-inbound/rev/7c84b1a609df
Part 6: Import patch to fix case conversion bug. r=Waldo
https://hg.mozilla.org/integration/mozilla-inbound/rev/4a310672bb05
Part 7: Updating ICU requires a clobber. r=Waldo

Comment 18

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/289ff944eadc
https://hg.mozilla.org/mozilla-central/rev/2ba22d8e422a
https://hg.mozilla.org/mozilla-central/rev/35887b27bae5
https://hg.mozilla.org/mozilla-central/rev/68eca3cfb965
https://hg.mozilla.org/mozilla-central/rev/dce28e8cfcd6
https://hg.mozilla.org/mozilla-central/rev/7c84b1a609df
https://hg.mozilla.org/mozilla-central/rev/4a310672bb05

Comment 19

[Sylvestre Ledru [:Sylvestre]]

As always, a new version of ICU is making coverity lose his mind :)
181 new defect(s) introduced to Firefox found with Coverity Scan.
190 defect(s), reported by Coverity Scan earlier, were marked fixed in the recent build analyzed by Coverity Scan.
(even if we are now ignoring icu)