1738422 - Update our in-tree ICU to 70

Status: RESOLVED FIXED

(Core :: Internationalization, enhancement)

Description

[André Bargull [:anba]]

Also updates to CLDR 40 and Unicode 14 and brings various bug fixes and improvements.

Release notes: https://icu.unicode.org/download/70

Comment 1

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 1: Update or remove ICU patches. r=#platform-i18n-reviewers!

Remove the patches for bug 1636984, bug 1648137, and bug 1714933 because the
underlying bugs were fixed in ICU 70 resp. the local patches were backports from
the ICU development branch.

And update the local patches for Sorbian to apply cleanly on ICU 70.

Comment 2

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 2: Update in-tree ICU to release 70.1. r=#platform-i18n-reviewers!

Update to ICU 70.1 by running "update-icu.sh" with "maint/maint-70" as the target.

Comment 3

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 3: Update SpiderMonkey to Unicode 14. r=tcampbell!

Update to Unicode 14 by running ./make_unicode.py --version=UNIDATA.

Comment 4

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 4: Update Gecko to Unicode 14. r=#platform-i18n-reviewers!

Update to Unicode 14 by running genUnicodePropertyData.pl and genSpecialCasingData.pl.

Depends on D129925

Comment 5

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 5: Update language tag mappings. r=#platform-i18n-reviewers!

Update language tag mappings by running ./make_intl_data.py langtags core.zip.

The direct invocation ./make_intl_data.py langtags doesn't work, because
"https://unicode.org/Public/cldr/40/core.zip" was incorrectly generated and
contains another "core.zip" file within "core.zip".

Depends on D129926

Comment 6

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 6: Update numbering systems. r=tcampbell!

Add the new numbering system "tnsa" and then update numbering systems by
by running ./make_intl_data.py numbering.

Adding "tnsa" to ECMA-402 is tracked at https://github.com/tc39/ecma402/pull/614.
ECMA-402 allows to support additional numbering systems, so even before that PR,
it's okay to support "tnsa". (https://tc39.es/ecma402/#annex-implementation-dependent-behaviour)

Depends on D129927

Comment 7

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 7: Update measurement units. r=tcampbell!

Update measurement units by running ./make_intl_data.py units.

Depends on D129928

Comment 8

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 8: Handle new narrow-quarter symbol type. r=tcampbell!

Depends on D129929

Comment 9

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 9: Add a separate UNumberFormatFields constant for the approximately sign. r=#platform-i18n-reviewers!

Our previous heuristic no longer works, so we now have to patch ICU directly to
support detecting the approximately sign.

Depends on D129930

Comment 10

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 10: Remove approximately sign detection code. r=#platform-i18n-reviewers!

This is no longer necessary after part 9.

Depends on D129931

Comment 11

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 11: Update expected test results. r=#platform-i18n-reviewers!

Depends on D129932

Comment 12

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 12: Bump minimum required ICU version to 70.1. r=#platform-i18n-reviewers!

Requiring at least ICU 70 allows to clean-up code in the next patch of this stack.

Depends on D129933

Comment 13

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 13: Remove no longer needed ICU workarounds and conditional code. r=#platform-i18n-reviewers!

Number range formatting is stable in ICU 70, which means we can remove
MOZ_INTL_HAS_NUMBER_RANGE_FORMAT and MOZ_INTL_PLURAL_RULES_HAS_SELECT_RANGE.

The following bugs have been fixed in ICU 70, so we can remove the workarounds
for them:

• https://unicode-org.atlassian.net/browse/ICU-21109
• https://unicode-org.atlassian.net/browse/ICU-21622
• https://unicode-org.atlassian.net/browse/ICU-21641
• https://unicode-org.atlassian.net/browse/ICU-21683
• https://unicode-org.atlassian.net/browse/ICU-21685
• https://unicode-org.atlassian.net/browse/ICU-21686

Depends on D129934

Comment 14

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 14: Allow a larger maximum positive exponent in ToIntlMathematicalValue. r=tcampbell!

https://unicode-org.atlassian.net/browse/ICU-21684 has been fixed in ICU 70,
so we no longer have to restrict the maximum positive exponent to 99'999. ICU
allows exponents up to 999'999'999, but formatting a number that large
requires multiple minutes in ICU. If we limit the maximum positive exponent to
9'999'999, we allow numbers much larger than just 99'999, while still
keeping the time needed to format the string at an acceptable level.

Depends on D129935

Comment 15

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 15: Updating ICU requires a clobber. r=#platform-i18n-reviewers!

Depends on D129936

Comment 16

[André Bargull [:anba]]

Attachment: Bug 1738422 - Part 4.2: Handle new script codes in GetCommonFallbackFonts. r=jfkthame!

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Hi André, these patches wedged Lando when they tried to land. We had to revert the repository to a known-good state to un-break things. Given that most of the people who know Lando are off until next week, can you please hold off on trying to re-land these patches until they're back? Feel free to reach out on Element to coordinate.

Comment 18

[André Bargull [:anba]]

Sure. Thanks for the info. I guess the conduit room is the best place to get in touch with the folks who know about Lando, right?

---

FWIW, here's the email I got from Lando about the failed request:

Your request to land D129937 failed.

See https://lando.services.mozilla.com/D129937/ for details.

Reason:
Aborting, could not apply patch buffer for 129921, 504440.
(255, b'', b"abort: abandoned transaction found!\n(run 'hg recover' to clean up transaction)")

Comment 19

[Ryan VanderMeulen [:RyanVM]]

(In reply to André Bargull [:anba] from comment #18)

Sure. Thanks for the info. I guess the conduit room is the best place to get in touch with the folks who know about Lando, right?

Yes, exactly the right place. EST is going to be the best hours to discuss during with respect to who'll be around.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ddb7978dfd7c
Part 1: Update or remove ICU patches. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/6826f1884d06
Part 2: Update in-tree ICU to release 70.1. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/8f74f6afa9db
Part 3: Update SpiderMonkey to Unicode 14. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/d1b1ebc61bd3
Part 4: Update Gecko to Unicode 14. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/c16119298a63
Part 4.2: Handle new script codes in GetCommonFallbackFonts. r=jfkthame
https://hg.mozilla.org/integration/autoland/rev/c81dbbfa120b
Part 5: Update language tag mappings. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/ee994717b6cb
Part 6: Update numbering systems. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/9939757f5b22
Part 7: Update measurement units. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/4635ffd59b2a
Part 8: Handle new narrow-quarter symbol type. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/9405f9389e29
Part 9: Add a separate UNumberFormatFields constant for the approximately sign. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/a6cbb70bc84e
Part 10: Remove approximately sign detection code. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/31f4bb2c3ca6
Part 11: Update expected test results. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/6d19255a7b6f
Part 12: Bump minimum required ICU version to 70.1. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/d65b061dd308
Part 13: Remove no longer needed ICU workarounds and conditional code. r=platform-i18n-reviewers,dminor
https://hg.mozilla.org/integration/autoland/rev/c75ac4fa3429
Part 14: Allow a larger maximum positive exponent in ToIntlMathematicalValue. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a1b606f473a0
Part 15: Updating ICU requires a clobber. r=platform-i18n-reviewers,dminor

Comment 21

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/ddb7978dfd7c
https://hg.mozilla.org/mozilla-central/rev/6826f1884d06
https://hg.mozilla.org/mozilla-central/rev/8f74f6afa9db
https://hg.mozilla.org/mozilla-central/rev/d1b1ebc61bd3
https://hg.mozilla.org/mozilla-central/rev/c16119298a63
https://hg.mozilla.org/mozilla-central/rev/c81dbbfa120b
https://hg.mozilla.org/mozilla-central/rev/ee994717b6cb
https://hg.mozilla.org/mozilla-central/rev/9939757f5b22
https://hg.mozilla.org/mozilla-central/rev/4635ffd59b2a
https://hg.mozilla.org/mozilla-central/rev/9405f9389e29
https://hg.mozilla.org/mozilla-central/rev/a6cbb70bc84e
https://hg.mozilla.org/mozilla-central/rev/31f4bb2c3ca6
https://hg.mozilla.org/mozilla-central/rev/6d19255a7b6f
https://hg.mozilla.org/mozilla-central/rev/d65b061dd308
https://hg.mozilla.org/mozilla-central/rev/c75ac4fa3429
https://hg.mozilla.org/mozilla-central/rev/a1b606f473a0