2000225 - Update our in-tree ICU to 78

Status: RESOLVED FIXED

(Core :: Internationalization, enhancement)

Description

[André Bargull [:anba]]

Also updates to CLDR 48 and Unicode 17 and brings various bug fixes and improvements.

Release notes: https://unicode-org.github.io/icu/download/78.html

Comment 1

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 1: Remove no longer needed ICU patch. r=#platform-i18n-reviewers!

This issue has been fixed upstream.

Comment 2

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 2: Update ICU patch files. r=#platform-i18n-reviewers!

Update patch files to apply cleanly on ICU 78.

Comment 3

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 3: Add patches for ICU bugs. r=#platform-i18n-reviewers!

Add patches to workaround ICU-23262 and ICU-23264.

Comment 4

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 4: Update in-tree ICU to ICU78. r=#platform-i18n-reviewers!

Update ICU by running:
update-icu.sh https://github.com/unicode-org/icu.git maint/maint-78.

Comment 5

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 5: Update minimum required system ICU version to 78.1. r=#spidermonkey-reviewers!

ICU 78 adds new UDateFormatSymbolType enum constants, so we have to bump
the minimum required ICU version.

Comment 6

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 6: Handle new UDateFormatSymbolType enum values. r=#platform-i18n-reviewers!

Both enum value are still in preview state, so they are only defined when
U_HIDE_DRAFT_API is not defined.

Comment 7

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 7: Add new line break class. r=#platform-i18n-reviewers!

Use CLASS_CHARACTER for UNAMBIGUOUS_HYPHEN to match HYPHEN.

Comment 8

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 8: Handle new script codes. r=#platform-i18n-reviewers!

Treating TRADITIONAL_HAN similar to TRADITIONAL_HAN_WITH_LATIN is probably
the right choice.

Comment 9

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 9: Update language tag mappings. r=#platform-i18n-reviewers!

Update language tag mappings by running ./make_intl_data.py langtags.

Comment 10

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 10: Update numbering and measurement units. r=#spidermonkey-reviewers!

Add new numbering systems from CLDR 47.

Updated by running ./make_intl_data.py numbering and ./make_intl_data.py units.

Comment 11

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 11: Update SpiderMonkey to Unicode 17.0. r=#spidermonkey-reviewers!

Updated by running ./make_unicode.py --version=UNIDATA.

Comment 12

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 12: Enable Georgian letters in reference html output. r=#platform-i18n-reviewers!

This part was missed in bug 1761482.

Comment 13

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 13: Update Gecko to Unicode 17.0. r=#platform-i18n-reviewers!

Updated by running genUnicodePropertyData.pl, genSpecialCasingData.pl, and cldr-quotes.pl.

Comment 14

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 14: Update expected test results. r=#spidermonkey-reviewers!

Update results to match CLDR 48 resp. Unicode 17.0.

Comment 15

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 15: Update expected test results in Gecko. r=#platform-i18n-reviewers!

Comment 16

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 16: Update IDNA test cases for Unicode 17. r=valentin!

Where possible I've added alternative inputs for the tested scripts.

From https://unicode-org.github.io/icu/download/78.html:

Identifiers & security: The majority of CJK ideographs, the Bopomofo script,
and more than 1100 other pre-Unicode 17 characters, are no longer recommended
for use in identifiers by default (Identifier_Status & Identifier_Type
properties), after thorough review.

• For CJK ideographs, only 19,842 that are in modern common use (out of now
  more than 100,000) remain recommended for default identifier use.

Comment 17

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 17: Remove unused CalendarDate code. r=#platform-i18n-reviewers!

And remove unused error-status out-param from monthCodeFrom.

Clean-up in preparation for part 18.

Comment 18

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 18: Change extended-year for ICU4XChineseBasedCalendar to related ISO year. r=#platform-i18n-reviewers!

ICU 78 changed the extended year of Chinese/Dangi calendars to the related
Gregorian year. Apply the same changes for ICU4XChineseBasedCalendar for
consistency.

This also makes it easier to update ICU4X (bug 1997048).

Comment 19

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 19: Updating ICU requires a clobber. r=#platform-i18n-reviewers!

Comment 20

[André Bargull [:anba]]

Attachment: Bug 2000225 - Part 13.a: Update Unicode data extraction scripts. r=jfkthame!

• Update "genUnicodePropertyData.pl" to generate smaller sCharProp2Values table.
• Update "genSpecialCasingData.pl" to sync with manual changes to clean-up includes.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/891ab80a21c4
https://hg.mozilla.org/integration/autoland/rev/61afc3d2d07e
Part 1: Remove no longer needed ICU patch. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/af6559113407
https://hg.mozilla.org/integration/autoland/rev/bf40915b2d21
Part 2: Update ICU patch files. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/de5c2a49d3c2
https://hg.mozilla.org/integration/autoland/rev/eb74ed26433c
Part 3: Add patches for ICU bugs. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/26b6c22c8671
https://hg.mozilla.org/integration/autoland/rev/ebe49d6e1e20
Part 4: Update in-tree ICU to ICU78. r=platform-i18n-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/61441119ead3
https://hg.mozilla.org/integration/autoland/rev/8dcb5940fe57
Part 5: Update minimum required system ICU version to 78.1. r=spidermonkey-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/cd22a94e74ac
https://hg.mozilla.org/integration/autoland/rev/c5635ef4ad50
Part 6: Handle new UDateFormatSymbolType enum values. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/f1b8425e8a9b
https://hg.mozilla.org/integration/autoland/rev/5b1712d5d0a3
Part 7: Add new line break class. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/f61551e7b5fe
https://hg.mozilla.org/integration/autoland/rev/e36ce3cbc4c5
Part 8: Handle new script codes. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/ad77c5799e39
https://hg.mozilla.org/integration/autoland/rev/39b81148aba8
Part 9: Update language tag mappings. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/e70a5de082af
https://hg.mozilla.org/integration/autoland/rev/dd5692f02e3d
Part 10: Update numbering and measurement units. r=spidermonkey-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/2edad62d28a6
https://hg.mozilla.org/integration/autoland/rev/5daa5e97905e
Part 11: Update SpiderMonkey to Unicode 17.0. r=spidermonkey-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/07f3b8e02009
https://hg.mozilla.org/integration/autoland/rev/df6514d9abca
Part 12: Enable Georgian letters in reference html output. r=platform-i18n-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/769650ddc633
https://hg.mozilla.org/integration/autoland/rev/3342869e51c8
Part 13.a: Update Unicode data extraction scripts. r=jfkthame
https://github.com/mozilla-firefox/firefox/commit/062c6c1503ec
https://hg.mozilla.org/integration/autoland/rev/4eb9fcd2f7ec
Part 13: Update Gecko to Unicode 17.0. r=platform-i18n-reviewers,layout-reviewers,jfkthame
https://github.com/mozilla-firefox/firefox/commit/7cb28b8b649f
https://hg.mozilla.org/integration/autoland/rev/6f4b43965c74
Part 14: Update expected test results. r=spidermonkey-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/7e75aeefed57
https://hg.mozilla.org/integration/autoland/rev/db3aafb15803
Part 15: Update expected test results in Gecko. r=platform-i18n-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/ae30782baeea
https://hg.mozilla.org/integration/autoland/rev/60cfa90029f3
Part 16: Update IDNA test cases for Unicode 17. r=valentin,necko-reviewers
https://github.com/mozilla-firefox/firefox/commit/cb03592107ea
https://hg.mozilla.org/integration/autoland/rev/365d6f2aa28f
Part 17: Remove unused CalendarDate code. r=platform-i18n-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/03bce8a97e74
https://hg.mozilla.org/integration/autoland/rev/272e5bc216d9
Part 18: Change extended-year for ICU4XChineseBasedCalendar to related ISO year. r=platform-i18n-reviewers,dminor
https://github.com/mozilla-firefox/firefox/commit/62867aa4c6a6
https://hg.mozilla.org/integration/autoland/rev/b9c5c25d8563
Part 19: Updating ICU requires a clobber. r=platform-i18n-reviewers,dminor

Comment 22

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/61afc3d2d07e
https://hg.mozilla.org/mozilla-central/rev/bf40915b2d21
https://hg.mozilla.org/mozilla-central/rev/eb74ed26433c
https://hg.mozilla.org/mozilla-central/rev/ebe49d6e1e20
https://hg.mozilla.org/mozilla-central/rev/8dcb5940fe57
https://hg.mozilla.org/mozilla-central/rev/c5635ef4ad50
https://hg.mozilla.org/mozilla-central/rev/5b1712d5d0a3
https://hg.mozilla.org/mozilla-central/rev/e36ce3cbc4c5
https://hg.mozilla.org/mozilla-central/rev/39b81148aba8
https://hg.mozilla.org/mozilla-central/rev/dd5692f02e3d
https://hg.mozilla.org/mozilla-central/rev/5daa5e97905e
https://hg.mozilla.org/mozilla-central/rev/df6514d9abca
https://hg.mozilla.org/mozilla-central/rev/3342869e51c8
https://hg.mozilla.org/mozilla-central/rev/4eb9fcd2f7ec
https://hg.mozilla.org/mozilla-central/rev/6f4b43965c74
https://hg.mozilla.org/mozilla-central/rev/db3aafb15803
https://hg.mozilla.org/mozilla-central/rev/60cfa90029f3
https://hg.mozilla.org/mozilla-central/rev/365d6f2aa28f
https://hg.mozilla.org/mozilla-central/rev/272e5bc216d9
https://hg.mozilla.org/mozilla-central/rev/b9c5c25d8563

Comment 23

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/61afc3d2d07e
https://hg.mozilla.org/mozilla-central/rev/bf40915b2d21
https://hg.mozilla.org/mozilla-central/rev/eb74ed26433c
https://hg.mozilla.org/mozilla-central/rev/ebe49d6e1e20
https://hg.mozilla.org/mozilla-central/rev/8dcb5940fe57
https://hg.mozilla.org/mozilla-central/rev/c5635ef4ad50
https://hg.mozilla.org/mozilla-central/rev/5b1712d5d0a3
https://hg.mozilla.org/mozilla-central/rev/e36ce3cbc4c5
https://hg.mozilla.org/mozilla-central/rev/39b81148aba8
https://hg.mozilla.org/mozilla-central/rev/dd5692f02e3d
https://hg.mozilla.org/mozilla-central/rev/5daa5e97905e
https://hg.mozilla.org/mozilla-central/rev/df6514d9abca
https://hg.mozilla.org/mozilla-central/rev/3342869e51c8
https://hg.mozilla.org/mozilla-central/rev/4eb9fcd2f7ec
https://hg.mozilla.org/mozilla-central/rev/6f4b43965c74
https://hg.mozilla.org/mozilla-central/rev/db3aafb15803
https://hg.mozilla.org/mozilla-central/rev/60cfa90029f3
https://hg.mozilla.org/mozilla-central/rev/365d6f2aa28f
https://hg.mozilla.org/mozilla-central/rev/272e5bc216d9
https://hg.mozilla.org/mozilla-central/rev/b9c5c25d8563

Comment 24

[Mayank Bansal]

This lead to an increaase in the installer size on Windows opt builds by 200KB.

Comment 25

[André Bargull [:anba]]

The file increase is kind of expected, because a new ICU version means a new CLDR version, which includes more locales resp. more localisations for already supported locales. And in addition to that, this also updated to Unicode 17, which also means more data for code points added to Unicode 17.

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Is this worth adding to the Fx147 relnotes? Seems like Unicode 17 support is a change worth noting at least. Please add a relnote-firefox? nomination to this bug if you agree.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Dan, do you have thoughts on comment 26 otherwise?

Comment 28

[Dan Minor [:dminor]]

Release Note Request (optional, but appreciated)
[Why is this notable]: Updating ICU and CLDR affect localization results which are web visible, and have lead to web compat problems in the past. Hopefully this won't happen again, but having this in the release notes will help people diagnose problems.
[Affects Firefox for Android]:
[Suggested wording]: We have updated our supported ICU version to 78.
[Links (documentation, blog post, etc)]: https://github.com/unicode-org/icu/releases/tag/release-78.1