Closed Bug 1792775 Opened 2 years ago Closed 2 years ago

Update our in-tree ICU to 72

Categories

(Core :: Internationalization, enhancement)

Product:
Core
Component:
Internationalization
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
109 Branch
Tracking Flags:
Tracking Status
firefox107 --- wontfix
firefox108 --- wontfix
firefox109 --- disabled
firefox110 --- fixed

People

(Reporter: anba, Assigned: anba)

References

(Blocks 1 open bug)

Details

Attachments

(16 files)

Bug 1792775 - Part 1: Update or remove ICU patches. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 2: Update in-tree ICU to 72. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 3: Update language tag mappings. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 4: Update measurement units. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 5: Update numbering systems. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 6: Update SpiderMonkey to Unicode 15. r=dminor!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 7: Update Gecko to Unicode 15. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 8: Remove no longer needed ICU draft API guards. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 9: Handle new script codes in GetCommonFallbackFonts. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 10: Normalise U+202F to U+0020 when parsing dates. r=dminor!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 11: Update expected test results. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 12: Bump minimum required ICU version to 72.1. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 13: Updating ICU requires a clobber. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 14: Temporarily disable some test262 tests. r=dminor!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 1.1: Remove unused "personNames" table from ICU data file. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1792775 - Part 11.1: Update expected test results in non-SpiderMonkey files. r=#platform-i18n-reviewers!
48 bytes, text/x-phabricator-request
Details | Review

Also updates to CLDR 42 and Unicode 15 and brings various bug fixes and improvements.

Release notes: https://icu.unicode.org/download/72

Summary: Update our in-tree ICU to 71 → Update our in-tree ICU to 72

Remove the patch for bug 1766220 because the local patch was just a backport
from the ICU development branch.

And update the local patches for Sorbian and WASI integration to apply cleanly
on ICU 72.

Update to ICU 72 by running ./update-icu.sh https://github.com/unicode-org/icu.git maint/maint-72.

Update language tags by running ./make_intl_data.py langtags.

Update measurement units by running ./make_intl_data.py units.

Depends on D160181

Update numbering systems by running ./make_intl_data.py numbering.

Depends on D160182

Update SpiderMonkey to Unicode 15 by running ./make_unicode.py --version=UNIDATA.

Depends on D160183

Update Gecko to Unicode 15 by running genUnicodePropertyData.pl and
genSpecialCasingData.pl.

Depends on D160184

UDAT_STANDALONE_NARROW_QUARTERS has been promoted from draft to stable.

Depends on D160185

Use the fallback path for both new script codes.

Depends on D160186

CLDR 42 replaced plain ASCII space U+0020 with U+202F when formatting dates.
For backward-compatibility reasons we want to ensure that
Date.parse(date.toLocaleString()) works when the default locale is English,
therefore we have to normalise U+202F to U+0020.

Depends on D160187

CLDR 42 replaced plain ASCII space U+0020 with U+202F when formatting dates.
This change requires to update the expected results for more tests than usual.

Depends on D160188

Require at least ICU 72, because we're using ICU APIs which are only stable
in ICU 72.

Depends on D160189

See bug 1315397 for why a clobber is necessary.

Depends on D160190

Temporarily disable some test262 tests which no longer produce the expected
results when using CLDR 42 resp. Unicode 15. We can re-enable those tests when
we next update test262 and the following PRs are all merged upstream:

Depends on D160191

(In reply to André Bargull [:anba] from comment #12)

Created attachment 9300075 [details]
Bug 1792775 - Part 11: Update expected test results. r=#platform-i18n-reviewers!

CLDR 42 replaced plain ASCII space U+0020 with U+202F when formatting dates.
This change requires to update the expected results for more tests than usual.

This is an interesting change. Do you know if any other browser has shipped it yet? I wonder if there are sites out there that do some kind of manipulation of a formatted date/time string, and depend on (ill-founded) assumptions about its contents. Or users who will be surprised when they copy/paste formatted dates somewhere and find that the "space" doesn't behave as expected -- e.g. perhaps they're copying the date from a webpage into an old form that only accepts ASCII.

(In reply to Jonathan Kew [:jfkthame] from comment #16)

This is an interesting change. Do you know if any other browser has shipped it yet?

Chrome is currently working on updating to ICU 72, too.

I wonder if there are sites out there that do some kind of manipulation of a formatted date/time string, and depend on (ill-founded) assumptions about its contents. Or users who will be surprised when they copy/paste formatted dates somewhere and find that the "space" doesn't behave as expected -- e.g. perhaps they're copying the date from a webpage into an old form that only accepts ASCII.

The output for date.toLocaleString() is localised, so it already can contain non-ASCII characters. But I've seen code which manually tries to parse the output of new Intl.DateTimeFormat("en", options).format(date). That could break with this change.

Yeah..... no question, if code depends on such things, it's inherently fragile, buggy code. But it could still look to users like "the browser updated itself, and broke my website/workflow/...". Hopefully such things are rare.

Maybe we should try to synchronize (as nearly as possible) with Chrome's schedule, given the possibility of some breakage?

Maybe we should try to synchronize (as nearly as possible) with Chrome's schedule

I would agree given the webcompat risk.

Blocks: 1796880

I've posted a comment on the Chromium bug tracker to ask for an updated schedule for their ICU-72 integration plans.

We could also try to replace U+202F with U+0020 after calling into ICU in mozilla::intl::DateTimeFormat, if it's necessary to update to ICU-72 sooner, for example because we want to support Unicode 15 to fix things like bug 1796880.

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:anba, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.

Flags: needinfo?(dminor)
Flags: needinfo?(andrebargull)

I think we're waiting on landing this to coordinate timing with Chromium's update to ICU 72.

Flags: needinfo?(dminor)

I didn't get a response in the Chromium bug tracker, but it looks like V8 was today updated to ICU 72. I guess that means the ICU 72 update will be in the 110 Chromium release, which is scheduled for Feb 7, 2023. If we land our ICU 72 update in the current Nightly, we'll have it in Firefox 110 (scheduled for Feb 14, 2023), so about the same time as the Chromium 110 release.

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=1371208#c15
[2] https://github.com/v8/v8/commit/08c469719caf45b7d1e898c9dc326b8d61cc3eb2

Flags: needinfo?(andrebargull)

We don't use the "personNames" data, so let's filter it out to reduce the ICU data file size.

Depends on D160179

Pushed by andre.bargull@gmail.com: https://hg.mozilla.org/integration/autoland/rev/838947448110 Part 1: Update or remove ICU patches. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/3ad9903f28d8 Part 1.1: Remove unused "personNames" table from ICU data file. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/afc4c848f6cd Part 2: Update in-tree ICU to 72. r=platform-i18n-reviewers,dminor
Pushed by andre.bargull@gmail.com: https://hg.mozilla.org/integration/autoland/rev/78811cdadd63 Part 3: Update language tag mappings. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/3ea531c10813 Part 4: Update measurement units. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/5e746d85926a Part 5: Update numbering systems. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/a7b39a5105a9 Part 6: Update SpiderMonkey to Unicode 15. r=dminor https://hg.mozilla.org/integration/autoland/rev/a4461db138b7 Part 7: Update Gecko to Unicode 15. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/a9aaa2387313 Part 8: Remove no longer needed ICU draft API guards. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/d396bec29ac2 Part 9: Handle new script codes in GetCommonFallbackFonts. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/8c97026ef2b5 Part 10: Normalise U+202F to U+0020 when parsing dates. r=dminor https://hg.mozilla.org/integration/autoland/rev/af6164024ade Part 11: Update expected test results. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/d810c2b2d0bd Part 11.1: Update expected test results in non-SpiderMonkey files. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/f5461baa7c5a Part 12: Bump minimum required ICU version to 72.1. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/3392ce042fb6 Part 13: Updating ICU requires a clobber. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/f2c8e2200194 Part 14: Temporarily disable some test262 tests. r=dminor

I had to push the patches in multiple steps, because Lando was always reporting an error when I tried to push everything at once. Landing the first three patches worked in the third try. I guess the other patches will also be backed out shortly, because they rely on the first three patches, which are now backed out. Sigh.

Flags: needinfo?(andrebargull)
Pushed by andre.bargull@gmail.com: https://hg.mozilla.org/integration/autoland/rev/a60b1bc33f91 Part 1: Update or remove ICU patches. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/654c5f659fed Part 1.1: Remove unused "personNames" table from ICU data file. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/7acee89c2699 Part 2: Update in-tree ICU to 72. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/486778427d64 Part 3: Update language tag mappings. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/7136c85641b7 Part 4: Update measurement units. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/0b9cf0354dc0 Part 5: Update numbering systems. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/8f5e0e60d195 Part 6: Update SpiderMonkey to Unicode 15. r=dminor https://hg.mozilla.org/integration/autoland/rev/d6300cbbf628 Part 7: Update Gecko to Unicode 15. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/6d76fe50cf35 Part 8: Remove no longer needed ICU draft API guards. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/5abc2704004e Part 9: Handle new script codes in GetCommonFallbackFonts. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/7009b35a05a0 Part 10: Normalise U+202F to U+0020 when parsing dates. r=dminor https://hg.mozilla.org/integration/autoland/rev/7db4fb376f48 Part 11: Update expected test results. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/0be61a39756f Part 11.1: Update expected test results in non-SpiderMonkey files. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/ee9cd4c8d5cd Part 12: Bump minimum required ICU version to 72.1. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/e15aea6bb46b Part 13: Updating ICU requires a clobber. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/240681aef215 Part 14: Temporarily disable some test262 tests. r=dminor

Interesting. Clang14/15 accepts "\u{...}" with -std=c++17, even though that syntax is actually from C++23.

Flags: needinfo?(andrebargull)
Pushed by andre.bargull@gmail.com: https://hg.mozilla.org/integration/autoland/rev/7c90c9d2f39c Part 1: Update or remove ICU patches. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/5ee93ac1caae Part 1.1: Remove unused "personNames" table from ICU data file. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/71a01400e4fe Part 2: Update in-tree ICU to 72. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/c92b71e1b41d Part 3: Update language tag mappings. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/8b10808f777b Part 4: Update measurement units. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/3e1f11ae3da5 Part 5: Update numbering systems. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/1c0774432aeb Part 6: Update SpiderMonkey to Unicode 15. r=dminor https://hg.mozilla.org/integration/autoland/rev/772dbda0b3a0 Part 7: Update Gecko to Unicode 15. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/a61361c04d92 Part 8: Remove no longer needed ICU draft API guards. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/6f75e695b8ea Part 9: Handle new script codes in GetCommonFallbackFonts. r=platform-i18n-reviewers,jfkthame https://hg.mozilla.org/integration/autoland/rev/ec6d6d1542df Part 10: Normalise U+202F to U+0020 when parsing dates. r=dminor https://hg.mozilla.org/integration/autoland/rev/ffd0f2c18d01 Part 11: Update expected test results. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/0ca8bf428e91 Part 11.1: Update expected test results in non-SpiderMonkey files. r=platform-i18n-reviewers,gregtatum https://hg.mozilla.org/integration/autoland/rev/9e9f7f88851a Part 12: Bump minimum required ICU version to 72.1. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/f01760fa98c6 Part 13: Updating ICU requires a clobber. r=platform-i18n-reviewers,dminor https://hg.mozilla.org/integration/autoland/rev/b12f89a6bbff Part 14: Temporarily disable some test262 tests. r=dminor

Since nightly and release are affected, beta will likely be affected too.
For more information, please visit auto_nag documentation.

status-firefox108: --- → affected
Regressions: 1803180
Regressions: 1806042

This update was reverted from Beta109 due to bug 1806042. It remains landed for 110+ at this point, though there's still some discussion happening about whether that's what we want or not.
https://hg.mozilla.org/releases/mozilla-beta/rev/94c0c172ef0b

status-firefox109: fixed → disabled
status-firefox110: --- → fixed
Regressions: 1820909
Regressions: 1821427
Regressions: 1818103
Regressions: 1820375
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: