Closed Bug 1714933 Opened 2 years ago Closed 2 years ago

Update our in-tree ICU to 69

Categories

(Core :: JavaScript: Internationalization API, enhancement)

Product:
Core
Component:
JavaScript: Internationalization API
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
91 Branch
Tracking Flags:
Tracking Status
firefox91 --- fixed

People

(Reporter: anba, Assigned: anba)

References

Details

Attachments

(8 files)

Bug 1714933 - Part 1: Update ICU patches. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 2: Update in-tree ICU to release 69.1. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 4: Update expected test results. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 5: Bump minimum required ICU version to 69.1. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 6: Remove guards around previous ICU draft APIs. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 7: Updating ICU requires a clobber. r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1714933 - Part 8: Update expected canonicalisation of "no". r=tcampbell!
48 bytes, text/x-phabricator-request
Details | Review

Also updates to CLDR 39 and brings various bug fixes and improvements.

Release notes: http://site.icu-project.org/download/69

Generate the WASI patch from bug 1706949 to apply cleanly on ICU 69.

Update to ICU 69.1 by running "update-icu.sh" with "maint/maint-69" as the target.

Depends on D116967

Re-run

./make_intl_data.py numbering
./make_intl_data.py units
./make_intl_data.py langtags
./make_intl_data.py tzdata

to update numbering systems, measurement units, language tags, and time zone data.

Also remove the "US/Pacific-New" link from otherICULegacyLinks() because it
is no longer needed. Output from ./make_intl_data.py tzdata with the link
still present:

Info: Link 'US/Pacific-New -> America/Los_Angeles' can be removed from otherICULegacyLinks()

Depends on D116968

Updating to CLDR 39 means a couple of format strings have changed, update the
expected results accordingly.

unicode-bcp47-locale-ids-language-mappings.js:
Replace "no -> nb" mapping with "tl -> fil" mapping, because CLDR 39 removed
the "no -> nb" mapping [1,2]. Use the "tl -> fil" mapping as the replacement,
because it is only present in CLDR, but not in the IANA language data registry.

[1] https://unicode-org.atlassian.net/browse/CLDR-2698
[2] https://unicode-org.atlassian.net/browse/CLDR-14493

Depends on D116969

Change the system requirement to ICU 69 in order to remove some conditional
code in the next patch.

Depends on D116971

udtitvfmt_formatToResult and udtitvfmt_formatCalendarToResult were both
promoted to stable in ICU 69, so we can remove the U_HIDE_DRAFT_API for this
code.

Depends on D116972

Blocks: 1636596
Pushed by andre.bargull@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/3de963fab6d6
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/e9f2a0d262c2
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/30e9aacecf56
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/695f3bd8e875
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/692101de3873
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/40c329123b7c
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/2b4a65b9c1ac
Part 7: Updating ICU requires a clobber. r=tcampbell

"no" is no longer canonicalised to "nb" in CLDR 39.

Flags: needinfo?(andrebargull)
Pushed by andre.bargull@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/85af7c63b9c5
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/f168a2d1d391
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/62bdba475376
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fca960ad267b
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a4f4e4026174
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/c7ccabae5816
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a42831b46643
Part 7: Updating ICU requires a clobber. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/2f72875b300f
Part 8: Update expected canonicalisation of "no". r=tcampbell

I don't understand why this failed. The error logs contain:

JS_Init failed: u_init() failed

which clearly indicates an error when initialising ICU. But when I push the changes to the try-server, I can't reproduce these failures: https://treeherder.mozilla.org/jobs?repo=try&revision=51fa58f8349acd6bea81d868db9ae215a41bb262. (And I can neither reproduce the failures locally.)

The errors for "Windows 2012 x64 debug" are also baffling to me:

[task 2021-06-14T20:54:58.624Z] 6:06.51 lld-link: error: relocation against symbol in discarded section: icudt69_dat
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:698
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:722
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 lld-link: error: relocation against symbol in discarded section: icudt69_dat
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:698
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:722
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.

Later:
I've downloaded the patches from Phabricator and the ICU data file (icudt69l.dat) doesn't match what I have locally: It is an empty file on Phabricator! WTF.

Ah, okay. I think I understand what happened here: When the patches were backed out the first time, for some reason the ICU data file was replaced with an empty file: https://phabricator.services.mozilla.com/D116968?vs=450584&id=450646#change-c3mtSev48Kjf.
And when attempting to re-push the changes, the empty data file led to the errors which caused the second backout.

Flags: needinfo?(andrebargull)

Hi Atila,
the backout in https://bugzilla.mozilla.org/show_bug.cgi?id=1714933#c9 overwrote a binary file (see comment #14). I don't know which tools/programs are used to perform backouts, but maybe you know who can be informed that there seems to be an issue in the backout process which can lead to deleting files.

Thanks,
André

Flags: needinfo?(atila.butkovits)

This should be bug 1709608.

Flags: needinfo?(atila.butkovits)

Yes, thanks! That's exactly the issue observed here.

Pushed by andre.bargull@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/7bc79fc0a919
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/c73fe793c9b2
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/ec885d83c826
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/ca2fccf42e3a
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fc4205b8d565
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/920e59de0c38
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a9def04a3236
Part 7: Updating ICU requires a clobber. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fd06b2ce2393
Part 8: Update expected canonicalisation of "no". r=tcampbell
Regressions: 1719150
No longer regressions: 1719150
You need to log in before you can comment on or make changes to this bug.