1714933 - Update our in-tree ICU to 69

Status: RESOLVED FIXED

(Core :: JavaScript: Internationalization API, enhancement)

Description

[André Bargull [:anba]]

Also updates to CLDR 39 and brings various bug fixes and improvements.

Release notes: http://site.icu-project.org/download/69

Comment 1

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 1: Update ICU patches. r=tcampbell!

Generate the WASI patch from bug 1706949 to apply cleanly on ICU 69.

Comment 2

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 2: Update in-tree ICU to release 69.1. r=tcampbell!

Update to ICU 69.1 by running "update-icu.sh" with "maint/maint-69" as the target.

Depends on D116967

Comment 3

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell!

Re-run

./make_intl_data.py numbering
./make_intl_data.py units
./make_intl_data.py langtags
./make_intl_data.py tzdata

to update numbering systems, measurement units, language tags, and time zone data.

Also remove the "US/Pacific-New" link from otherICULegacyLinks() because it
is no longer needed. Output from ./make_intl_data.py tzdata with the link
still present:

Info: Link 'US/Pacific-New -> America/Los_Angeles' can be removed from otherICULegacyLinks()

Depends on D116968

Comment 4

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 4: Update expected test results. r=tcampbell!

Updating to CLDR 39 means a couple of format strings have changed, update the
expected results accordingly.

unicode-bcp47-locale-ids-language-mappings.js:
Replace "no -> nb" mapping with "tl -> fil" mapping, because CLDR 39 removed
the "no -> nb" mapping [1,2]. Use the "tl -> fil" mapping as the replacement,
because it is only present in CLDR, but not in the IANA language data registry.

[1] https://unicode-org.atlassian.net/browse/CLDR-2698
[2] https://unicode-org.atlassian.net/browse/CLDR-14493

Depends on D116969

Comment 5

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 5: Bump minimum required ICU version to 69.1. r=tcampbell!

Change the system requirement to ICU 69 in order to remove some conditional
code in the next patch.

Depends on D116971

Comment 6

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 6: Remove guards around previous ICU draft APIs. r=tcampbell!

udtitvfmt_formatToResult and udtitvfmt_formatCalendarToResult were both
promoted to stable in ICU 69, so we can remove the U_HIDE_DRAFT_API for this
code.

Depends on D116972

Comment 7

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 7: Updating ICU requires a clobber. r=tcampbell!

Depends on D116973

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3de963fab6d6
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/e9f2a0d262c2
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/30e9aacecf56
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/695f3bd8e875
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/692101de3873
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/40c329123b7c
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/2b4a65b9c1ac
Part 7: Updating ICU requires a clobber. r=tcampbell

Comment 9

[Atila Butkovits]

Backed out for causing failures on test_ManifestProcessor_lang.html

Backout link: https://hg.mozilla.org/integration/autoland/rev/7d2575d16c1d2be25ecb3a8cfcacb7f75953f9a1

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=G6r4j6DRQ0iJL5n1yjqeuQ.0&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=2b4a65b9c1ac02bcfb1c956785198989d68d2de5&searchStr=Linux%2C18.04%2Cx64%2Cdebug%2CMochitests%2Ctest-linux1804-64%2Fdebug-mochitest-plain-e10s%2C6

Failure log: https://treeherder.mozilla.org/logviewer?job_id=342716397&repo=autoland&lineNumber=6934

Comment 10

[André Bargull [:anba]]

Attachment: Bug 1714933 - Part 8: Update expected canonicalisation of "no". r=tcampbell!

"no" is no longer canonicalised to "nb" in CLDR 39.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/85af7c63b9c5
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/f168a2d1d391
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/62bdba475376
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fca960ad267b
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a4f4e4026174
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/c7ccabae5816
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a42831b46643
Part 7: Updating ICU requires a clobber. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/2f72875b300f
Part 8: Update expected canonicalisation of "no". r=tcampbell

Comment 12

[Noemi Erli[:noemi_erli]]

Backed out 8 changesets (Bug 1714933) for causing build bustages in 1659595.js CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=2f72875b300fe1b495d63fe90682f536ed1fc142&selectedTaskRun=eY69lNiRRNGXG3ySSJ3I8Q.0

Log: https://treeherder.mozilla.org/logviewer?job_id=342744581&repo=autoland&lineNumber=60511
https://treeherder.mozilla.org/logviewer?job_id=342744560&repo=autoland&lineNumber=994

We've also started to see these failures:
https://treeherder.mozilla.org/logviewer?job_id=342745711&repo=autoland&lineNumber=1516
https://treeherder.mozilla.org/logviewer?job_id=342745710&repo=autoland&lineNumber=1516
https://treeherder.mozilla.org/logviewer?job_id=342745714&repo=autoland&lineNumber=1515
https://treeherder.mozilla.org/logviewer?job_id=342745715&repo=autoland&lineNumber=1515
https://treeherder.mozilla.org/logviewer?job_id=342745713&repo=autoland&lineNumber=1516
https://treeherder.mozilla.org/logviewer?job_id=342745716&repo=autoland&lineNumber=1515
https://treeherder.mozilla.org/logviewer?job_id=342745712&repo=autoland&lineNumber=1516
which appeared on tier1 afterwards as well

Backout: https://hg.mozilla.org/integration/autoland/rev/30ff85170f4f637ac4fd9e9fa14e039898753446

Comment 13

[André Bargull [:anba]]

I don't understand why this failed. The error logs contain:

JS_Init failed: u_init() failed

which clearly indicates an error when initialising ICU. But when I push the changes to the try-server, I can't reproduce these failures: https://treeherder.mozilla.org/jobs?repo=try&revision=51fa58f8349acd6bea81d868db9ae215a41bb262. (And I can neither reproduce the failures locally.)

The errors for "Windows 2012 x64 debug" are also baffling to me:

[task 2021-06-14T20:54:58.624Z] 6:06.51 lld-link: error: relocation against symbol in discarded section: icudt69_dat
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:698
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:722
[task 2021-06-14T20:54:58.624Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 lld-link: error: relocation against symbol in discarded section: icudt69_dat
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:698
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> referenced by z:\task_1623694336\src\intl\icu\source\common\udata.cpp:722
[task 2021-06-14T20:54:58.625Z] 6:06.51 >>> ......\config\external\icu\common\udata.obj:(struct UDataMemory * __cdecl openCommonData(char const *, int, enum UErrorCode *))
[task 2021-06-14T20:54:58.625Z] 6:06.51 PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.

---

Later:
I've downloaded the patches from Phabricator and the ICU data file (icudt69l.dat) doesn't match what I have locally: It is an empty file on Phabricator! WTF.

Comment 14

[André Bargull [:anba]]

Ah, okay. I think I understand what happened here: When the patches were backed out the first time, for some reason the ICU data file was replaced with an empty file: https://phabricator.services.mozilla.com/D116968?vs=450584&id=450646#change-c3mtSev48Kjf.
And when attempting to re-push the changes, the empty data file led to the errors which caused the second backout.

Comment 15

[André Bargull [:anba]]

Hi Atila,
the backout in https://bugzilla.mozilla.org/show_bug.cgi?id=1714933#c9 overwrote a binary file (see comment #14). I don't know which tools/programs are used to perform backouts, but maybe you know who can be informed that there seems to be an issue in the backout process which can lead to deleting files.

Thanks,
André

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This should be bug 1709608.

Comment 17

[André Bargull [:anba]]

Yes, thanks! That's exactly the issue observed here.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7bc79fc0a919
Part 1: Update ICU patches. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/c73fe793c9b2
Part 2: Update in-tree ICU to release 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/ec885d83c826
Part 3: Update numbering systems, measurement units, language tags, and time zone data. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/ca2fccf42e3a
Part 4: Update expected test results. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fc4205b8d565
Part 5: Bump minimum required ICU version to 69.1. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/920e59de0c38
Part 6: Remove guards around previous ICU draft APIs. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/a9def04a3236
Part 7: Updating ICU requires a clobber. r=tcampbell
https://hg.mozilla.org/integration/autoland/rev/fd06b2ce2393
Part 8: Update expected canonicalisation of "no". r=tcampbell

Comment 19

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7bc79fc0a919
https://hg.mozilla.org/mozilla-central/rev/c73fe793c9b2
https://hg.mozilla.org/mozilla-central/rev/ec885d83c826
https://hg.mozilla.org/mozilla-central/rev/ca2fccf42e3a
https://hg.mozilla.org/mozilla-central/rev/fc4205b8d565
https://hg.mozilla.org/mozilla-central/rev/920e59de0c38
https://hg.mozilla.org/mozilla-central/rev/a9def04a3236
https://hg.mozilla.org/mozilla-central/rev/fd06b2ce2393