1281448 - update Unicode character data to version 9.0

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P3)

Description

[Loic]

+++ This bug was initially created as a clone of Bug #1183209 +++

Unicode 9.0 is out (released 2016.06.21), so we should update our data.

Comment 1

[Axel Hecht [:Pike]]

Simon, are you up for this?

Also, no idea what priority this bug would have in the light of https://wiki.mozilla.org/Bugmasters/Process/Triage#How_Do_You_Triage

Comment 2

[Jonathan Kew [:jfkthame]]

We may want to wait until there's an ICU release with Unicode 9.0; we rely on their constants for things like script and line-break codes, so a mismatch between the version of ICU we're using and our other character-property data may be problematic.

Oh, on the other hand we may need to hold off on updating ICU, as there may be WinXP compatibility issues with that; last I heard, they were planning to drop support for XP in ICU58. So updating may be trickier than usual....

Comment 3

[Axel Hecht [:Pike]]

Waldo said that he'd backport ICU updates for XP for our in-tree copy. At least gandalf said that Waldo said .. ;-)

Comment 4

[Jeff Walden [:Waldo]]

...yeah, backporting is an uncertain task.  If we're just copying backwards CLDR data files and the like, that's one thing.  But if the CLDR data files change format -- as I understand they're doing right now, or are likely to do very soon now, to include desirable localization info *that we want to use* -- it becomes a trickier proposition.  Backporting extra parsing code as well ups the risk/complexity level.

Given that bug 1215247 finally has indications of headway on getting ICU into Firefox on the last OS/platform we ship on, it may be worth waiting for that to happen, as much as we can.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

:waldo, so when the new ICU comes out do you want us to hold back on upgrading to it until we have ICU in Android before we port it?

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

Oh, I see.

Ok, so basically - starting with Fx 53 we're dropping support for WinXP (bug 1305453), 53 opens up on 2016-11-07 which means that by the time we get ICU 58 (with no support for WinXP), we'll be able to merge it without any porting. Yay!

Comment 7

[Krasnaya Ploshchad’]

Recently ICU 58 is released.

http://blog.unicode.org/2016/10/icu-58-released.html

Comment 8

[Jonathan Kew [:jfkthame]]

Attachment: part 1 - Update character property table generator script for Unicode 9 (in particular, security/xidmodifications.txt is replaced by security/IdentifierStatus.txt and IdentifierType.txt), and adjust APIs to fit the new identifier-type property model

The ICU58 update (including Unicode 9 data) was landed in bug 1299615, so we should go ahead with the corresponding update here. It turned out to be not-quite-trivial because the format used for the UTR39 (security) data has changed.

Comment 9

[Jonathan Kew [:jfkthame]]

Attachment: part 2 - Generated files for Unicode 9.0

The generated data tables. (The two parts will need to be folded together for landing.)

Comment 10

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5d9a785a37c4cb6468ef5b3b610b8b84c4a337b7
Bug 1281448 - part 1+2 - Update character property table generator script for Unicode 9 (in particular, security/xidmodifications.txt is replaced by security/IdentifierStatus.txt and IdentifierType.txt), and adjust APIs to fit the new identifier-type property model; update the generated data files. r=m_kato

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for Android bustage:

https://hg.mozilla.org/integration/mozilla-inbound/rev/f9f15852463a68cef4cfa7b36d82f1234de8f215

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=5d9a785a37c4cb6468ef5b3b610b8b84c4a337b7
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=39135150&repo=mozilla-inbound

[task 2016-11-14T09:37:07.348042Z] 09:37:07     INFO -  In file included from /home/worker/workspace/build/src/obj-firefox/intl/unicharutil/util/internal/Unified_cpp_util_internal0.cpp:47:0:
[task 2016-11-14T09:37:07.348143Z] 09:37:07     INFO -  /home/worker/workspace/build/src/intl/unicharutil/util/nsUnicodeProperties.cpp: In function 'const nsCharProps2& GetCharProps2(uint32_t)':
[task 2016-11-14T09:37:07.348223Z] 09:37:07     INFO -  /home/worker/workspace/build/src/intl/unicharutil/util/nsUnicodeProperties.cpp:70:5: error: narrowing conversion of '-1' from 'int' to 'unsigned char' inside { } [-Werror=narrowing]
[task 2016-11-14T09:37:07.348564Z] 09:37:07     INFO -       };
[task 2016-11-14T09:37:07.351145Z] 09:37:07     INFO -       ^
[task 2016-11-14T09:37:07.351237Z] 09:37:07     INFO -  /home/worker/workspace/build/src/intl/unicharutil/util/nsUnicodeProperties.cpp:70:5: error: large integer implicitly truncated to unsigned type [-Werror=overflow]
[task 2016-11-14T09:37:07.352461Z] 09:37:07     INFO -  cc1plus: all warnings being treated as errors

Comment 12

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7cdecdce25a6ba2fa7fd1198bdd0233b057d259d
Bug 1281448 - part 1+2 - Update character property table generator script for Unicode 9 (in particular, security/xidmodifications.txt is replaced by security/IdentifierStatus.txt and IdentifierType.txt), and adjust APIs to fit the new identifier-type property model; update the generated data files. r=m_kato

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/7cdecdce25a6

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Does this still block bug 1130266 (xp-eol) when http://site.icu-project.org/download/58 states 'This is the last ICU4C release that works on Windows XP and Windows Vista.' ?

Comment 15

[Jonathan Kew [:jfkthame]]

(In reply to Nick Thomas [:nthomas] from comment #14)
> Does this still block bug 1130266 (xp-eol) when
> http://site.icu-project.org/download/58 states 'This is the last ICU4C
> release that works on Windows XP and Windows Vista.' ?

I think this was intended to depend on (not block) the XP/Vista EOL, based on earlier proposals for dropping XP support from ICU, but as we've been able to take ICU 58 (and Unicode 9) without losing XP, we can forget that dependency.

Updating to ICU 59 (when the time comes for that) will presumably be dependent on the XP EOL.