795025 - update Unicode support to v6.2

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Simon Montagu :smontagu]

+++ This bug was initially created as a clone of Bug #723509 +++

We should update the various data tables, etc., that depend on Unicode, now that v6.2 is officially released.

This mainly affects code/data under intl/, presumably, but also some in gfx/thebes/.

Quite a lot has changed since the last update, so we should also confirm that https://wiki.mozilla.org/I18n:Updating_Unicode_version is up to date

Comment 1

[Jonathan Kew [:jfkthame]]

Attachment: update Unicode character property data to Unicode 6.2.

Comment 2

[Jonathan Kew [:jfkthame]]

Attachment: update transliteration properties for Unicode 6.2.

Comment 3

[Jonathan Kew [:jfkthame]]

In principle, nsUnicodeRange is also affected, but I believe it's been way out-of-date for a long time anyway. We should just get rid of it and replace our dependencies on it with something new.

Comment 4

[Jonathan Kew [:jfkthame]]

(as per bug 723045)

Comment 5

[Simon Montagu :smontagu]

Comment on attachment 668479 [details] [diff] [review]
update Unicode character property data to Unicode 6.2.

Has this passed tryserver? I ask because I tried to patch this myself and there were all kinds of fail (https://tbpl.mozilla.org/?tree=Try&rev=1f8bc0df26dc).

Comment 6

[Jonathan Kew [:jfkthame]]

Hmm, you're right - it produces an alarming amount of orange. :( Further investigation required...

Comment 7

[Jonathan Kew [:jfkthame]]

Attachment: part 0 - update the genUnicodePropertyData tool for rearranged harfbuzz headers.

Comment 8

[Jonathan Kew [:jfkthame]]

Attachment: part 1 - update Unicode character property data to Unicode 6.2.

Comment 9

[Jonathan Kew [:jfkthame]]

Attachment: part 2 - update transliteration properties for Unicode 6.2.

Comment 10

[Jonathan Kew [:jfkthame]]

Attachment: part 2 - update transliteration properties for Unicode 6.2.

Comment 11

[Jonathan Kew [:jfkthame]]

The problem was some refactoring of harfbuzz header files that has happened since last time we did this, which meant that the data-generation tool didn't find all the constants it was looking for - particularly for General Category, IIRC. I've fixed up the tool accordingly - and added checks so that it'll die with an appropriate message if/when this happens again, instead of giving us tables full of generic default properties.

I think this should come back green now. (It looks much better locally, at least.) Tryserver run in progress at https://tbpl.mozilla.org/?tree=Try&rev=376bba3115b0.

Comment 12

[Jonathan Kew [:jfkthame]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3fb90364f6d5
https://hg.mozilla.org/integration/mozilla-inbound/rev/2c7c166cc002
https://hg.mozilla.org/integration/mozilla-inbound/rev/def584354ffc

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3fb90364f6d5
https://hg.mozilla.org/mozilla-central/rev/2c7c166cc002
https://hg.mozilla.org/mozilla-central/rev/def584354ffc

Should this have tests or is it covered by the existing ones?

Comment 14

[Jonathan Kew [:jfkthame]]

The changes in this version were very minor (a grand total of one new character; no additions to things like case-conversion that would have visible effects for us). So I don't think test updates are needed. Our existing tests correctly showed us when the data-import process didn't work properly (as per comment #5 and following).