1183209 - update Unicode character data to version 8

Status: RESOLVED FIXED

(Core :: Internationalization, defect)

Description

[Simon Montagu :smontagu]

+++ This bug was initially created as a clone of Bug #1057548 +++

Unicode 8.0 is out (released 2015.06.17), so we should update our data.

Comment 1

[Simon Montagu :smontagu]

We also need to update https://wiki.mozilla.org/I18n:Updating_Unicode_version

Comment 2

[Simon Montagu :smontagu]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b38730e78387

Comment 3

[Simon Montagu :smontagu]

Attachment: Patch 1: add new script codes to harfbuzz

Comment 4

[Simon Montagu :smontagu]

Attachment: Patch 2: update data generation tools to Unicode 8

Apart from the new scripts, xidmodification.txt has been updated and reworked with new categories. A later patch will remove the special-casing that we did up till now for some characters which is now unnecessary.

The patch also removes the obsolete gentransliterate.pl.

Comment 5

[Simon Montagu :smontagu]

Attachment: Patch 3: the machine-generated files

This is the output of the updated genUnicodePropertyData.pl and genSpecialCasingData.pl (SpecialCasing.txt hasn't changed, but we need to run genSpecialCasingData.pl to update the casing reftests)

Comment 6

[Simon Montagu :smontagu]

Attachment: Patch 4: adapt internal Xidmod types to the new types

I'll fold this into the previous patch before checkin so that there isn't an intermediate state where tests fail.

Comment 7

[Behdad Esfahbod]

HarfBuzz patch is in upstream master now.

Comment 8

[Jonathan Kew [:jfkthame]]

Comment on attachment 8633415 [details] [diff] [review]
Patch 1: add new script codes to harfbuzz

Review of attachment 8633415 [details] [diff] [review]:
-----------------------------------------------------------------

Fine for now, so we can move forward here; our next harfbuzz update will provide this anyway.

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/904481c253bf
https://hg.mozilla.org/integration/mozilla-inbound/rev/6cad0d0b5ea5
https://hg.mozilla.org/integration/mozilla-inbound/rev/aedab23a2ff9

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/904481c253bf
https://hg.mozilla.org/mozilla-central/rev/6cad0d0b5ea5
https://hg.mozilla.org/mozilla-central/rev/aedab23a2ff9