1793749 - (implicithan) Build ICU with the implicithan collation root

Status: RESOLVED FIXED

(Core :: JavaScript: Internationalization API, enhancement, P2)

Description

[Henri Sivonen (:hsivonen)]

ICU supports two variants of the root collation: unihan and implicithan.

unihan puts all Han characters across blocks of different ages into unified radical-stroke order, which is theoretically proper but involves explicit data.

implicithan explicitly orders the blocks (main ideograph block before Extension A, even though Extension A comes first in code point order) and then within each block implies the order from the codepoint (radical-stroke within each block), which is OK enough in practice and involves less data.

The reason why implicithan is OK enough in practice is two-fold:

1. None of the CJK locales use the root order for common characters in the respective languages. They all use tailorings, so unihan vs. implicithan is relevant only for the purpose of giving some order to characters that are so rare that the language-specific tailoring doesn't cover them.
2. Since each block, including the main ideographic block, is internally ordered by radical-stroke, the difference is irrelevant to comparison of characters that are common enough to be covered by the main ideographic block.

Firefox and Safari use unihan and Chrome, even on desktop, uses implicithan. Since Chrome uses implicithan deliberately, it's unlikely that they'd move to unihan. Since Web authors are already unable to rely on the properties of unihan, I suggest that we, too, switch to implicithan and make our data size smaller.

Demo

Comment 1

[Henri Sivonen (:hsivonen)]

Attachment: Bug 1793749 - Build ICU with the implicithan collation root.

Makes ICU data 353 KB smaller.

Comment 2

[Henri Sivonen (:hsivonen)]

anba, How should I verify that only the collation root changed?

https://treeherder.mozilla.org/jobs?repo=try&revision=873b21f93b0847edd2dbab5988f6bfa59d2e93cf

Comment 3

[André Bargull [:anba]]

I don't think ICU provides easy tools to verify this. There's a Debugging Tips section, but those tips seem a bit low level. You'd need to modify icu_sources_data.py to retain the object files and then check the generated resource files. Maybe it's easier to just create an Intl.Collator object and spot check for differences?

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6d0b00b3488c
Build ICU with the implicithan collation root. r=anba

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/6d0b00b3488c