1739983 - Unship Korean search collations and search root merging with language-specific tailorings

Status: RESOLVED FIXED

(Core :: JavaScript: Internationalization API, enhancement, P3)

Description

[Henri Sivonen (:hsivonen)]

ECMA-402 explicitly exposes collator usage "sort" vs. "search". However, there's no API surface for searching—only for comparison.

The search collations take space for questionable benefit in general and especially in the absence of a search API.

• Every search collation appears to contain tailoring for Thai-like scripts (presumably to undo reordering contractions that are relevant for sorting but irrelevant for equality check) and for Hangul (unclear why). It seems bad for these to take space is every search collation due to there not being a place to put "search root" differences from the root.
• Finnish and Swedish search collations are in the YAGNI department: they provide a dubious behavior in the light of user expectations: matching v to w.
• The Catalan search collation contains a tweak around L-dot to make it not match plain L for search that is otherwise configured to ignore diacritics. I don't know Catalan to be able to judge the necessity of this tweak.
• The German search collation is the phonebook collation with the tailorings for Thai-like scripts and Hangul, so the German-relevant part is available as the phonebook collation anyway, so we could return the phonebook collation proper when the German search collation is requested.
• The purpose of the Korean search collation seems to be to allow archaic Hangul to be searched by search keys entered with a modern-only IME, which is very edge casey considering that most Korean text on the Web is modern Hangul.

In the interest of Firefox binary size, we should stop exposing these to the Web and stop carrying these as part of Firefox if Web compat permits.

Firefox's own ctrl-f is not collator-based, so we don't need to carry these collations for our own search. Unfortunately, it's probably unlikely that Chrome and Safari would stop exposing search collations, since their ctrl-f is collation-based, so they are carrying search collations for that anyway.

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

Thank you for the analysis! I think this is a significant proposal that may be both valuable for the web-reality and challenging for the web compat.

Having an explicit API offered to Web Authors that is specified and implemented in a way that performs in unexpected way is an anti-pattern that we should storngly consider using significant amount of web-compat-breaking capital that we have.

My concern is here is more focused on developer experience and in result user experience of using systems which use the Collator "search" API, rather than payload cost for implementers, but the latter is also important as payload affects barrier to entry for ECMA-402 implementations.

I think the next step should be to file https://github.com/tc39/ecma402/ issue and discuss the problem with the task group.

Would you like to do that?

Comment 2

[Henri Sivonen (:hsivonen)]

I'll file an ECMA-402 issue about this.

As far as Web compat goes, there's the issue that in general we want exact behaviors even then the exact doesn't matter for use cases: For search, what well-behaved API users should care about is the case that compares equal, and the specifics of unequal shouldn't matter. So to the extent the Thai-like contraction suppression is about run-time performance and doesn't alter what strings compare equal, it shouldn't matter to well-behaved API users if the search collation order changes.

The "search root" Korean behavior is at least partially about being able to match across syllable boundaries. I'll ask Jungshik Shin about it.

Anyway, all this would be a lesser concern for data size, if there was a way to have the "search root" diff somewhere other than either root or tailoring, but having a third place would be complicated both in terms of run-time branches and builder logic. At least the Korean and Thai-like changes are conceptually easy to isolate. (And for ICU4X, isolating the Korean search root behavior into a single copy instead of making multiple copies may be the future.)

After filing, I noticed there are search root differences for Arabic and Hebrew as well. I haven't yet figured out what those are about.

(Leaving myself needinfoed for now.)

Comment 3

[Henri Sivonen (:hsivonen)]

Commented on an existing issue.

Comment 4

[Thorin [:thorin]]

IDK what this means, but FYI: in TZP I want to trap language/locale lies (e.g. from extensions), and one way to do that is to check what undefined returns in various Intl. contructors. It doesn't make a stick of difference to entropy, as resolvedOptions is the entropy - but it allows me to get close to parity (to expose lies)

For Intl.Collator I have a test - https://arkenfox.github.io/TZP/tests/collation.html - where I just added search to see if that resulted in more unique hashes (it did, by a whopping 2). The locales tested (175 supported in collator in 110+) are from a much larger set which I have reduced and which I use in all Intl. PoCs, as what I have discarded adds no further entropy - so excuse the list as being somewhat incomplete/weird

The 68 characters tested, by default, I cannot seem to improve on to gain more entropy (but I might try again now I have added search). Anyway, in the console you can type oSortvsSearch - the base control is en

On Nightly I get 111 locales where "search_is_different" (to sort), and of those most result in search matching our en control. Those that don't number 14, and if we ignore the countries, it boils down to az, bs, da, de, hr, sk, and sr. But that may be because I'm not testing enough character strings to expose more.

edit: I modified the code to use different char sets per collation: if you set sort to something like a, b and leave search as blank, the results show two results covering 22 locales

Again, IDK what this means, just thought I would share

Comment 5

[Henri Sivonen (:hsivonen)]

This seems to save about 176 KB.

Again, IDK what this means, just thought I would share

I expect the change I'm proposing here wouldn't make official-config Firefox more fingerprintable across instances. I expect it to be infeasible to make Firefox indistinguishable from other browsers via fingerprinting vectors. This would make Firefox built with --with-system-icu more distinctive from normally-built Firefox, but I expect them to be distinguishable already. If one wants non-fingerprintability, one shouldn't build with --with-system-icu.

Comment 6

[Henri Sivonen (:hsivonen)]

Attachment: WIP: Bug 1739983 - Stop exposing search collations to the Web.

Comment 7

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/jobs?repo=try&revision=9e5c2bd4c1b0c264366651cb3c5bc58c71cdffbf

Comment 8

[Henri Sivonen (:hsivonen)]

All try pushes need rebasing. Rebased:
https://treeherder.mozilla.org/jobs?repo=try&revision=705a7df84ca586cc9b1b18bb6c736f12d928af8a

Comment 9

[Henri Sivonen (:hsivonen)]

Somehow, it looks like the filter change here regressed bug 1630920.

Comment 10

[Henri Sivonen (:hsivonen)]

(In reply to Henri Sivonen (:hsivonen) from comment #9)

Somehow, it looks like the filter change here regressed bug 1630920.

Oops. I actually accidentally deleted a relevant line.

Comment 11

[Henri Sivonen (:hsivonen)]

Saving 269 KB now. That makes more sense.

Comment 12

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/jobs?repo=try&revision=96f953ef2016f3cdccab636896d5d32571a1b2e1

Comment 13

[Henri Sivonen (:hsivonen)]

Test cases test search collation for German behaving like the phonebook collation for German (for German input; tests don't test e.g. Hangul or Thai input with German locale). The German behavior change seems to be the thing that people (at least ones writing test cases) know about, so there is some risk that someone is misusing the German search collation when they want to do German phonebook sorting.

anba, if we were to make it appears as though the search collation for German existed but it actually used the phonebook collation data exactly (without the search-oriented changes for other scripts), where (on what layer) should be put the hack?

Comment 14

[André Bargull [:anba]]

Probably in NewIntlCollator: Instead of adding u-co-search, there should be something along the lines of:

if (tag.Language().EqualTo("de")) {
  if (!keywords.emplaceBack("co", cx->names().phonebk)) {
    return nullptr;
  }
}

cx->names().phonebk requires to add "phonebk" to CommonPropertyNames.h.

Comment 15

[Henri Sivonen (:hsivonen)]

something along the lines of

Thanks!

https://treeherder.mozilla.org/jobs?repo=try&revision=1d87223301921be7591668e5d3e7a66770f1ed34

Comment 16

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/jobs?repo=try&revision=de210f6d4ce20c16f1ff1a14796d8e2c9e12d62e

Comment 17

[Henri Sivonen (:hsivonen)]

Shippable aarch64 Android build with patch:
https://treeherder.mozilla.org/jobs?repo=try&revision=5ae00d247eb0fc78cb7a77320f82e0e0ad623bf0

Baseline:
https://treeherder.mozilla.org/jobs?repo=try&revision=f446f748c0e6c9cc80448eb610c8e3c33e59b4eb

Comment 18

[Henri Sivonen (:hsivonen)]

I think this level of unshipping doesn't make sense, but I think it makes sense to:

1. update to the next version of CLDR which removes the Hebrew-script search tailoring due to the relevant thing having migrated all the way to the sort root,
2. unship the Korean search collations,
3. for the remaining non-root search collations, instead of carrying a tailoring that merges the search root and a sort tailoring, use the sort tailoring,
4. for the remaining search tailorings (Catalan and Slovak), remove the search root merging,
5. remove the contraction suppressions from the search root (relevant for interactive incremental search but not relevant to full-string matching),
6. for consistency between cases 3 and 4 and the search root, remove the special-casing of the not equal sign from the search root.

Downsides: When a (Latin-script) language has a tailoring, the fuzzy rules for the Arabic and Thai scripts would not apply. The equals sign and the not equals sign would be considered to match. (It's unclear to me why these are primary-equal in the sort root.)

Comment 19

[Henri Sivonen (:hsivonen)]

Baseline:
https://treeherder.mozilla.org/jobs?repo=try&revision=26a21f170500153104f9350105f8dce90ad8766a

Without search root inheritance, without Hangul in the search root, without not equals special case, and without contraction suppressions for Thai-like scripts:
https://treeherder.mozilla.org/jobs?repo=try&revision=0e73ff4fc891331a45a5bc03881b5f64649272ab

Also without Korean search collations:
https://treeherder.mozilla.org/jobs?repo=try&revision=a87f31734bb59c17ebfe061069632df9029b3416

That's a 152 KB reduction for unshipping the root inheritance and a 200 KB additional reduction for unshipping Korean search. 352 KB total.

Comment 20

[Henri Sivonen (:hsivonen)]

(FWIW, it was a surprise to me how the 352 KB is split between root inheritance and Korean. I had assumed much more of the total to be attributable to root inheritance.)

Comment 21

[Henri Sivonen (:hsivonen)]

Bug 1937541 landed and removed Korean search collations. It also generated the data for the remaining non-Korean search collations in a way that omits size-increasing performance optimizations that are applied to non-search collation data.