977540 - Don't apply detectors to foreign domains

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P4)

Description

[Henri Sivonen (:hsivonen)]

Charset menu usage by locale for...
Firefox 25: https://bug965707.bugzilla.mozilla.org/attachment.cgi?id=8381393
Firefox 26: https://bug965707.bugzilla.mozilla.org/attachment.cgi?id=8381394

It's not worthwhile to pay attention to the gray rows. Those locales have so few reported sessions a even a couple of sessions with the menu used make the locales sort to the top by session percentage.

Looking at the black rows, the top ones are Traditional Chinese and Thai, which accidentally have wrong fallbacks, because the old way that localizations used to define the fallback made it easy to make the error of having the intended fallback in a generic file but having Windows, Mac and *nix-specific files always override the generic file with something unintentional. Firefox 28 fixes Traditional Chinese and Thai.

Other than that, the top locales in black are the ones that had detectors available in the menu (off by default for Simplified Chinese and Korean; on by default for Japanese, Russian and Ukrainian) in Firefox 25 and 26.

Since the Korean really has a single legacy encoding in practice, other things being equal, one should expect the situation to be similar for Korean, Hebrew and Greek. Yet, it's not. Once versions that no longer offer a Korean detector make it to the release channel, we'll learn if the difference between Korean on one hand and Hebrew and Greek on the other can be attributed to the detector.

Also note how the Bulgarian localization is doing much better than Russian or Ukrainian localizations. (The Belarusian one has too few sessions to draw conclusions.)

But in any case, now that we have per-TLD baseline guesses from bug 910211, it doesn't really make sense to let the Japanese detector run if the baseline guess isn't Shift_JIS and it doesn't make sense to let the Russian or Ukrainian detector run if the baseline guess isn't windows-1251. (Maybe this could even be stricter: Running the Japanese detector only on .jp, Russian detector only on .ru and Ukrainian detector only on .ua.)

We should probably make the detectors scoped that way. And if we do, the current UI no longer makes sense. At most a boolean pref for turning all three on or off would make sense.

Once the detectors are scoped not to run on foreign sites, we can assess if they do more harm than good even domestically.

Comment 1

[Masatoshi Kimura [:emk]]

How about "generic" TLDs (e.g. .com)?

Comment 2

[Henri Sivonen (:hsivonen)]

(In reply to Masatoshi Kimura [:emk] from comment #1)
> How about "generic" TLDs (e.g. .com)?

Whether a localization-dependent detector runs on unlabeled .com/org/net/ needs to be decided. Hence, "*Maybe* this could be stricter" in comment 0.

Comment 3

[Henri Sivonen (:hsivonen)]

(In reply to Masatoshi Kimura [:emk] from comment #1)
> How about "generic" TLDs (e.g. .com)?

My patch will let detection run on .com/net/org.

Comment 4

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c161041898618142c69fee09985416bfc01f2b68

Comment 5

[Henri Sivonen (:hsivonen)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dd33fbc34edb5df103eb70388f9e2520d9a758df

Comment 6

[Henri Sivonen (:hsivonen)]

Attachment: Don't feed chardet if the fallback encoding doesn't match the chardet-affiliated encoding

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a2df400cb88c
Avoid running Japanese, Russian and Ukrainian detectors on domains associated with different encoding legacies. r=emk.

Comment 8

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/a2df400cb88c

Comment 9

[Masatoshi Kimura [:emk]]

Attachment: enter.html