1702246 - Tolerate unmapped Big5 byte sequences in chardetng

Status: RESOLVED FIXED

(Core :: Internationalization, defect, P3)

Description

[Henri Sivonen (:hsivonen) (temporarily away from Bugzilla)]

Telemetry suggests that users in Taiwan and Hong Kong are experiencing misdetection more often than the global average, which suggests that chardetng is rejecting Big5-ish input.

Comment 1

[Henri Sivonen (:hsivonen) (temporarily away from Bugzilla)]

The said telemetry results are shown at: https://hsivonen.fi/encoding-telemetry/

Comment 2

[Henri Sivonen (:hsivonen) (temporarily away from Bugzilla)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=95576c68fa193122fe5a5e75471844621a50aa5b

Comment 3

[Henri Sivonen (:hsivonen) (temporarily away from Bugzilla)]

Attachment: Bug 1702246 - Make the encoding detector tolerate extensions to legacy CJK encodings.

This patch tries to address the issue that legacy CJK extensions have various
extended variants where the core of the encoding is compatible but the edges
are incompatible. Without this patch, we reject e.g. Big5 if it has a single
character from the UAO extension or a single Windows end-user-defined character.

Likewise for the other legacy CJK encodings.

This patch tolerates:

• All Big5 extensions (the motivating part of this patch).
• Windows EUDC for EUC-KR.
• Classic Mac OS extensions to Shift_JIS, EUC-KR, GBK, and Big5 to the
  extent practical considering conflicting definitions of what constitutes
  a lead byte in the Encoding Standard but a single-byte extension in
  Classic Mac OS.
• JIS X 0213 / 2004 extensions to Shift_JIS and EUC-JP. (It's unclear if
  these have actual deployment.)

Tolerating means that the occurrence of an extension character doesn't
disqualify a candidate but only applies a penalty to the pending score.
If there is enough other convincing content, it should be able to overcome
the penalty.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8c8448138787
Make the encoding detector tolerate extensions to legacy CJK encodings. r=emk

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/8c8448138787