1820618 - TranslationsDocument should use a word segmenter not a regex to support CJK-like languages

Status: RESOLVED INVALID

(Firefox :: Translations, enhancement, P3)

Description

[Greg Tatum [:gregtatum]]

In toolkit/components/translations/content/translations-document.sys.mjs the TranslationsDocument segments using a whitespace regex, which works decently well in Latin script languages, but doesn't scale to other languages, like CJK languages. In order to properly implement reporting words, we need to use the Intl.Segmenter, which at this time hasn't landed yet.

Comment 1

[Greg Tatum [:gregtatum]]

This is only in reportWordsInViewport which we may just want to remove.

Comment 2

[Greg Tatum [:gregtatum]]

I suspect we should also use the browser's sentence segmenter as well, rather than ship one with Bergamot.

Comment 3

[Meera Murthy]

Hi All,

    I want to work on this bug. 

Does CJK languages mean Chinese/Japanese/Korean??
If yes, I see that these languages are not yet supported in Firefox browser. So how do we test it??
Also checking if this bug is still open, if all the information is up to date in this thread, and if someone is already working on it.

Cheers,
Meera
Outreachy

Comment 4

[Greg Tatum [:gregtatum]]

I'm sorry, but this one is not ready to be worked on yet. It is blocked by Bug 1423593.

Comment 5

[Greg Tatum [:gregtatum]]

I'm going to mark this as invalid, as this information is only being reported to the console as a debug message. The only time it is important to properly segment is if we reported this to telemetry. I filed Bug 1904418 for that work, and I will mark this one as invalid.