1830176 - Ensure language tag correctness

Status: RESOLVED FIXED

(Firefox :: Translations, defect)

Description

[Erik Nordin [:nordzilla]]

The language tags returned by fastText language detection are codes from the ISO639-3 specification:

https://iso639-3.sil.org/code_tables/639/data

The language tags that we use internally in Firefox are from the BCP47 specification:

https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry

We should ensure that all of the possible mappings from fastText are valid.

Secondly, the existing code has been created with the notion that language tags are two characters. This is true of all of the languages that Firefox Translations currently supports, however BCP47 language tags can be more than two characters long.

We should ensure that the code can support language tags of any length.

Comment 1

[Erik Nordin [:nordzilla]]

Attachment: Bug 1830176 - Reword languageLabel to langTag r=gregtatum!

Ensure that the terminology of "langTag" is consistent across
all of the Firefox Translations code. Some code was previously
using "languageLabel" which is a term used by fastText.

Comment 2

[Erik Nordin [:nordzilla]]

Attachment: Bug 1830176 - Improve langTag formatting from fastText r=gregtatum!

Covers edge cases when converting fastText language tags
to the language tags used internally by Firefox.

Depends on D176584

Comment 3

[Erik Nordin [:nordzilla]]

Attachment: Bug 1830176 - Support langTags longer than two characters r=gregtatum!

Ensures that Firefox Translations supports langTags
of more than two characters. All of our currently supported
languages have two-character tags, but future languages
may be longer.

Depends on D176585

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/08155114af96
Reword languageLabel to langTag r=gregtatum
https://hg.mozilla.org/integration/autoland/rev/66762cdff9ef
Improve langTag formatting from fastText r=gregtatum
https://hg.mozilla.org/integration/autoland/rev/fe53ffc783bf
Support langTags longer than two characters r=gregtatum

Comment 5

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/08155114af96
https://hg.mozilla.org/mozilla-central/rev/66762cdff9ef
https://hg.mozilla.org/mozilla-central/rev/fe53ffc783bf