739861 - Add support for script subtags

Status: NEW

(Core :: Spelling checker, enhancement)

Description

[Gordon P. Hemsley [:GPHemsley]]

A search on MXR for 'Names.properties' suggests that languageNames.properties and regionNames.properties are only really used for the spellchecker:

http://mxr.mozilla.org/mozilla-central/search?string=Names.properties&find=&findi=&filter=^[^\0]*%24&hitlimit=&tree=mozilla-central

As such, there seems to be little reason to not add support for the script subtags by adding a scriptNames.properties file and amending InlineSpellChecker.jsm to use it (after bug 730209 is fixed, of course).

This will slightly increase the burden on localizers, but there are only ~160 script subtags that would have to be localized (and that data is likely already available in CLDR).

This (or a reasonable facsimile thereof) is the file that would be added:
https://github.com/GPHemsley/BCP47/blob/master/scriptNames.properties

Comment 1

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Add support for BCP 47 script subtags

Adds support for parsing BCP 47 script subtags in the spellchecker, and adds the file toolkit/locales/en-US/chrome/global/scriptNames.properties, which adds 155 strings to be localized. Also turns on 8 todo tests.

This patch presupposes the patch in bug 730209.

Comment 2

[Gordon P. Hemsley [:GPHemsley]]

Try server builds and (basic) test results available:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-34db65aca0e5/
https://tbpl.mozilla.org/?tree=Try&rev=34db65aca0e5

Comment 3

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Screenshot of patch in action

This demonstrates how the patch works. (Look for entries that have "Cyrillic" or "Latin" in them.)

This screenshot includes code from bug 730209 and bug 716321, and demonstrates a custom modification of the existing mk-MK dictionary to use valid BCP 47 tags. (It also obviously includes various nonsense testing dictionaries, including those which have 'Qaaa' as their script subtag.)

Comment 4

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Add support for BCP 47 script subtags (v2)

Improve the test suite to cover all expected cases, including numeric region subtags and extension and privateuse subtags. (The regexp has been updated to allow the latter two subtag types in dictionary names, but their contents are ignored.)

This patch assumes the presence of bug 716321.

Comment 5

[Axel Hecht [:Pike]]

Sorry for the lag, a few questions:

Comparing with http://en.wikipedia.org/wiki/ISO_15924, there are a few codes not listed, what are the criteria for that?

Also, I guess the wikipedia article would make a good localization comment. As you'd be adding comments, MPL2 license header would be good.

What's the rationale for how you're serializig the language codes?

Comment 6

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Add support for BCP 47 script subtags (v3)

Spun off the unrelated changes into bug 741842.

Comment 7

[Gordon P. Hemsley [:GPHemsley]]

(In reply to Axel Hecht [:Pike] from comment #5)
> Sorry for the lag, a few questions:
> 
> Comparing with http://en.wikipedia.org/wiki/ISO_15924, there are a few codes
> not listed, what are the criteria for that?

There are only a handful that I removed on purpose, and they were all script subtags that were not likely to be relevant in any context in Firefox.

In particular, the private use subtags and the "special" subtags that refer to some meta-property of a script (or lack thereof), rather than any real script. (Namely: Qaaa—Qabx, Zinh, Zxxx, Zyyy, Zzzz.)

Were there any other subtags that you saw were missing?

> Also, I guess the wikipedia article would make a good localization comment.
> As you'd be adding comments, MPL2 license header would be good.

Are you referring to toolkit/locales/en-US/chrome/global/scriptNames.properties?

Do you want me to add a localization note to the top of that file?

Regarding the license, is it particularly beneficial to have this list under MPL2 rather than public domain? It's merely a compendium of data from other sources. (It makes no difference to me, but I don't think any of the other lists have such headers.)

> What's the rationale for how you're serializig the language codes?

I'm not sure what you mean. (If you're referring to the tests, those are now in bug 741842; if you're referring to the updated list of language subtags, that's in bug 716321; otherwise, I don't think there are any language subtags that are touched in this patch.)

Comment 8

[:Gavin Sharp [email: [e-mail]]]

Comment on attachment 611863 [details] [diff] [review]
Add support for BCP 47 script subtags (v3)

I don't have any issues with the code; I can't really provide any guidance on the content of scriptNames.properties, but it seems like Axel has those concerns covered.

Comment 9

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Add support for BCP 47 script subtags (v4)

Fix bitrot fallout from bug 716478.

Comment 10

[Gordon P. Hemsley [:GPHemsley]]

Attachment: Add support for BCP 47 script subtags (v5)

Rebase patch.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.