1203171 - Change regionNames.properties to use GENC data

Status: RESOLVED FIXED

(Toolkit :: General, defect)

Description

[Gervase Markham [:gerv]]

The file http://hg.mozilla.org/releases/mozilla-aurora/file/default/toolkit/locales/en-US/chrome/global/regionNames.properties contains a list of region code / region mappings. As per the post in mozilla.governance, we are standardising across the project on the GENC list of codes and mappings, so we need to replace what's there with that list, and get the file localized. 

See https://github.com/gerv/genc2json for data.

Gerv

Comment 1

[Gervase Markham [:gerv]]

Hmm. Where does this data appear in the Firefox UI? The only include I can find for regionNames.properties is in browser/components/preferences/languages.xul, and it displays languages, not regions... (It's the Accept-Language config UI.)

ISTR Pascal said it was used for charsets, but I don't see a list of regions in the Charset menu.

Gerv

Comment 2

[Gervase Markham [:gerv]]

Attachment: Patch v.1

Here's a patch anyway...

Gerv

Comment 3

[Gervase Markham [:gerv]]

Pascal: can you tell where, if at all, this file is used?

Gerv

Comment 4

[Pascal Chevrel:pascalc]

(In reply to Gervase Markham [:gerv] from comment #3)
> Pascal: can you tell where, if at all, this file is used?
> 
> Gerv

No I can't, this file has been there for at least a decade and given that this is part of Toolkit, it could also be used by non-Firefox apps (Thunderbird, Seamonkey, Kompozer...).

Comment 5

[Axel Hecht [:Pike]]

Note, the languages panel shows regions for some languages. English being one of them. The full list of languages and lang/region combos we show is in https://dxr.mozilla.org/mozilla-central/source/intl/locale/language.properties.

Note, I'm pretty sure that there are a lot of usages that piggy-back on the information in this file.

Comment 6

[Gervase Markham [:gerv]]

(In reply to Axel Hecht [:Pike] from comment #5)
> Note, the languages panel shows regions for some languages. English being
> one of them.

But that data's not sourced from this file, is it?

> The full list of languages and lang/region combos we show is in
> https://dxr.mozilla.org/mozilla-central/source/intl/locale/language.

Where are the actual strings?

> Note, I'm pretty sure that there are a lot of usages that piggy-back on the
> information in this file.

"This file" being language.properties, or regionNames.properties?

Gerv

Comment 7

[Axel Hecht [:Pike]]

https://dxr.mozilla.org/mozilla-central/source/browser/components/preferences/languages.js#31 is the code.

And yes, I think a bunch of things piggy-back on regionNames.properties.

Comment 8

[Gervase Markham [:gerv]]

OK, it seems to be still used, at least in part. And anyway, it's as good a place of any to make and keep a list of regions and their localized names.

Gerv

Comment 9

[Pascal Chevrel:pascalc]

Comment on attachment 8658788 [details] [diff] [review]
Patch v.1

Review of attachment 8658788 [details] [diff] [review]:
-----------------------------------------------------------------

The patch looks fine to me, I guess it should be applied on mozilla-central

Comment 10

[Gervase Markham [:gerv]]

Attachment: Patch for checkin

Comment 11

[Gervase Markham [:gerv]]

The patch for checkin has the checkin comment in it.

Gerv

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/cb711870d2b8

Comment 13

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I had to back this out in https://hg.mozilla.org/integration/fx-team/rev/7400c2750b7b for mochitest-bc failures:

https://treeherder.mozilla.org/logviewer.html#?job_id=4691165&repo=fx-team

Comment 14

[Gervase Markham [:gerv]]

Looks like there's a bunch of tests we need to fix too :-| Sorry about that.

Gerv

Comment 15

[Gervase Markham [:gerv]]

Pascal: if this isn't a straight substitution, I'd prefer one of the l10n team to drive this patch. Is this something you could take over? Or nominate someone?

Gerv

Comment 16

[Pascal Chevrel:pascalc]

I don't work on our products but websites and this patch requires updating mozilla-central, so I am probably not the right person for that. Pike should be the one doing this, another option is to put that into the list of projects that our intern (that should start end of October) could work on.

Comment 17

[Gervase Markham [:gerv]]

:pike: is this something you could make happen? This one is particular important because it's the one that gets localized, and a lot of other places will pull from this source.

Gerv

Comment 18

[Axel Hecht [:Pike]]

I'm not sure what the right fix is.

Per https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#User-assigned_code_elements

> The following alpha-2 codes can be user-assigned: AA, QM to QZ, XA to XZ, and ZZ.

That's why the tests use QZ assuming that they can.

It might be the right fix to just not uplift user-assigned code elements in region.properties.

CCing Simon for input, in the end this is much more intl than l10n. Also Ehsan, 'cause I think the spellchecker tests are to some extent his.

Comment 19

[Gervase Markham [:gerv]]

(In reply to Axel Hecht [:Pike] from comment #18)
> It might be the right fix to just not uplift user-assigned code elements in
> region.properties.

No, we definitely want to use the entire list, and it makes sense to use the codes they assign rather than pick new ones.

So we should change the test to use a code they haven't used.

Gerv

Comment 20

[Simon Montagu :smontagu]

(In reply to Gervase Markham [:gerv] from comment #19)
> So we should change the test to use a code they haven't used.

Agreed.

Since you opened the bottle by cc-ing me, I have some other comments about the changes:

Have we considered using the union of the old list and the new rather than removing entries that are in ISO-3166 but not in GENC? AN (Netherlands Antilles) has been deprecated and can safely be removed, but AX, PS, SJ are all still valid ISO-3166 codes. (In the case of AX (Åland Islands), GENC treats them as part of Finland; PS and SJ are replaced in GENC by finer-grained divisions in the user-assigned area)

The old names deliberately used natural language order in names like "North Korea" because we thought that made more sense for displaying names in the UI, rather than the reversed forms like "Korea, North" which are more appropriate for alphabetized lists.

I don't see why we should change "Myanmar" to "Burma".

Comment 21

[Gervase Markham [:gerv]]

(In reply to Simon Montagu :smontagu from comment #20)
> Since you opened the bottle by cc-ing me, I have some other comments about
> the changes:

Hi Simon,

After much discussion in mozilla.governance and with Legal, we decided the best way to get a good list and avoid arguments about what changes are legitimate and what changes are not is to use an existing list with no modifications, and the GENC list is the best one available. So we are deliberately using it wholesale, rather than tweaking it or augmenting it.

> The old names deliberately used natural language order in names like "North
> Korea" because we thought that made more sense for displaying names in the
> UI, rather than the reversed forms like "Korea, North" which are more
> appropriate for alphabetized lists.

I _might_ be persuaded that in this one case, this does not rise to the level of a significant change.

> I don't see why we should change "Myanmar" to "Burma".

The US Government seems somewhat confused about what it thinks this country is called, but nevertheless their official list still uses Burma, and see the principle above.

Gerv

Comment 22

[Gervase Markham [:gerv]]

So Pike: are you able to take this on and fix it? It's the implementation of a decision reached after quite extensive discussion, and input from Mozilla Legal. We want to use this list of countries and regions everywhere we need a list of countries and regions, and this is the premier place.

Gerv

Comment 23

[Axel Hecht [:Pike]]

Sorry, I know about those failing tests as much as you do, can't be of much help here.

Comment 24

[Gervase Markham [:gerv]]

Attachment: Patch v.2

OK, here's a patch with updated tests, which passed a Try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=498d8c34c503

Gerv

Comment 25

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/92668b769325

Comment 26

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/92668b769325