733417 - Update product-details to match GENC list

Status: RESOLVED FIXED

(www.mozilla.org :: Product Details, defect)

Description

[Fred Wenzel [:wenzel]]

The regions lists in product details do not contain either of the two internationally recognized "new countries" established in 2011. We should add them.

CCing Axel and Pascal: Can you advise what's involved in updating these localized properties files?

<http://viewvc.svn.mozilla.org/vc/libs/product-details/>

Comment 1

[Fred Wenzel [:wenzel]]

(In reply to Fred Wenzel [:wenzel] from comment #0)
> established in 2011.

(for the sake of accuracy: Kosovo has declared independence in 2008).

Comment 2

[Pascal Chevrel:pascalc (PTO until April 26)]

If you are refering to files such as http://viewvc.svn.mozilla.org/vc/libs/product-details/regions/ak.properties?revision=87934&view=markup

Those were imported from our Firefox codebase where all the country names are already translated, we can reimport them all so as to update them with the latest versions but if we want them to list the countries you mention, they need to be in Firefox source files.

In the past they were created when we needed to create pages on mozilla.com allowing to send SMS to people in the world. Out of curiosity, what is the use you have for these files?

Comment 3

[Axel Hecht [:Pike]]

Kosovo doesn't have a ISO 3166 code, which is our safest argument to not go there. It's one of those regions that gives you heat no matter what you do.

South Sudan is different having ISO code SS (we need a non-german to land that).

Comment 4

[Fred Wenzel [:wenzel]]

(In reply to Pascal Chevrel:pascalc from comment #2)
> Those were imported from our Firefox codebase where all the country names
> are already translated, we can reimport them all so as to update them with
> the latest versions but if we want them to list the countries you mention,
> they need to be in Firefox source files.

All right. Is there a chance for this to happen?

> In the past they were created when we needed to create pages on mozilla.com
> allowing to send SMS to people in the world. Out of curiosity, what is the
> use you have for these files?

The dependent bug of this one is bug 732720 from a Kosovo contributor asking for his country to show up in ReMo's country list, which I think is a fair request. For contests etc etc we routinely use product-details' region list to populate a list of countries.

(In reply to Axel Hecht [:Pike] from comment #3)
> Kosovo doesn't have a ISO 3166 code, which is our safest argument to not go
> there. It's one of those regions that gives you heat no matter what you do.

FWIW, this change seems to be pending:
http://geonames.wordpress.com/2010/03/08/xk-country-code-for-kosovo/

The commonly used interim code seems to be XK. I don't think we need to keep that country off our country list until ISO comes around.

> South Sudan is different having ISO code SS (we need a non-german to land
> that).

Sigh.

Comment 5

[Fred Wenzel [:wenzel]]

Fwiw:
<https://github.com/mozilla/bedrock/blob/28b3f3377de31a3566c7e134ba9718bdf11dcbde/apps/collusion/templates/collusion/collusion.html#L248>

There shouldn't be a need for us to hardcode this stuff.

Comment 6

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Fred Wenzel [:wenzel] from comment #4)
> (In reply to Pascal Chevrel:pascalc from comment #2)
> > Those were imported from our Firefox codebase where all the country names
> > are already translated, we can reimport them all so as to update them with
> > the latest versions but if we want them to list the countries you mention,
> > they need to be in Firefox source files.
> 
> All right. Is there a chance for this to happen?

That's a question for Pike :)

Comment 7

[Axel Hecht [:Pike]]

Well, that merge isn't going to help, because I don't think we have either in the product.

Re XK, I don't read that to be a change happening to the ISO codes. I failed to find a reference why people are using it, which makes it hard to figure out if that's appropriate for some sites at mozilla, or all.

Comment 8

[Fred Wenzel [:wenzel]]

(In reply to Axel Hecht [:Pike] from comment #7)
> Re XK, I don't read that to be a change happening to the ISO codes. I failed
> to find a reference why people are using it, which makes it hard to figure
> out if that's appropriate for some sites at mozilla, or all.

X<anything> is a private name space in the ISO standard allowing people to use those codes for their own needs. The point of XK, therefore is, to stand in for a country code until the ISO assigns one.

I am strongly in favor of not just ignoring the existence of this country based on the ISO being slow. There's a handy little map on Wikipedia illustrating who else has chosen not to wait:
<http://en.wikipedia.org/wiki/File:CountriesRecognizingKosovo.svg>

Comment 10

[Pierros Papadeas [:pierros]]

Ping, any update on that?

Comment 11

[Pierros Papadeas [:pierros]]

It has been seven months now :) Can we have a decision on this? Thanks!

Comment 12

[Altin Ukshini]

Hello guys,

I was about to file a similar bug (about Kosovo) but luckily found this one.
I apologize if I should not be continuing the discussion here furthermore.

While reading through the bug, I saw that Axel mentioned that Kosovo doesn't have a ISO 3166 code - that's true, but I would like to ask if the following is valid for adding Kosovo (if I may say temporary) on the list?

ISO 3166-1 was notified that the alpha-2 user-reserved code element XK was used by the European Union to represent Kosovo. And moreover ISO 3166/MA agreed when ISO/IEC
JTC 1 asked for an ‘exceptional reserved’ status for the alpha-3 code element UNK to represent KOSOVO. Therefore SO 3166/MA is essentially recommending the use of the
reserved code elements ‘XK’ and ‘UNK’ to represent the country name ‘Kosovo’ when necessary.

KOSOVO, which is not a UN Member State, has neither a corresponding code
element inside the UNSD M.49 code nor a corresponding entry within ISO 3166-1. However, Clause 8.1.3. (User-assigned code elements) of the ISO 3166-1 normative
text states that:

“If users need code elements to represent country names not included inside ISO 3166-1, the series of letters AA, QM to QZ, XA to XZ, and ZZ, and the series AAA to AAZ, QMA to QZZ, XAA to XZZ and ZZA to ZZZ respectively and the series of numbers 900 to 999 are available. These users should inform the ISO 3166/MA of such use.”

Moreover, clause 7.5.4. (Exceptional reserved code elements) states that:

“Code elements may be reserved, in exceptional cases, for country names
which the ISO 3166/MA has decided not to include in ISO 3166-1, but for which an interchange requirement exists. Before such code elements can be
reserved, advice from the relevant authority must be sought.”

Thanks,
Altin

Comment 13

[Gervase Markham [:gerv]]

We should not let discussions about Kosovo prevent us from landing the uncontroversial South Sudan. :kohei: is that something you can do?

Gerv

Comment 14

[Kohei Yoshino [:kohei]]

I don't have commit access to product-details. Maybe Pascal?

Comment 15

[Pascal Chevrel:pascalc (PTO until April 26)]

Back from PTO, taking this bug to add South Sudan.

Comment 16

[Altin Ukshini]

What's the status gonna be for Kosovo?
I see nobody replied to my last comment :/

Thanks,
Altin

Comment 17

[Gervase Markham [:gerv]]

Altin: no decision has been made yet. As you can imagine, this is a difficult question. See mozilla.governance for some discussion. I'm afraid you will have to be patient :-|

Gerv

Comment 18

[Altin Ukshini]

You're right, it can be difficult, if we only could really understand why?

If I do recall the issue correctly (as it's been a long time), we never got a strong reason to be patient for more than 3 years for this bug to open again (including other bugs we filed). 

I don't get why these institutions didn't find it difficult to list Kosovo on the list of countries: W3C, IEEE, Oracle, Dropbox, Amazon, CISCO, Apple, Google, Facebook, Linkedin and 70+ others (http://www.digitalkosovo.org) but Mozilla does?

May I call this issue a bit disrespectful for the community members and the user base that Mozilla has in Kosovo?

I hope it'll change soon.

Thanks for your understanding!

Best,
Altin

Comment 19

[Gervase Markham [:gerv]]

(In reply to Altin Ukshini from comment #18)
> I don't get why these institutions didn't find it difficult to list Kosovo
> on the list of countries: W3C, IEEE, Oracle, Dropbox, Amazon, CISCO, Apple,
> Google, Facebook, Linkedin and 70+ others (http://www.digitalkosovo.org) but
> Mozilla does?

If you have links to places which have explicit policies on these matters, and what their policy is, and explain how they decided their policy, that would be very useful. (It's not useful for these purposes to just list websites which took their existing list and added "Kosovo".)

Gerv

Comment 20

[Gervase Markham [:gerv]]

I recently posted the following in mozilla.governance:

Thanks to everyone for their input on this. It's clear that none of the
options available to us here is without any problems. So, after
discussions with Mozilla legal, we've decided that we will work towards
standardizing Mozilla's websites on using the GENC list[0].

The GENC list is based on ISO3166 but modified by the US Government for
their purposes. As it so happens, it makes what seem to us like a very
sensible set of changes. Of particular relevance to this thread, it adds
an entry for Kosovo, but it also makes other changes which seem
sensible, such as calling Taiwan "Taiwan".

I will be contacting the relevant responsible people over the next
little while to encourage them to move their sites across to using this
data source. If you know of places where Mozilla is using a list of
countries on a website, please let me know by email.

If you can see some massive objection to this course of action, other
than "it's from the US Government", then please also let me know.
Otherwise, thanks to everyone for their participation in the discussion.

Gerv


[0] https://nsgreg.nga.mil/doc/view?i=2500 is the latest version,
version 3.0 - although unfortunately this website requires a security
exception in all Firefoxes and Chromes due to its cert chaining to the
untrusted DOD root, and also won't work in Nightly or Developer Edition
because their server is officially busted and won't negotiate anything
other than RC4 even though it advertises it. <shrug>

Comment 21

[Gervase Markham [:gerv]]

pascalc: can you make the product-details file match the GENC list, including adding Kosovo and South Sudan?

The data at https://github.com/gerv/genc2json might be useful to you.

Gerv

Comment 22

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Gervase Markham [:gerv] from comment #21)
> pascalc: can you make the product-details file match the GENC list,
> including adding Kosovo and South Sudan?
> 
> The data at https://github.com/gerv/genc2json might be useful to you.
> 
> Gerv

Could you change your script so as that in the generated Json file the country codes are the keys and not the values? That will make it simpler to compare it with our own json files http://svn.mozilla.org/libs/product-details/json/regions/en-US.json. That should just be a matter of swapping variables in https://github.com/gerv/genc2json/blob/master/genc2json#L44

I tried to run your script locally but it just got stuck,  I think it needs an xml file to parse that doesn't seem to be in your repository.

The difficulty is not to update product-details with English values, the difficulty is to get the updated list of countries translated in all of our languages, that's why we imported them from Firefox source code. If the additions are for websites only and not for Firefox, then we should get those updated country lists translated by localizers and not just imported from Firefox l10n files. I am going to investigate if this was not already done by other projects so as to not reinvent the wheel.

Comment 23

[Gervase Markham [:gerv]]

Done; hash inverted.

I haven't checked in the GENC file because I haven't checked on its copyright status. I would expect it's in the public domain, as a work of the US government, but I'd need to be sure. If anyone needs it, they can get it from the website mentioned in comment 20.

Hopefully, much of the translation work that has already been done can be reused.

Gerv

Comment 24

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Gervase Markham [:gerv] from comment #23)
> Done; hash inverted.
> 

Thanks

> I haven't checked in the GENC file because I haven't checked on its
> copyright status. I would expect it's in the public domain, as a work of the
> US government, but I'd need to be sure. If anyone needs it, they can get it
> from the website mentioned in comment 20.

ok, makes sense

> 
> Hopefully, much of the translation work that has already been done can be
> reused.

yes, it's just a matter of scripting and if I can't find an existing l10n ressource with the new countries already added and translated for all of our locales, then I will need to expose the translation of the country names to our web localizers, nothing blocking, it might just take some time to get it done as we have multiple projects already on our plates.

Comment 25

[Pascal Chevrel:pascalc (PTO until April 26)]

We have naming differences between GENC and what we use today, should we update product-details for those region/country names too ? Here are the naming differences (left is Mozilla, right is US government):

====== Countries with diverging names ====== 
ae: U.A.E.  /  United Arab Emirates
bl: Saint Barthélemy  /  Saint Barthelemy
bn: Brunei Darussalam  /  Brunei
bs: Bahamas  /  Bahamas, The
cd: Congo-Kinshasa  /  Congo (Kinshasa)
cg: Congo-Brazzaville  /  Congo (Brazzaville)
ci: Ivory Coast  /  Côte d’Ivoire
cv: Cape Verde  /  Cabo Verde
fk: Falkland Islands (Malvinas)  /  Falkland Islands (Islas Malvinas)
fm: Micronesia  /  Micronesia, Federated States of
gm: Gambia  /  Gambia, The
gs: South Georgia and the South Sandwich Islands  /  South Georgia and South Sandwich Islands
kp: North Korea  /  Korea, North
kr: South Korea  /  Korea, South
mk: Macedonia, F.Y.R. of  /  Macedonia
mm: Myanmar  /  Burma
mo: Macao  /  Macau
pn: Pitcairn  /  Pitcairn Islands
ru: Russian Federation  /  Russia
sh: Saint Helena  /  Saint Helena, Ascension, and Tristan da Cunha
tf: French Southern Territories  /  French Southern and Antarctic Lands
vg: British Virgin Islands  /  Virgin Islands, British
vi: U.S. Virgin Islands  /  Virgin Islands, U.S.

Comment 26

[Gervase Markham [:gerv]]

pascalc: All those changes are fine. The entire point of using someone else's list is that we don't have exceptions. While I might personally go with "North Korea", rather than "Korea, North", the advantage of not having to decide what's in and what's out far outweighs a small quibble like that.

Gerv

Comment 27

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Gervase Markham [:gerv] from comment #26)
> pascalc: All those changes are fine. The entire point of using someone
> else's list is that we don't have exceptions. While I might personally go
> with "North Korea", rather than "Korea, North", the advantage of not having
> to decide what's in and what's out far outweighs a small quibble like that.
> 
> Gerv

OK, then all these strings will need a retranslation/review of existing translation for all of our locales.

Comment 28

[Gervase Markham [:gerv]]

pascalc: if you aren't able to get this done, can you say who can? It seems like it's just a case of editing the file...

Gerv

Comment 29

[Pascal Chevrel:pascalc (PTO until April 26)]

The right person to ask is Pike (also note that I am no longer working in l10n so it's very unlikely that I will work on that).

Comment 30

[Axel Hecht [:Pike]]

Let's play a bit of reorg ping pong.

Jeff, I think ownership of product-details is something to clarify with relman. Also, really still SVN?

Comment 31

[Jeff Beatty [:gueroJeff]]

Just spoke with Sylvestre. Since product-details is generated automatically from ship-it now, it seems appropriate for relman to own this. CC-ing him to confirm.

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Confirmed but I cannot say when I will be able to update the list.

Comment 33

[Sylvestre Ledru [:Sylvestre]]

I am going to take care of this.
The idea is to write a script which will take the info here:
https://hg.mozilla.org/mozilla-central/file/tip/toolkit/locales/en-US/chrome/global/regionNames.properties
for english and sync the other locales from Transvision:
https://transvision.mozfr.org/api/v1/entity/aurora/?id=toolkit/chrome/global/regionNames.properties:xt 

We should run the script from time to time.

Comment 34

[Gervase Markham [:gerv]]

Fantastic! :-)

Gerv

Comment 35

[Sylvestre Ledru [:Sylvestre]]

Attachment: Implementation

Comment 36

[Sylvestre Ledru [:Sylvestre]]

Attachment: Fixed pr

Comment 37

[Sylvestre Ledru [:Sylvestre]]

Now live here:
https://product-details.mozilla.org/1.0/regions/
for example, in:
https://product-details.mozilla.org/1.0/regions/fr.json
We can find "Bande de Gaza" and "Cisjordanie" as new country codes (used to be "Palestinien Occupé, territoire")

Comment 38

[Sylvestre Ledru [:Sylvestre]]

We consider that the source of truth is the list of country shipped in Firefox code (see bug 1203171)
This is updated using the script available here: https://github.com/gerv/genc2json

For the record, to update the list:
$ git clone [e-mail]:mozilla-releng/ship-it.git
$ cd ship-it/scripts/
$ python sync-and-check-l10n.py 
Synchronizing from https://hg.mozilla.org/releases/mozilla-aurora/raw-file/tip/toolkit/locales/en-US/chrome/global/regionNames.properties
[...]
$ git commit -m "Update the list of countries and their translations" -a