1252634 - Provide the ability to override the locale separate from distribution.ini

Status: RESOLVED FIXED

(Firefox :: Distributions, defect)

Description

[Mike Kaply [:mkaply]]

Yahoo has multiple locales that aren't official Firefox locales (en-HK, es-MX, etc).

Because these aren't used in general.useragent.locale, we can't properly specify them in distribution.ini.

We should allow a locale.ini file in the distribution directory that overrides the default locale used for distribution.ini.

This will allow us to easily have one distribution.ini for all, but override the individual locale on a case by case basis.

Comment 1

[Mike Kaply [:mkaply]]

Attachment: What I'm thinking...

This is what I'm thinking.

This allows us to have one big distribution.ini and small locale.ini for the individual locales.

Comment 2

[Shane Caraveo (:mixedpuppy)]

side note, we do have es-MX on our firefox downloads page.

I think we're starting to stumble across an issue that is more complex than just locale prefs.

How will one big distribution.ini file handle multiple distributions for the same locale?

e.g. de vs. de_keyword vs. de_foxload, en-HK (really en-US with regional differences) vs. en-US

ISTM that there are two elements to a repack that are being conflated in certain cases.  Locale and Distribution.  The distribution has a repack.cfg file which defines the locales used in that distribution.  Confusion occurs because we also include the locale (and sometimes bogus) in the distribution name.  ie. the directory name is the distribution name, but the locale is from repack.cfg.

Maybe a slightly different approach is needed.  Take the existing /yahoo/desktop/ distributions for the example.  we remove the sometimes bogus locale from the distribution name (which is the directory name)

/yahoo/desktop/general/repack.cfg => locale list has all locales wanted by partner
/yahoo/desktop/foxload/repack.cfg => locale list has all foxload distros
/yahoo/desktop/keyword/repack.cfg => locale list has all keyword distros

en-HK adds a level of complexity because it is en-US + regional changes + distribution changes.  So it really is en-US_HK_keyword.  Is HK a locale, a region or a distribution?

unfortunately dist_id in repack.cfg also contains the locale sometimes as well as some key to the distribution.  For de_keyword dist_id is yahoodek.  Just looking at differences between en-US and en-GB, I'm not sure they could be combined into one with localized preferences alone.

Right now I think having the locale based prefs in distribution.ini working correctly can fix part of the problem and reduce some of the directory replication, but I'm not sure that we can accomplish one big distribution for yahoo via distribution.ini.

Comment 3

[Mike Kaply [:mkaply]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #2)
> side note, we do have es-MX on our firefox downloads page.
> 
> I think we're starting to stumble across an issue that is more complex than
> just locale prefs.
> 
> How will one big distribution.ini file handle multiple distributions for the
> same locale?
> 
> e.g. de vs. de_keyword vs. de_foxload, en-HK (really en-US with regional
> differences) vs. en-US

I think we'll still have multiple distribution.ini files in those cases. I pictured
one distribution.ini for things that were of the same type.

> 
> ISTM that there are two elements to a repack that are being conflated in
> certain cases.  Locale and Distribution.  The distribution has a repack.cfg
> file which defines the locales used in that distribution.  Confusion occurs
> because we also include the locale (and sometimes bogus) in the distribution
> name.  ie. the directory name is the distribution name, but the locale is
> from repack.cfg.
> 
> Maybe a slightly different approach is needed.  Take the existing
> /yahoo/desktop/ distributions for the example.  we remove the sometimes
> bogus locale from the distribution name (which is the directory name)
> 
> /yahoo/desktop/general/repack.cfg => locale list has all locales wanted by
> partner
> /yahoo/desktop/foxload/repack.cfg => locale list has all foxload distros
> /yahoo/desktop/keyword/repack.cfg => locale list has all keyword distros
> 
> en-HK adds a level of complexity because it is en-US + regional changes +
> distribution changes.  So it really is en-US_HK_keyword.  Is HK a locale, a
> region or a distribution?

That's true. And honestly it should probably be based on en-GB, not en-US.

But I think in this case, the main thing we need is for general.useragent.locale to be set to en-HK.
That would make our stuff work great. My locale.ini is a workaround for that.

> unfortunately dist_id in repack.cfg also contains the locale sometimes as
> well as some key to the distribution.  For de_keyword dist_id is yahoodek. 
> Just looking at differences between en-US and en-GB, I'm not sure they could
> be combined into one with localized preferences alone.
> 
> Right now I think having the locale based prefs in distribution.ini working
> correctly can fix part of the problem and reduce some of the directory
> replication, but I'm not sure that we can accomplish one big distribution
> for yahoo via distribution.ini.

Agreed. I think we'll only be able to group types (keyword, non keyword, etc).

I think if we can find a way to specify the locale outside of the distribution.ini, we could go a long way.

Have we ever asked the build team of we can build say an "en-HK" distribution for yahoo that sets general.useragent.locale to en-HK?

Essentially some way to have new locale builds based on existing locales?

Comment 4

[Mike Kaply [:mkaply]]

FYI, Yahoo doesn't base their es-MX on our es-MX

yahoo-es-MX/repack.cfg:locales="es-ES"
yahoo-es-MX_keyword/repack.cfg:locales="es-ES"

Comment 5

[Shane Caraveo (:mixedpuppy)]

per irc;

Lets use repack.cfg for this instead.  It already has a list of locales for a build, and partner-repacks.py already modifies distribution.ini.

straw man:

repack.cfg:
locales="en-HK en-MX en-US"
locale.en-HK=en-GB


partner-repacks.py
update createOverrideIni to set general.useragent.locale to the locale.X value in repack.cfg if one is present.

Comment 6

[Mike Kaply [:mkaply]]

Attachment: Python change

I think this is what your thinking.

I decided to make it a new section at the end of the file.

You can specify a Locale that's used as the default locale when parsing the file.

So we'll be able to do locale mapping in repack.cfg, and I'll update the distribution code to use that locale when determining which strings to use in the file.

Comment 7

[Mike Kaply [:mkaply]]

Attachment: A correct fix

OK, this implements your original idea.

So here's the scenario. Yahoo wants to build an en-HK build, but we don't have an en-HK Firefox.

So they put this in repack.cfg:

locales="en-HK"
locale.en-HK="en-GB"

What this says is "Build an en-HK locale, and use the en-GB build as the basis"

So what does building an en-HK locale mean?

1. All the directories/naming will be en-HK for our purposes.
2. In distribution.ini, we'll add a new section:

[Locale]
locale=en-HK

what this tells distribution.js is that is should process the en-HK values in this file. We need this because otherwise it would use the general.useragent.locale of the build (which is en-GB).

We do NOT change the general.useragent.locale of Firefox to en-HK, primarily because I would be concerned that might break something.

Hopefully that makes sense.

Comment 8

[Mike Kaply [:mkaply]]

Attachment: Corresponding distribution.js change

Here's the corresponding distribution.js change.

Comment 9

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8737923 [details] [diff] [review]
Corresponding distribution.js change

I can imagine future confusion here.  Add a comment about why we have the locale in distribution.ini.

Comment 10

[Shane Caraveo (:mixedpuppy)]

Comment on attachment 8737378 [details] [diff] [review]
A correct fix

This is good.  I don't like the name "source-locale", feels confusing, but couldn't think of anything better.

Comment 11

[Mike Kaply [:mkaply]]

I wasn't sure about locale.en-GB in the repack.cfg either.

Naming is the hardest thing I do.

Is github where the repack python scripts live?

When builds are done, are they pulled from those scripts?

Comment 12

[Shane Caraveo (:mixedpuppy)]

(In reply to Mike Kaply [:mkaply] from comment #11)
> I wasn't sure about locale.en-GB in the repack.cfg either.

I was just using that as an example, but I think en-GB is better than en-US for HK.


> Is github where the repack python scripts live?

yeah, it's https://github.com/mozilla-partners/repack-scripts

> When builds are done, are they pulled from those scripts?

partner-repack.py is used to build the repacks.  Other scripts elsewhere are used in the build systems/automation, etc. coop would know more about that.

Comment 13

[Mike Kaply [:mkaply]]

> I was just using that as an example, but I think en-GB is better than en-US for HK.

I was more referring to the first part (locale.). I wasn't sure that was a good description.

source-locale.en-HK=en-GB :)

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #12)
> (In reply to Mike Kaply [:mkaply] from comment #11)
> When builds are done, are they pulled from those scripts?
> 
> partner-repack.py is used to build the repacks.  Other scripts elsewhere are
> used in the build systems/automation, etc. coop would know more about that.

There's a mozharness wrapper script in-tree that encapsulates some of the arguments for release purposes, but those should change very little over time. You can find it here:

https://hg.mozilla.org/mozilla-central/file/68c0b7d6f16c/testing/mozharness/scripts/desktop_partner_repacks.py

When we need to change that script, we need to worry about uplift to get it on the beta and release branches. Since it's NPOTB, uplift has never been a problem.

Comment 15

[Mike Kaply [:mkaply]]

This got reverted in mozilla-partners because I didn't do my python right.

https://github.com/mozilla-partners/repack-scripts/commit/dbaf41277d58d63e76149b165d07b72dd76215cf

I've pushed a new patch. It shouldn't go in until after FF46