1420236 - Support versions in multilocale.json

Status: NEW

(Firefox Build System :: General, defect)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

In bug 1420173 we are bringing runtime support for versioned locale packages.

This nicely covers how we assign versions to locale packages we add to language packs, where we take the current changeset of the locale we load (say, ~/.mozbuild/l10n-central/it) and we look for a push timestamp for it using our pushlog API [0]

The result is a numeric version that we can then compare to another version of the same locale and decide which one is newer.

An example is when we have two langpacks which have `it` locale, one langpack with version `20170101102300` and another with version `20170323192900`.
Knowing that allows us to compare those two and first try to use the newer one in L10nRegistry, before lazily falling back on the older one if needed.

This system currently is limited to language packs, because for packaged locales, we only store the language tag, without a version.

I'd like to turn multilocale.json from `{"locales": ["en-US"]}` to `{"locales": {"en-US": "20170101102300"}}`
and then at runtime feed the version to FileSource and into L10nRegistry in [1].

The code that currently generates multilocale.json is in [2].

:ted - can you advise on how can I convert [2] to use the same API as [0] to get the pushlog timestamp out of l10n-basedir repo (or mozilla-central pushlog timestamp for en-US)?



[0] https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/langpack_manifest.py#52
[1] https://searchfox.org/mozilla-central/source/browser/components/nsBrowserGlue.js#671
[2] https://searchfox.org/mozilla-central/source/toolkit/locales/gen_multilocale.py

Comment 1

[Nick Alexander :nalexander [he/him]]

> :ted - can you advise on how can I convert [2] to use the same API as [0] to
> get the pushlog timestamp out of l10n-basedir repo (or mozilla-central
> pushlog timestamp for en-US)?

Does this even make sense?  multilocale.json is populated from MOZ_CHROME_MULTILOCALE, which can be set a build time.  None of these l10n repositories need to be checked out at that time, and in automation I rather think they might not be checked out until _after_ the build.

So again, this comes back to the "original sin" of the l10n system: the build is not told all the information about l10n up front, and therefore does not have sufficient information to do the correct thing.

I think what we should do is bake all the information about the set of locales and the relevant hg repositories into mozconfig settings, and then have configure inspect the hg repositories for the timestamps and wrap all the information up into a tidy package, ready to be pushed out into multilocale.json.

The alternative is to add a "silent dependency" from gen_multilocale.py onto $(HG) and a bunch of hg repositories somewhere on the local disk.  It's just not a good approach.

gps may have other opinions.

Comment 2

[Zibi Braniecki [:zbraniecki][:gandalf]]

> None of these l10n repositories need to be checked out at that time, and in automation I rather think they might not be checked out until _after_ the build.

I don't know. Maybe it doesn't, maybe we should always assume that the locale in the build is "the oldest" (e.g. revision 0) and treat any langpack version as "newer"?

Log from #build:

2:42 PM <@ted> gandalf: you can import anything that's available in the in-objdir virtualenv
2:42 PM <@ted> so importing anything from mozbuild should be fine
2:42 PM <gandalf> ok
2:42 PM <@ted> we use the same $(PYTHON) to run those scripts as anything else
2:43 PM <@ted> you can import buildconfig and use buildconfig.substs['L10NBASEDIR']

2:44 PM <@ted> gandalf: actually hitting the network during a normal build is a bit weird
2:44 PM <@ted> so we might want to do something different for the en-US case