678111 - Replace static nightly_locales.xml with a dynamically generated version for the ftp download site

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Joey Armstrong [:joey]]

Extract strings during the build and use them to dynamically generate a new xml file for the ftp site.  Include date, buildid, path language and build data.  Details noted below in John's email.

libs- target in the l10n.mk makefile can probably be used as the launch point for generation.  Wesley provided URLS for source content, static content, script possibly used for repackaging and links to the upload site.


http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-linux-l10n/linux-i686/xpi/
http://mxr.mozilla.org/mozilla-central/source/mobile/locales/generic/install.rdf
https://addons.allizom.org/en-US/mobile/api/1.5/get_language_packs
http://people.mozilla.com/~wjohnston/nightly_locales.xml
http://mxr.mozilla.org/mozilla-central/source/toolkit/locales/l10n.mk#191
http://mxr.mozilla.org/mozilla-central/source/mobile/locales/en-US/chrome/localepicker.properties
http://hg.mozilla.org/build/tools/file/6d8a3b443306/scripts/l10n/release-mobile-repacks.py#l67

===========================================================================

hi;

Quick notes from meeting of wesj and joduinn to keep everyone in the loop.


1) joduinn: why are
ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-macosx-l10n/
and
ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2011-08-09-03-07-51-mozilla-central-macosx/
different? Should be the same?

2) joduinn: why is ftp://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/
different to ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ ?

3)  change location on ftp.m.o that .xpi files are uploaded to, so
RelEng  can keep the xpis for each nightly, along with the en-US
nightly. Avoids  possible problems when using newer xpi with older
nightly build.
TODO: joduinn to verify that changing xpi files does not break any l10n
dashboards.

4) change makefiles to generate new .xml file with each nightly fennec
build. change automation to upload .xml file into same directory on
ftp.m.o alongside the build.
http://people.mozilla.com/~wjohnston/nightly_locales.xml
https://addons.allizom.org/en-US/mobile/api/1.5/get_language_packs
https://bugzilla.mozilla.org/show_bug.cgi?id=674253
see also:
http://mxr.mozilla.org/mozilla-central/source/mobile/locales/generic/install.rdf

NOTE: directory name already includes the buildid, with extra "-" dividers.
NOTE: idea is to generate one xml file, which contains strings from
every locale present in that
nightly. How to get locale list? all_locales? shippped_locales file?

TODO: wesj to talk with "joey", about getting Makefile changes done, and
then file bug in RelEng.



5)  how far back to keep? files are small, but we do a *lot* of
nightlies every night.

** TODO: joduinn to verify we can keep xpi's as long as we keep nightlies.


TODO: how to get these xpi files to AMO for release builds? unknown,
emails w/wil clouser.
TODO: any differences between these xml files and what is needed on AMO?
unknown - look into using AMO for nightly builds - pointing to "static"
files on the ftp server (with build id)


John.
=====
On 8/5/11 10:32 AM, Wesley Johnston wrote:
> Just had a call with Mossop and mfinkle about this, and think we have a good plan for how to move forward.
>
> Summary: We're going to fix up the addon-manager so that locales can be installed in a restartless way. We're also hoping releng can store nightly/aurura locales for us in a location that can key off of build-id so that we can ensure we download a locale compatible with the current build of Fennec when nightly usrs update. We're also hoping RelEng can produce/store a (static?) list of compatible locales we can use to determine what locales are available for nightly/aurora users. For release/beta users, AMO will provide the same function.
>
> Stuff below here is sorta for my own documentation, but feel free to read it over if you want more details:
>
> Mossop:
>
> Mossop had some conerns about the user expeience of Fennec restarting twice in a row. We think that it should be easy to package locales as "Restartless" addons. Someone is currently working on a similar restartless patch for dictionaries that is close to done, and should be simple to adapt for language packs.
>
> RelEng:
> For nightly users we are planning to update their locale during the nightly update process. This is going to require some help from RelEng. We would like (for nightly and aurora users) to store the last "x" (week?) locale xpi's on the ftp server in a particular buildid directory. When the nightlies are built, we will also build the locale packs and store them in the appropriate folder. When Fennec downloads the update, we believe (need to verify) that we also download the build-id of the new version. We can use that to determine if there is a new locale pick for this particular build id and download it as well.
>
> We could have a new get_locales.xml file also built by releng each night, that could be stored along with the xpi's in the correct build id folder. Fennec would look for it by replacing tokens in an update url. Or we could have a single static get_locales.xml file that listed the location of xpi's as "ftp://ftp.mozilla.org/.../%buildid%/english.xpi, and just update it when absolutely necessary. RelEng - Do you have a preference for this?
>
> If we find a good locale that matches the current, we silently download and install it. If we don't, we still remove the old locale, and just revert the user back to English.
>
> Finally, we're going to add some checks into our commandline handler to also make sure the current locale is compatible. This is especially important, as Release and Beta users won't be using our updater to update anyway, so the above steps won't apply for them. Instead, their locale will be automatically marked incompatible by the AddonsManager.
>
> When the commandline handler fires, we check if the user selected language is still compatible. If it is (either because we downloaded before we updated, or because nothing changed from day to day (release users)), we're done! If it isn't, we'll hit up AMO/FTP site again to see if we have a compatible locale. If we do, we'll use it. Otherwise we'll probably just have to fall back to English (or we can offer to let them choose a language).

Comment 1

[Joey Armstrong [:joey]]

Setup a wiki for tracking content.  Currently contains a parse of nightly_locales.xml, an itemized list of fields that will be needed.

http://wiki.mozilla.org/User:JoeyArmstrong/locales

Comment 2

[Joey Armstrong [:joey]]

Tag list for xml file generation.

# parse of the latest nightly_locales.xml file:
http://people.mozilla.com/~jarmstrong/bugs/678111/parse_20110901.html

# older parse containing some differences
http://people.mozilla.com/~jarmstrong/bugs/678111/parse_old.html

Comment 3

[Joey Armstrong [:joey]]

Work to date: script/function setup to accept a formatted data structure that will be recursively expanded into nightly_locales.xml.  Helper functions added w/unit tests allowing data elements to be handled/formatted as atomic elements that can later be inserted into the expansion structure.

Gathering data from different soruces for insertion into the structure for generation.

Comment 4

[Joey Armstrong [:joey]]

Attachment: early checkpoint -- script used to gather date and reformat content into nightly_locales

Logic is still scattered between a few standalone scripts but for highlights so far:
1) Functions have been setup to download most of the data sources -- installable images, *.xpi files, locale.picker, install.rdf, etc, etc.
  * code from bug # 554752 can be reused after landing.  The bug contains a module able to unpack tarballs/archives/zipfiles/mount dmg images, etc, etc.
2) Library functions will handle data extraction from sources into key/value pairs.
3) genSubXML() subroutine is the heart of the xml generator.  The function will accept a formatted datastructure (dicts, lists, scalars) and will recursively expand content into nightly_locales.xml.
4) Helper functions are being setup to format key/value pairs into data that genSubXML() can act on.
5) Unit tests have been setup to back the generator and most of the extraction/parsing routines.

After more data is extracted, formatted and passed into genSubXML() we should be able to generate the nightly xml files.

Comment 5

[Wesley Johnston (:wesj)]

Attachment: Extension

Just realized I've never put this extension up anywhere. Probably because its not very good. This is what I've been using to generate nightly_locales.xml and now aurora_locales.xml as well. It just runs on every startup, so probably not the best extension to keep installed and enabled.

Downloads all the locales from ftp into a "locales" directory on your desktop. Overwrites things that are there. Iterates through and eventually leaves you with those two files + a stray install.rdf I've never bothered to clean up.

Comment 6

[Wesley Johnston (:wesj)]

Attachment: Extension

Whoops. Last extension had a little bug in one of the files it generated.

Comment 7

[Joey Armstrong [:joey]]

python's lxml module is unable to parse data out of localepicker.properties because of missing tags.  If tags are not nested too deeply short term a small fixit script could be run on the downloaded file to add closing tags.

Comment 8

[Joey Armstrong [:joey]]

(In reply to Joey Armstrong [:joey] from comment #7)
> because of missing tags

-- missing html tag closure.

Comment 9

[Joey Armstrong [:joey]]

(In reply to Joey Armstrong [:joey] from comment #7)
> python's lxml module is unable to parse data out of localepicker.properties
> because of missing tags.  If tags are not nested too deeply short term a
> small fixit script could be run on the downloaded file to add closing tags.

ERROR:root:xmlToStruct(test/data/localepicker.properties) failed: Opening and ending tag mismatch: link line 7 and head, line 11, column 8

Comment 10

[Joey Armstrong [:joey]]

For unit testing: strip xml tokens and return key=value pairs
=============================================================

http://mxr.mozilla.org/mozilla-central/source/mobile/locales/en-US/chrome/localepicker.properties?raw=1

or for non-english locales:

http://mxr.mozilla.org/l10n-central/source/ru/mobile/chrome/localepicker.properties?raw=1

those are what actually ship, and won't contain any xml at all.

Comment 11

[Joey Armstrong [:joey]]

bug on hold for now:

>> I'm not sure if you're aware, but as of two weeks ago we threw away everything and started a new frontend for Fennec. That means all of my frontend code for this is (currently) gone. I'd like to add something similar back in some day, but as of right now this (rather suddenly) became a low priority feature for us. i.e. if you have other stuff to work on you probably should.

Comment 12

[Joey Armstrong [:joey]]

Attachment: alpha code: automate generation of nightly_locales.xml

Patch is incomplete -- uploading current rev for posterity until recent fennec work settles down and the dust clears.

o scripts and unit tests to automate generation of nightly_locales.xml
o download *.xpi, language packs, etc, etc.
o extract *.rdf files.
o rdflib used to parse rdf files
o unicode formatting for data
o lxml used to generate an xml stream
o mapping logic added to match fields, extract content and generate a data hash using keys named for the tags in nightly_locales.
o an internal data structure is populated with content that will later be expanded into a hierarchical xml stream.
o not all fields have been populated <type id=5>lang pack (app)</type>, <status id=4>Public</status>.
o stream needs to be written to disk when --xml is passed.

Comment 13

[Armen [:armenzg]]

RESOLVE INVALID and re-open if the plan changes back?