1280683 - L10n repacks need to support l20n - not removing en-US

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Axel Hecht [:Pike]]

Filing this to get the l10n repack system part of l20n-in-gecko into the buglist.

The details will depend on the actual build story, which we figure out in bug 1280680.

I suggest to use the user story to describe the actionable details once we have them. P5 until this is actionable, HTH.

Comment 1

[Axel Hecht [:Pike]]

Mass change dependency tree for bug 1279002 into a whiteboard keyword.

Comment 2

[Axel Hecht [:Pike]]

Attachment: package top-level localization in language pack

I've started to take a look at this, but this needs a real design.

There's a nit in l10n.mk that we need to actually package the top-level localization dir, but that's just the surface.

I've looked at mozbuild/mozpack/packager/l10n.py. The immediate idea to just add "**/localization" to toolkit/mozapps/installer/l10n-repack.py doesn't do what we need, as it removes the original localization files.

I'm trying to wrap my head around the repack script, and I struggle. In particular in the light of use running several repacks in one go.

Do we end up repacking en-US -> de, then de -> fr, and then fr -> it?

Comment 3

[Axel Hecht [:Pike]]

glandium, can you help us out here?

Recognized yesterday on #releng that both I and Callek have very old-fashioned expectations on what repacks do ;-)

Comment 4

[Mike Hommey [:glandium]]

As discussed on irc, I think on the long term we should just make l10n repacks artifact builds. However, thinking in terms of quickly implementable, low-risk, changes, I think we can go ahead with something that puts us on that path while not requiring much changes:
- Make the wget-en-us step put the downloaded tarball in some directory other than the objdir. This is kind of gross, but that's mostly what artifact builds do for their cache.
- Make each locale repack start with a clobber (as in remove the objdir) (we've had multiple problems in the past where $locale n+1 would end up with stuff from $locale n), and re-run configure, etc.

Those two steps would get us in a place where implementing l20n would become more tractable, since they would all start from en-us. Technically speaking, only the second step is really required, but the first would make us avoid a download between each repack.

From there, further improvements can be made, such as:
- Make configure faster on l10n repacks by skipping irrelevant stuff.
- Make l10n repacks faster by skipping irrelevant stuff.
- Actually turn l10n repacks into artifact builds.

Comment 5

[Axel Hecht [:Pike]]

Attachment: bug 1280683, keep a pristine copy of en-US installer for repacks,

Let's use a pristine unpackaged directory of the en-US package, and just
rsync that to l10n-stage. That way, all repacks start off with a copy
of en-US without modifications of the previous repack.

Removing clobber-zip, that hasn't been used in ages, on the way.

Review commit: https://reviewboard.mozilla.org/r/94402/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/94402/

Comment 6

[Axel Hecht [:Pike]]

glandium, I considered going down the full route, but then chickened out.

This just does the unpack against a separate dir, and then rsyncs that over to l10n-stage. I'm using --delete to remove files that are not in en-US.

This might be more of a f? than an r?, I only tested this on the mac so far. Try-l10n won't cover this, as that still repacks nightlies, and not try builds, and the logic here only really matters at a phase where the source code updated to nightly.

If this looks OK to you, I'll pound this a bit more on a linux and windows VM.

PS: the base revision is larch, but this should import gracefully. I can confirm that the mac builds start, and have the en-US l20n files in them, and that they're french outside of that.

As Callek said he'd be interested starting from vanilla en-US, too, this might be good to land on m-c regardless of l20n.

Comment 7

[Justin Wood (:Callek)]

(In reply to Axel Hecht [:Pike] from comment #6)
> I only tested this on the mac so far.
> Try-l10n won't cover this, as that still repacks nightlies, and not try
> builds, and the logic here only really matters at a phase where the source
> code updated to nightly.

Try l10n has `"update_gecko_source_to_enUS": False,` so build system changes in the l10n codepaths should work fine.

I also have a way to trigger l10n that *does* depend on the build now (based on the `date` project branch, where we are working on tc nightly stuff)... if for some reason we need that to test this properly.

Comment 8

[Axel Hecht [:Pike]]

Comment on attachment 8812802 [details]
bug 1280683, keep a pristine copy of en-US installer for repacks,

With Callek's help, I pushed this to try, https://treeherder.mozilla.org/#/jobs?repo=try&revision=6f7023b753a02e258c9ae90df62c79dcc7ca96e0&filter-tier=1&filter-tier=2&filter-tier=3, and it busted.

rsync doesn't exist on windows, but there's $(call copy_dir). Not as nice as rsync in the log for sure.

Which brings me to the successful linux builds, they're actually looking nicer in the logs, I think. The rsync is really telling on what's been modified in the repack, and at least to me, is affirmative.

The macs are busted, still need to dig in to that.

The android single-locale builds note that the rsync is removing fennec-53.0a1.de.android-arm-unsigned-unaligned.apk etc. Callek, is that good or bad?

Didn't look at the multi-locale build in that try push yet.

Cancelling review until I have an updated patch.

Comment 9

[Axel Hecht [:Pike]]

Comment on attachment 8812802 [details]
bug 1280683, keep a pristine copy of en-US installer for repacks,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/94402/diff/1-2/

Comment 10

[Axel Hecht [:Pike]]

This now passed on try, https://treeherder.mozilla.org/#/jobs?repo=try&revision=e78505f1deb7764e17576b60db74f8c02d0e3178.

Comment 11

[Mike Hommey [:glandium]]

Comment on attachment 8812802 [details]
bug 1280683, keep a pristine copy of en-US installer for repacks,

https://reviewboard.mozilla.org/r/94402/#review96358

::: mobile/android/locales/Makefile.in
(Diff revision 2)
> -# When we unpack fennec on MacOS X the platform.ini and application.ini are in slightly
> -# different locations that on all other platforms
> -ifeq (Darwin, $(OS_ARCH))
> -GECKO_PLATFORM_INI_PATH='$(STAGEDIST)/platform.ini'
> -FENNEC_APPLICATION_INI_PATH='$(STAGEDIST)/application.ini'
> -else

I'm not a fan of unrelated changes. Please move this to a separate patch.

Comment 12

[Axel Hecht [:Pike]]

Comment on attachment 8812802 [details]
bug 1280683, keep a pristine copy of en-US installer for repacks,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/94402/diff/2-3/

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/274739860180
keep a pristine copy of en-US installer for repacks, r=glandium

Comment 14

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/274739860180