654152 - Move all-locales for repacks from source repo into build config

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Axel Hecht [:Pike]]

We should move all-locales out of mozilla-central and friends and into build-config.

For the rapid release cycle stuff, we want less locales on central, most locales on aurora, and from there less on beta, and less on release.

The merge logic is going to make all all-locales files the one of the upper branch every six weeks, so using all-locales in the source code repo is wrong.

Putting them into build config sounds much more like it. Challenge as with other cases is that the l10n dashboard merrily goes down on short notice without failover (working on that with laura t.). So probably want to do something that lands in buildbot-configs. How to make sure that that and the l10n dashboard use the same list, not sure. I'm open for suggestions.

This blocks building less locales on central than on aurora.

Comment 1

[Armen [:armenzg]]

It seems like we would need to modify HgAllLocalesPoller (or use another class) to parse those new configs instead of parsing something like http://hg.mozilla.org/releases/l10n-mozilla-1.9.1/?style=raw

[1] http://hg.mozilla.org/build/buildbotcustom/file/tip/changes/hgpoller.py#l388

Comment 2

[Axel Hecht [:Pike]]

There are two pieces here:

- nightlies, or any other breed of "on en-US change" builds.
- l10n deps, as in, "on l10n change" builds.

The latter are broken anyway, that's hitting anything but Firefox. If a localizer lands a toolkit string, fennec and thunderbird kick off builds, no matter what.

This is more about the nightlies, and release configs. We don't want those to break just because they're trying to do builds for locales that aren't there. To be precise, that won't even have hg repos on those branches, on purpose.

Comment 3

[Armen [:armenzg]]

You are right. This is a change to the list of locales that get triggered.

For that we use TriggerableL10n (IIRC) which should use point to a new localesFile and process it accordingly.
http://hg.mozilla.org/build/buildbotcustom/file/d590bb35a80b/misc.py

Comment 4

[Armen [:armenzg]]

Axel what is your expected timeline?

Comment 5

[Armen [:armenzg]]

For a similar bug for "release" rather than "nightly" changes see bug 415895.

Comment 6

[Axel Hecht [:Pike]]

I don't have a timeline, there's stuff that I won't be able to work on 'til this is getting fixed. Like adding more locales on aurora, or dropping locales on central. Adding locales is of course the earlier the better. Removing locales doesn't have a priority on my list.

Comment 7

[Armen [:armenzg]]

Makes sense.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #0)
> We should move all-locales out of mozilla-central and friends and into
> build-config.
> 
> For the rapid release cycle stuff, we want less locales on central, most
> locales on aurora, and from there less on beta, and less on release.

Split this out to separate bug#655129.




> The merge logic is going to make all all-locales files the one of the upper
> branch every six weeks, so using all-locales in the source code repo is wrong.
> 
> Putting them into build config sounds much more like it. Challenge as with
> other cases is that the l10n dashboard merrily goes down on short notice
> without failover (working on that with laura t.). So probably want to do
> something that lands in buildbot-configs. How to make sure that that and the
> l10n dashboard use the same list, not sure. I'm open for suggestions.
Remaining work here is about tidying up where configs are stored.


> This blocks building less locales on central than on aurora.
Per triage, we do not believe this blocks bug#655129, but if we're missing
something please provide details.

Comment 9

[Dwayne Bailey]

Could you give me some indication of when this will get fixed?  Can we see this in time for Fx5?

The reason I ask is that one of our African teams, doing Swahili is affected by this.  They were not in Fx4 so they aren't on Axel's dashboard.  We can't sign off, etc.  They are really motivated having done the whole translation in about 6 weeks, Phenomenal. But seriously demotivating if they can't get their work to the Swahili community.

Comment 10

[Armen [:armenzg]]

Attachment: [buildbotcustom] allowing passing localesURL to L10nMixin

This patch adds one new way of using L10nMixin:
* pass a localesURL from where to fetch the list of locales

Before we could also do the following:
* pass a list of locales (no fetching)
* construct a URL that pointed to the source repo's browser/locales/all-locales

This patch should be backward compatible.

I have tested this on staging and was allowed to reduce the number of locales.

Comment 11

[Armen [:armenzg]]

Attachment: [buildbot-configs] pass localesURL and add all-locales

Comment 12

[Axel Hecht [:Pike]]

Comment on attachment 533772 [details] [diff] [review]
[buildbotcustom] allowing passing localesURL to L10nMixin

r=me.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 533773 [details] [diff] [review]
[buildbot-configs] pass localesURL and add all-locales

I know I'm not marked for review here, but I don't think you should be pulling the default branch -- use "production". Also, you need to set localesURL for all l10n branches, not just these three.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 533772 [details] [diff] [review]
[buildbotcustom] allowing passing localesURL to L10nMixin

branch should still be passed, as it's used in building the SourceStamp. r=me with that change.

Comment 15

[Armen [:armenzg]]

Attachment: [buildbot-configs] pass localesURL and add all-locales (this also includes 1.9.1, 1.9.2 and 2.0)

Might as well use the same style for all release branches where we have L10n enabled.
This patch also includes 1.9.1, 1.9.2 and 2.0

Comment 16

[chris hofmann]

My understanding is that we have locales that we need to add for firefox 5 and we can't add them until this bug gets fix.  -> blocking 5.0

Comment 17

[Armen [:armenzg]]

I will be landing all of this on Monday.

Comment 18

[Armen [:armenzg]]

Unfortunately we are avoiding doing landings until a very large set of patches land from bug 557260.

If anyone is eager for the configs patch to land (since it can't affect anything) before Wednesday I can do so. Otherwise, I will land both on Wednesday.

Comment 19

[Armen [:armenzg]]

Landed but not yet deployed:
http://hg.mozilla.org/build/buildbot-configs/rev/4aa4a623fc7f
http://hg.mozilla.org/build/buildbotcustom/rev/c8b279a75aac

Comment 20

[Armen [:armenzg]]

This got deployed.
I triggered a new Linux nightly and it triggered all the locales.

Axel shall we remove all-locales from all the different source repos?

I don't see anything left to be done besides that.

Comment 21

[Axel Hecht [:Pike]]

I'll need to drop the use of all-locales on l10n master, too, before we can remove them.

Also 

(In reply to comment #13)
> Comment on attachment 533773 [details] [diff] [review] [review]
> [buildbot-configs] pass localesURL and add all-locales
> 
> I know I'm not marked for review here, but I don't think you should be
> pulling the default branch -- use "production". Also, you need to set
> localesURL for all l10n branches, not just these three.

Did that review comment get addressed? That'd be interesting to understand.

And, the configs are only set up for Firefox so far, not for mobile, or other apps?

Comment 22

[Armen [:armenzg]]

Yes, all of the branches were we generate L10n nightly repacks.

Not for mobile or any other apps.

Comment 23

[Armen [:armenzg]]

Attachment: [configs] poll production rather than default

We are polling default rather than production let's fix it.

I missed what bhearsum pointed out on his review.

Comment 24

[Armen [:armenzg]]

I see a bunch of nightly builds on buildbot & tinderbox but not on Axel's dashboard.

Axel can you confirm that this is fixed? (besides the s/default/production/ fix)

On another note, can we file a bug to take care of removing all-locales from source repos?

Shall the next step of this bug to manage mobile nightly builds in our repos as well?

Comment 25

[Armen [:armenzg]]

Comment on attachment 536715 [details] [diff] [review]
[configs] poll production rather than default

http://hg.mozilla.org/build/buildbot-configs/rev/ef803d48f0ef

Comment 26

[[e-mail] (:bhearsum)]

This hit production today.

Comment 27

[Justin Wood (:Callek)]

Comment on attachment 533772 [details] [diff] [review]
[buildbotcustom] allowing passing localesURL to L10nMixin


>     def __init__(self, platform, repo='http://hg.mozilla.org/', branch=None,
>             baseTag='default', localesFile="browser/locales/all-locales",
>-            locales=None):
>+            locales=None, localesURL=None):
>+        """
>+        You can call this class with either a defined list of locales or
>+        a URL that contains the list of locales
>+        """
>         self.branch = branch
>         self.baseTag = baseTag
>-        self.repo = repo
>-        self.localesFile = localesFile
>+        self.localesURL = localesURL if localesURL else "%s%s/raw-file/%s/%s" \
>+                          % (repo, branch, revision or self.baseTag, localesFile)

>@@ -141,25 +145,22 @@ class L10nMixin(object):
>         in the scheduler OR it returns a Deferred.
> 
>         You want to call this method via defer.maybeDeferred().
>         """
>         if self.locales:
>             log.msg('L10nMixin.getLocales():: The user has set a list of locales')
>             return self.locales
>         else:
>-            revision = revision or self.baseTag
>-            localesURL = "%s%s/raw-file/%s/%s" \
>-                                            % (self.repo, self.branch, revision, self.localesFile)
>-            log.msg("L10nMixin:: Getting locales from: "+localesURL)
>+            log.msg("L10nMixin:: Getting locales from: "+self.localesURL)
>             # we expect that getPage will return the output of "all-locales"
>             # or "shipped-locales" or any file that contains a locale per line
>             # in the begining of the line e.g. "en-GB" or "ja linux win32"
>             # getPage returns a defered that will return a string
>-            d = getPage(localesURL, timeout = 5 * 60)
>+            d = getPage(self.localesURL, timeout = 5 * 60)
>             d.addCallback(lambda data: ParseLocalesFile(data))
>             return d

Since this is a defferred section, moving the localesURL generation up to init, actually drops the potential to use a source-stamps "revision" here. Which used to exist.

I also haven't verified, but I suspect the else clause that creates the localesURL would throw since revision is not in scope there.

What I would suggest/have suggested is to "generate" the localesURL where it is now, but allow us to replace with %s{revision} even in the config based url. But at least fixing the basic generation case is requested. (That said, it is fairly easy for me to import this code in a way that won't break SeaMonkey, so if it is desired to drop the support for using a sourcestamp revision, I'm ok with that.)

Comment 28

[Armen [:armenzg]]

Sorry Callek this already went live a while ago but thanks for the review.
Feel free to file a bug for any follow up work and we'll deal with it there.

Comment 29

[Armen [:armenzg]]

(In reply to comment #24)
> I see a bunch of nightly builds on buildbot & tinderbox but not on Axel's
> dashboard.
> 
> Axel can you confirm that this is fixed? (besides the s/default/production/
> fix)
> 
> On another note, can we file a bug to take care of removing all-locales from
> source repos?
> 
> Shall the next step of this bug to manage mobile nightly builds in our repos
> as well?

Axel can you please respond to these?

Comment 30

[Axel Hecht [:Pike]]

I have no idea what's up with my nightly dashboard, the cron job doesn't seem to produce anything no more, though running the script manually works. Didn't bother investigating yet.

We can file a bug on removing all-locales, but we can't fix it before the dashboard stop polling it.

Yes for mobile.

Comment 31

[Armen [:armenzg]]

(In reply to comment #30)
> I have no idea what's up with my nightly dashboard, the cron job doesn't
> seem to produce anything no more, though running the script manually works.
> Didn't bother investigating yet.
> 
> We can file a bug on removing all-locales, but we can't fix it before the
> dashboard stop polling it.
> 
> Yes for mobile.

OK. I have filed the follow up work as bug 664806 and bug 664809.

This specific bug is done.

Comment 32

[Justin Wood (:Callek)]

(In reply to comment #28)
> Sorry Callek this already went live a while ago but thanks for the review.
> Feel free to file a bug for any follow up work and we'll deal with it there.

This is now Bug 668724