1019919 - android single locale central/aurora nightlies not reporting to balrog

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

The regular multilocale builds are, but the single locales aren't. Looks like I missed these when I switched everything else over. Nobody noticed until we realized it today while talking about bug 1019724

Comment 1

[Kim Moir [:kmoir] ET]

So I was just looking at 

https://aus4-admin-dev.allizom.org/releases.html

And I can't find any recent info for Fennec builds from this year

Also, this link doesn't yield anything.   
https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-latest/data

Does it need to be inserted from another db?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] from comment #1)
> So I was just looking at 
> 
> https://aus4-admin-dev.allizom.org/releases.html
> 
> And I can't find any recent info for Fennec builds from this year
> 
> Also, this link doesn't yield anything.   
> https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-
> latest/data
> 
> Does it need to be inserted from another db?

Yeah, probably your best bet is to download one from production. Eg: https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-latest/data

And then upload it to dev using the form on https://aus4-admin-dev.allizom.org/releases.html. Just let me know if you any trouble with that.

Comment 3

[Kim Moir [:kmoir] ET]

So I've been familiarizing myself with the code base over the last while since I currently know very little about l10n or how en-US vs multi repacks are built.  In any case, I have a few questions

https://github.com/mozilla/build-mozharness/blob/master/scripts/mobile_l10n.py

is used to generate en-US repacks for Android and submit them to balrog. Currently these are enabled for m-c and m-a nightlies.

However, the single locales are not broken out from the multi apks and added to balrog like they are for desktop.  

Desktop example
https://aus4-admin.mozilla.org/releases/Firefox-mozilla-aurora-nightly-20140722004002/data

The multi locale builds for Android are currently built by this script
https://github.com/mozilla/build-mozharness/blob/master/scripts/multil10n.py
which really invokes
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/l10n/multi_locale_build.py

which doesn't upload the snippets to balrog.

So the work that needs to be done is to modify the existing multi_locale mozharness scripts so that they update the balrog db, is that correct?

Comment 4

[[e-mail] (:bhearsum)]

A couple of things upfront to make sure there's no confusion:
- multilocale.py generates the multilocale apk, eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-07-22-03-02-01-mozilla-central-android/fennec-34.0a1.multi.android-arm.apk. It doesn't generate any individual locale repacks.
- mobile_l10n.py generates a set of individual locale repacks, eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-07-22-03-02-01-mozilla-central-android-l10n/

I know this is all muddled, hopefully this helps to clarify it...

(In reply to Kim Moir [:kmoir] from comment #3)
> So I've been familiarizing myself with the code base over the last while
> since I currently know very little about l10n or how en-US vs multi repacks
> are built.  In any case, I have a few questions
> 
> https://github.com/mozilla/build-mozharness/blob/master/scripts/mobile_l10n.
> py
> 
> is used to generate en-US repacks for Android and submit them to balrog.
> Currently these are enabled for m-c and m-a nightlies.

Not quite. This script is used to generate _l10n_ repacks. It doesn't know how to submit anything to Balrog yet. We have logic in mozharness that knows how to do that (https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/updates/balrog.py) but it's not used by this script yet. You can see an example of how to use the Mixin in b2g_build.py: https://github.com/mozilla/build-mozharness/blob/master/scripts/b2g_build.py

> However, the single locales are not broken out from the multi apks and added
> to balrog like they are for desktop.  

The single locales are "broken out" (in that they're generated and uploaded, per above), but you're right that they aren't submitting to Balrog.

> The multi locale builds for Android are currently built by this script
> https://github.com/mozilla/build-mozharness/blob/master/scripts/multil10n.py
> which really invokes
> https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/
> l10n/multi_locale_build.py
> 
> which doesn't upload the snippets to balrog.
> 
> So the work that needs to be done is to modify the existing multi_locale
> mozharness scripts so that they update the balrog db, is that correct?

I know it's rather confusing how, but the multilocale builds already report to Balrog. Note that multilocale builds always set their locale code to "en-US", which is why we have en-US information in places such as https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-latest/data.

The trick here is that while multil10n.py is what does the multilocale build, it's called by one of our complicated Buildbot factories, which then does the Balrog submission afterwards. This is a little clearer if you look at a Build like: http://buildbot-master73.srv.releng.usw2.mozilla.com:8001/builders/Android%202.2%20mozilla-central%20nightly/builds/14 -- late in the job, "mozharness_multilocale" runs, and then the factory both uploads a snippets to aus3, and runs the "submit_balrog_updates" step.

Comment 5

[Kim Moir [:kmoir] ET]

So I have had to do a lot of hacking to get a limited set of builders on my dev-master up and disable all the superfluous nightly builds that were being generated and stealing my slaves. I should be able to test the changes in my mh repo now that my test environment is in a more stable state.

Comment 6

[Kim Moir [:kmoir] ET]

I'm iterating through patches for this on my dev-master.  It's slow going but I'm making progress.

Comment 7

[Kim Moir [:kmoir] ET]

This is the data I have submitted through the automation of patches I have on my dev-master.

https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-20140818030205/data

This is just for Android 2.3 mozilla-central l10n nightly-1, I didn't run the other builds yet.  

As a sanity check, does the data look right? It looks good to me but I have been staring at JSON all day.

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] from comment #7)
> This is the data I have submitted through the automation of patches I have
> on my dev-master.
> 
> https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-
> 20140818030205/data
> 
> This is just for Android 2.3 mozilla-central l10n nightly-1, I didn't run
> the other builds yet.  
> 
> As a sanity check, does the data look right? It looks good to me but I have
> been staring at JSON all day.

It looks reasonable to me. The same data is also in Fennec-mozilla-central-nightly-latest (which is expected - it's supposed to go in a dated and a -latest blob), which has a rule pointing at it. URLs like https://aus4-dev.allizom.org/update/4/Fennec/27.0a1/20130930093916/Android_arm-eabi-gcc3/be/nightly/4.2.2/default/default/27.0a1/update.xml are showing a response, so that's a very good sign.

Comment 9

[Kim Moir [:kmoir] ET]

I have green builds on m-c for all android 10n nightly- 1 to 5, now testing patches for m-a

Comment 10

[Kim Moir [:kmoir] ET]

Attachment: log for dev master from runng Android 2.3 mozilla-aurora l10n nightly-1

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/Android%202.3%20mozilla-aurora%20l10n%20nightly-1/builds/9

So I wasn't able to get m-a Android l10n nightlies to run on my dev-master today.  They all fail with make installers.  At first I thought it was my changes but in fact the same thing happened when I changed my master to point to the production mozharness scripts versus my user repo.  The nightly builds did work last night, so I guess I'll look at tonight's nightlies and see if the error occurs.

Comment 11

[Kim Moir [:kmoir] ET]

Okay, last night's nightly changeset compiled and I was able to get the single locale builds submitting to balrog for m-a.  So I'll have patches up soon.

Comment 12

[Kim Moir [:kmoir] ET]

Last comment should have included link

https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-aurora-nightly-20140820004001/data

Comment 13

[Kim Moir [:kmoir] ET]

Attachment: bug1019919.patch

patch tested in staging with green nightly l10n builds on m-c and m-a

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 8475977 [details] [diff] [review]
bug1019919.patch

Review of attachment 8475977 [details] [diff] [review]:
-----------------------------------------------------------------

Most of this is great!

My only quibble is that you shouldn't redefine the balrog* username and api root. Those are already defined for production and staging in https://github.com/mozilla/build-mozharness/tree/master/configs/balrog. You should add those as additional configs for these builders rather than duplicate them. This should have the added bonus of making submission work in staging without needing your own mozharness repo. You can see an example of this in b2g_config.py: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/b2g_config.py#L867

Comment 15

[Kim Moir [:kmoir] ET]

Attachment: bug1019919-2.patch

mozharness patch

Comment 16

[Kim Moir [:kmoir] ET]

Attachment: bug1019919bbcustom.patch

mobile_l10n.py is actually added via buildbotcustom so I added the link to balrog/production.py here.  If I added both staging and production files it defaults to production and failed on my dev-master.  Not sure why.

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 8476231 [details] [diff] [review]
bug1019919bbcustom.patch

Review of attachment 8476231 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +1971,5 @@
>                      mobile_l10n_builders.append(builderName)
>                      extra_args = ['--cfg',
>                                    'single_locale/%s_%s.py' % (name, platform),
> +                                  '--cfg',
> +                                  'balrog/production.py',

We shouldn't be hardcoding production here...it should come out of localconfig.py (which is a symlink to production_config.py or staging_config.py) instead. I _think_ this is accessible through config['mozharness_configs']['balrog'].

Comment 18

[Kim Moir [:kmoir] ET]

Attachment: bug1019919bbcustom-2.patch

Thanks for the suggestion for improvement Ben. I tested this in staging and it worked.

Comment 19

[[e-mail] (:bhearsum)]

Comment on attachment 8476694 [details] [diff] [review]
bug1019919bbcustom-2.patch

Review of attachment 8476694 [details] [diff] [review]:
-----------------------------------------------------------------

Woot, ship it!

Comment 20

[Kim Moir [:kmoir] ET]

In production

Comment 21

[Kim Moir [:kmoir] ET]

Attachment: to fix bustage

I missed getting this patch approved for production.  It was in my dev-master.

This is the log from last night that this patch addresses
04:32:01    FATAL - Uncaught exception: Traceback (most recent call last):
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1251, in run
04:32:01    FATAL -     self.run_action(action)
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1193, in run_action
04:32:01    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1134, in _possibly_run_method
04:32:01    FATAL -     return getattr(self, method_name)()
04:32:01    FATAL -   File "scripts/scripts/mobile_l10n.py", line 607, in submit_to_balrog
04:32:01    FATAL -     self.submit_balrog_updates()
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/mozilla/updates/balrog.py", line 33, in submit_balrog_updates
04:32:01    FATAL -     "--username", c["balrog_usernames"][product],
04:32:01    FATAL - KeyError: u'mobile'
04:32:01    FATAL - Running post_fatal callback...
04:32:01    FATAL - Exiting -1

Comment 22

[Kim Moir [:kmoir] ET]

Comment on attachment 8477384 [details] [diff] [review]
to fix bustage

r=bustage

Comment 23

[Kim Moir [:kmoir] ET]

I retriggered the builds that failed last night and they were fine, and updated data to balrog as expected.