The default bug view has changed. See this FAQ.

android single locale central/aurora nightlies not reporting to balrog

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: bhearsum, Assigned: kmoir)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(5 attachments, 1 obsolete attachment)

(Reporter)

Description

3 years ago
The regular multilocale builds are, but the single locales aren't. Looks like I missed these when I switched everything else over. Nobody noticed until we realized it today while talking about bug 1019724
(Reporter)

Updated

3 years ago
Blocks: 1019724
(Assignee)

Updated

3 years ago
Assignee: nobody → kmoir
(Assignee)

Comment 1

3 years ago
So I was just looking at 

https://aus4-admin-dev.allizom.org/releases.html

And I can't find any recent info for Fennec builds from this year

Also, this link doesn't yield anything.   
https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-latest/data

Does it need to be inserted from another db?
Flags: needinfo?(bhearsum)
(Reporter)

Comment 2

3 years ago
(In reply to Kim Moir [:kmoir] from comment #1)
> So I was just looking at 
> 
> https://aus4-admin-dev.allizom.org/releases.html
> 
> And I can't find any recent info for Fennec builds from this year
> 
> Also, this link doesn't yield anything.   
> https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-
> latest/data
> 
> Does it need to be inserted from another db?

Yeah, probably your best bet is to download one from production. Eg: https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-latest/data

And then upload it to dev using the form on https://aus4-admin-dev.allizom.org/releases.html. Just let me know if you any trouble with that.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 3

3 years ago
So I've been familiarizing myself with the code base over the last while since I currently know very little about l10n or how en-US vs multi repacks are built.  In any case, I have a few questions

https://github.com/mozilla/build-mozharness/blob/master/scripts/mobile_l10n.py

is used to generate en-US repacks for Android and submit them to balrog. Currently these are enabled for m-c and m-a nightlies.

However, the single locales are not broken out from the multi apks and added to balrog like they are for desktop.  

Desktop example
https://aus4-admin.mozilla.org/releases/Firefox-mozilla-aurora-nightly-20140722004002/data

The multi locale builds for Android are currently built by this script
https://github.com/mozilla/build-mozharness/blob/master/scripts/multil10n.py
which really invokes
https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/l10n/multi_locale_build.py

which doesn't upload the snippets to balrog.

So the work that needs to be done is to modify the existing multi_locale mozharness scripts so that they update the balrog db, is that correct?
Flags: needinfo?(bhearsum)
(Reporter)

Comment 4

3 years ago
A couple of things upfront to make sure there's no confusion:
- multilocale.py generates the multilocale apk, eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-07-22-03-02-01-mozilla-central-android/fennec-34.0a1.multi.android-arm.apk. It doesn't generate any individual locale repacks.
- mobile_l10n.py generates a set of individual locale repacks, eg: http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-07-22-03-02-01-mozilla-central-android-l10n/

I know this is all muddled, hopefully this helps to clarify it...

(In reply to Kim Moir [:kmoir] from comment #3)
> So I've been familiarizing myself with the code base over the last while
> since I currently know very little about l10n or how en-US vs multi repacks
> are built.  In any case, I have a few questions
> 
> https://github.com/mozilla/build-mozharness/blob/master/scripts/mobile_l10n.
> py
> 
> is used to generate en-US repacks for Android and submit them to balrog.
> Currently these are enabled for m-c and m-a nightlies.

Not quite. This script is used to generate _l10n_ repacks. It doesn't know how to submit anything to Balrog yet. We have logic in mozharness that knows how to do that (https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/updates/balrog.py) but it's not used by this script yet. You can see an example of how to use the Mixin in b2g_build.py: https://github.com/mozilla/build-mozharness/blob/master/scripts/b2g_build.py

> However, the single locales are not broken out from the multi apks and added
> to balrog like they are for desktop.  

The single locales are "broken out" (in that they're generated and uploaded, per above), but you're right that they aren't submitting to Balrog.

> The multi locale builds for Android are currently built by this script
> https://github.com/mozilla/build-mozharness/blob/master/scripts/multil10n.py
> which really invokes
> https://github.com/mozilla/build-mozharness/blob/master/mozharness/mozilla/
> l10n/multi_locale_build.py
> 
> which doesn't upload the snippets to balrog.
> 
> So the work that needs to be done is to modify the existing multi_locale
> mozharness scripts so that they update the balrog db, is that correct?

I know it's rather confusing how, but the multilocale builds already report to Balrog. Note that multilocale builds always set their locale code to "en-US", which is why we have en-US information in places such as https://aus4-admin.mozilla.org/releases/Fennec-mozilla-central-nightly-latest/data.

The trick here is that while multil10n.py is what does the multilocale build, it's called by one of our complicated Buildbot factories, which then does the Balrog submission afterwards. This is a little clearer if you look at a Build like: http://buildbot-master73.srv.releng.usw2.mozilla.com:8001/builders/Android%202.2%20mozilla-central%20nightly/builds/14 -- late in the job, "mozharness_multilocale" runs, and then the factory both uploads a snippets to aus3, and runs the "submit_balrog_updates" step.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 5

3 years ago
So I have had to do a lot of hacking to get a limited set of builders on my dev-master up and disable all the superfluous nightly builds that were being generated and stealing my slaves. I should be able to test the changes in my mh repo now that my test environment is in a more stable state.
(Assignee)

Comment 6

3 years ago
I'm iterating through patches for this on my dev-master.  It's slow going but I'm making progress.
(Assignee)

Comment 7

3 years ago
This is the data I have submitted through the automation of patches I have on my dev-master.

https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-20140818030205/data

This is just for Android 2.3 mozilla-central l10n nightly-1, I didn't run the other builds yet.  

As a sanity check, does the data look right? It looks good to me but I have been staring at JSON all day.
Flags: needinfo?(bhearsum)
(Reporter)

Comment 8

3 years ago
(In reply to Kim Moir [:kmoir] from comment #7)
> This is the data I have submitted through the automation of patches I have
> on my dev-master.
> 
> https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-central-nightly-
> 20140818030205/data
> 
> This is just for Android 2.3 mozilla-central l10n nightly-1, I didn't run
> the other builds yet.  
> 
> As a sanity check, does the data look right? It looks good to me but I have
> been staring at JSON all day.

It looks reasonable to me. The same data is also in Fennec-mozilla-central-nightly-latest (which is expected - it's supposed to go in a dated and a -latest blob), which has a rule pointing at it. URLs like https://aus4-dev.allizom.org/update/4/Fennec/27.0a1/20130930093916/Android_arm-eabi-gcc3/be/nightly/4.2.2/default/default/27.0a1/update.xml are showing a response, so that's a very good sign.
Flags: needinfo?(bhearsum)
(Assignee)

Comment 9

3 years ago
I have green builds on m-c for all android 10n nightly- 1 to 5, now testing patches for m-a
(Assignee)

Comment 10

3 years ago
Created attachment 8475623 [details]
log for dev master from runng   Android 2.3 mozilla-aurora l10n nightly-1

http://dev-master1.srv.releng.scl3.mozilla.com:8039/builders/Android%202.3%20mozilla-aurora%20l10n%20nightly-1/builds/9

So I wasn't able to get m-a Android l10n nightlies to run on my dev-master today.  They all fail with make installers.  At first I thought it was my changes but in fact the same thing happened when I changed my master to point to the production mozharness scripts versus my user repo.  The nightly builds did work last night, so I guess I'll look at tonight's nightlies and see if the error occurs.
(Assignee)

Comment 11

3 years ago
Okay, last night's nightly changeset compiled and I was able to get the single locale builds submitting to balrog for m-a.  So I'll have patches up soon.
(Assignee)

Comment 12

3 years ago
Last comment should have included link

https://aus4-admin-dev.allizom.org/releases/Fennec-mozilla-aurora-nightly-20140820004001/data
(Assignee)

Comment 13

3 years ago
Created attachment 8475977 [details] [diff] [review]
bug1019919.patch

patch tested in staging with green nightly l10n builds on m-c and m-a
Attachment #8475977 - Flags: feedback?(bhearsum)
(Reporter)

Comment 14

3 years ago
Comment on attachment 8475977 [details] [diff] [review]
bug1019919.patch

Review of attachment 8475977 [details] [diff] [review]:
-----------------------------------------------------------------

Most of this is great!

My only quibble is that you shouldn't redefine the balrog* username and api root. Those are already defined for production and staging in https://github.com/mozilla/build-mozharness/tree/master/configs/balrog. You should add those as additional configs for these builders rather than duplicate them. This should have the added bonus of making submission work in staging without needing your own mozharness repo. You can see an example of this in b2g_config.py: https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/b2g_config.py#L867
Attachment #8475977 - Flags: feedback?(bhearsum) → feedback+
(Assignee)

Comment 15

3 years ago
Created attachment 8476226 [details] [diff] [review]
bug1019919-2.patch

mozharness patch
Attachment #8476226 - Flags: review?(bhearsum)
(Assignee)

Comment 16

3 years ago
Created attachment 8476231 [details] [diff] [review]
bug1019919bbcustom.patch

mobile_l10n.py is actually added via buildbotcustom so I added the link to balrog/production.py here.  If I added both staging and production files it defaults to production and failed on my dev-master.  Not sure why.
Attachment #8476231 - Flags: review?(bhearsum)
(Reporter)

Updated

3 years ago
Attachment #8476226 - Flags: review?(bhearsum) → review+
(Reporter)

Comment 17

3 years ago
Comment on attachment 8476231 [details] [diff] [review]
bug1019919bbcustom.patch

Review of attachment 8476231 [details] [diff] [review]:
-----------------------------------------------------------------

::: misc.py
@@ +1971,5 @@
>                      mobile_l10n_builders.append(builderName)
>                      extra_args = ['--cfg',
>                                    'single_locale/%s_%s.py' % (name, platform),
> +                                  '--cfg',
> +                                  'balrog/production.py',

We shouldn't be hardcoding production here...it should come out of localconfig.py (which is a symlink to production_config.py or staging_config.py) instead. I _think_ this is accessible through config['mozharness_configs']['balrog'].
Attachment #8476231 - Flags: review?(bhearsum) → review-
(Assignee)

Comment 18

3 years ago
Created attachment 8476694 [details] [diff] [review]
bug1019919bbcustom-2.patch

Thanks for the suggestion for improvement Ben. I tested this in staging and it worked.
Attachment #8476231 - Attachment is obsolete: true
Attachment #8476694 - Flags: review?(bhearsum)
(Reporter)

Comment 19

3 years ago
Comment on attachment 8476694 [details] [diff] [review]
bug1019919bbcustom-2.patch

Review of attachment 8476694 [details] [diff] [review]:
-----------------------------------------------------------------

Woot, ship it!
Attachment #8476694 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

3 years ago
Attachment #8476226 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Attachment #8476694 - Flags: checked-in+
(Assignee)

Comment 20

3 years ago
In production
(Assignee)

Comment 21

3 years ago
Created attachment 8477384 [details] [diff] [review]
to fix bustage

I missed getting this patch approved for production.  It was in my dev-master.

This is the log from last night that this patch addresses
04:32:01    FATAL - Uncaught exception: Traceback (most recent call last):
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1251, in run
04:32:01    FATAL -     self.run_action(action)
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1193, in run_action
04:32:01    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/base/script.py", line 1134, in _possibly_run_method
04:32:01    FATAL -     return getattr(self, method_name)()
04:32:01    FATAL -   File "scripts/scripts/mobile_l10n.py", line 607, in submit_to_balrog
04:32:01    FATAL -     self.submit_balrog_updates()
04:32:01    FATAL -   File "/builds/slave/m-cen-and-l10n_5-0000000000000/scripts/mozharness/mozilla/updates/balrog.py", line 33, in submit_balrog_updates
04:32:01    FATAL -     "--username", c["balrog_usernames"][product],
04:32:01    FATAL - KeyError: u'mobile'
04:32:01    FATAL - Running post_fatal callback...
04:32:01    FATAL - Exiting -1
(Assignee)

Comment 22

3 years ago
Comment on attachment 8477384 [details] [diff] [review]
to fix bustage

r=bustage
Attachment #8477384 - Flags: checked-in+
(Assignee)

Comment 23

3 years ago
I retriggered the builds that failed last night and they were fine, and updated data to balrog as expected.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.