1000208 - make it possible to post b2g mar info to balrog

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Probably requires some mozharness work. bug 858797 may have some helpful work in it.

Comment 1

[[e-mail] (:bhearsum)]

Attachment: add balrog mixin

This patch is step 1 - it just adds a balrog mixin that relies on the existing balrog scripts. It's mostly ripped off from Jordan's work.

I tested it with a quick small script: https://github.com/bhearsum/mozharness/blob/bd57297473259c0abde5e59e2766c7f95129a245/scripts/balrog_test.py

Which created these two blobs:
vhttps://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-20140202020202/data
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-latest/data

Comment 2

[Aki Sasaki (not active)]

Comment on attachment 8411254 [details] [diff] [review]
add balrog mixin

I wonder if we want this under work_dir instead of base_work_dir, since we tend to clobber build/ rather than build/ + all siblings of build/.

I don't think fatal_exit_code is needed if halt_on_failure is False.

I don't think either of these things bother me enough to delay landing.

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 8411254 [details] [diff] [review]
add balrog mixin

(In reply to Aki Sasaki [:aki] from comment #2)
> Comment on attachment 8411254 [details] [diff] [review]
> add balrog mixin
> 
> I wonder if we want this under work_dir instead of base_work_dir, since we
> tend to clobber build/ rather than build/ + all siblings of build/.

Good point, I think that makes sense.

> I don't think fatal_exit_code is needed if halt_on_failure is False.

I'll drop that, too.

Landed as https://hg.mozilla.org/build/mozharness/rev/c48a76ec161a with both points addressed.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: wip patch

This obviously needs more work but I wanted to get some feedback and maybe help on where to source some of the information we need from.

I noticed that we currently get the mar url from releng-private-updates.py, but it seems like we should be able to get it from post_upload.py since we're uploading to FTP, which is more authoritative.

Any idea on how to read PRODUCT_MODEL from the build system would be appreciated too.

Comment 5

[Jordan Lund (:jlund)]

mozharness patch is in production: http://hg.mozilla.org/build/mozharness/rev/9accabdd4358 :)

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 8411985 [details] [diff] [review]
wip patch

Argh, so hard to get the right person!!!!

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8411985 [details] [diff] [review]
wip patch

I don't know how to get PRODUCT_MODEL.  That may be in one of the dot files, maybe .config .
Sure, we can get the mar from post_upload.py, though in bug 979554 I actually want to explicitly define what filename/glob is what, rather than guess from a post_upload.py regex.  I think Jordan may have a post_upload regex already for his desktop work.

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 8411985 [details] [diff] [review]
wip patch

Note: we default to default_actions.

You could:

a) add submit-to-balrog to default_actions, and have a way to skip it if it's not an m-c/m-a flame nightly, or

b) add ["--add-action", "submit-to-balrog"] to the flame build command line args (buildbot-configs/mozilla/b2g_config.py), which will add that action to only flame builds on m-c or m-a (you'll need to avoid adding those options to, say, project branches).


We already have things like

        if not self.query_is_nightly():
            self.info("Not a nightly build. Skipping...")
            return
        if not self.config.get('update'):
            self.info("No updates. Skipping...")
            return

in upload_updates().  We could have some flag in flame's in-tree config that specifies enabling balrog.

Comment 9

[[e-mail] (:bhearsum)]

Attachment: add-flame-mapping.diff

(In reply to Aki Sasaki [:aki] from comment #7)
> Comment on attachment 8411985 [details] [diff] [review]
> wip patch
> 
> I don't know how to get PRODUCT_MODEL.  That may be in one of the dot files,
> maybe .config .

I poked around and AFAICT we can't get it. We already maintain a mapping for Desktop/Android, so it's not the end of the world to do it for b2g, too.

Comment 10

[Aki Sasaki (not active)]

Comment on attachment 8412748 [details] [diff] [review]
add-flame-mapping.diff

> update_platform_map = {
>     'android': ['Android_arm-eabi-gcc3'],
>     'android-armv6': ['Android_arm-eabi-gcc3-armv6'],
>     'android-x86': ['Android_x86-gcc3'],
>     'linux': ['Linux_x86-gcc3'],
>     'linux64': ['Linux_x86_64-gcc3'],
>     'macosx64': ['Darwin_x86-gcc3-u-i386-x86_64', 'Darwin_x86_64-gcc3-u-i386-x86_64'],
>     'win32': ['WINNT_x86-msvc'],
>     'win64': ['WINNT_x86_64-msvc'],
>+    'flame'; ['Flame'],

Maybe this is where your semicolon went ;-)
r=me with a colon.

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8412748 [details] [diff] [review]
add-flame-mapping.diff

Landed, with the semicolon fixed.

Comment 12

[[e-mail] (:bhearsum)]

Attachment: latest wip for mozharness

Currently testing this on my dev master. Changes include:
* Create balrog configs for staging/prod.
** Need to decide if we want to use a separate "b2gbld" user for submission. I suspect we do, since we use different keys/accounts on ftp. Probably need to reorg for the config a bit for this.
* Add as an action for anything using releng-private-updates.py. Not sure if this is the right place, it's enough for testing though.
* Let balrog submitter figure out the update platform. I can't find a better way, and this is what we do for desktop/android.

Comment 13

[[e-mail] (:bhearsum)]

Attachment: use balrog configs

Again, may need to change this a bit if we use b2gbld. We might need separate configs per product.

Also need a patch to flip use_credentials_file=True for some/all device builds. Not sure the best way to do that yet - I've hardcoded it for my staging test.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: latest wip mozharness patch

fixes a bunch of bugs. right now i'm hitting:
13:07:35     INFO -  "PUT /releases/B2G-mozilla-central-nightly-20140508155931/builds/Flame/en-US HTTP/1.1" 400 64
13:07:35     INFO -  Caught HTTPError: Couldn't update release: Release blob contains forbidden domain.
13:07:35     INFO -  Traceback (most recent call last):
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/scripts/updates/balrog-submitter.py", line 55, in <module>
13:07:35     INFO -      props['completeMarUrl'], **partialKwargs)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/cli.py", line 149, in run
13:07:35     INFO -      buildData=data, copyTo=copyTo, alias=alias)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 157, in update_build
13:07:35     INFO -      url_template_vars=url_template_vars)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 103, in request
13:07:35     INFO -      return self.do_request(url, data, method, url_template_vars)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 118, in do_request
13:07:35     INFO -      headers=headers)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
13:07:35     INFO -      r.send(prefetch=prefetch)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
13:07:35     INFO -      self.response.raise_for_status()
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
13:07:35     INFO -      raise http_error
13:07:35     INFO -  requests.exceptions.HTTPError: 400 Client Error


Probably need to start pulling the update URL from the log or something, so we actually get the right one (instead of the update.boot2gecko.org one).

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 8412882 [details] [diff] [review]
use balrog configs

This, in combination with the stuff I'm about to attach seems to work in staging. Build is here:
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/b2g_mozilla-central_flame_nightly/builds/8/steps/run_script/logs/stdio

Balrog data is here:
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-20140509122946/data
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-latest/data

(Ignore the "flame" platform in the second one - that's leftover from some earlier tests.)

Comment 16

[[e-mail] (:bhearsum)]

Attachment: pass credentials file to device build factories

This also passes it to a bunch of other builds that don't need it. Meh?

Comment 17

[[e-mail] (:bhearsum)]

Attachment: fully tested mozharness patch

Here's a summary of the changes:
* Add "submit-to-balrog" action for anything using releng-private-updates.py. I've only tested flame, the others should just work though, right? :)
* Add mar_base_url for the same set of things. I was originally going to change base_url, but that'll break update.boot2gecko.org updates, and I'd like to land this patch before we're ready to switchover, so we can dogfood for a day or two.
* Add balrog configs for staging/prod. Buildbot-configs handles using the right one.
* Fix some silly bugs in BalrogMixin
* Change query_marfile_path to detect the MAR filename rather than hardcode it. This is needed because I'll be changing the filename to include device name in bug 1000207.
* Implement submit_to_balrog step. Not much to say about this one, it just gathers data and gives it BalrogMixin.

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #15)
> Comment on attachment 8412882 [details] [diff] [review]
> use balrog configs
> 
> This, in combination with the stuff I'm about to attach seems to work in
> staging. Build is here:
> http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/b2g_mozilla-
> central_flame_nightly/builds/8/steps/run_script/logs/stdio
> 
> Balrog data is here:
> https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-
> 20140509122946/data
> https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-
> latest/data
> 
> (Ignore the "flame" platform in the second one - that's leftover from some
> earlier tests.)

Actually, I just realized that we're using PRODUCT_DEVICE in the update URL (instead of PRODUCT_MODEL). This means I'll need to update platforms.py to use that, and we'll end up submitting with a lowercase "flame".

Comment 19

[[e-mail] (:bhearsum)]

Attachment: use lowercase flame

Comment 20

[Aki Sasaki (not active)]

Comment on attachment 8412882 [details] [diff] [review]
use balrog configs

More context in the patch would be nice, but looks right.

Comment 21

[Aki Sasaki (not active)]

Comment on attachment 8420205 [details] [diff] [review]
pass credentials file to device build factories

Probably harmless, since we're already somewhat paranoid about device builds.  If we care, we could add another platform config key that enables/disables this.

Comment 22

[Aki Sasaki (not active)]

Comment on attachment 8420206 [details] [diff] [review]
fully tested mozharness patch

>     def query_marfile_path(self):
>         if self.config.get("update_type", "ota") == "fota":
>             output_dir = self.query_device_outputdir()
>             return "%s/fota-update.mar" % output_dir
>         else:
>-            return "%s/dist/b2g-update/b2g-gecko-update.mar" % self.objdir
>+            mardir = "%s/dist/b2g-update" % self.objdir
>+            files = os.listdir(mardir)
>+            if len(files) != 1:
>+                self.fatal("Found none or too many marfiles, don't know what to do.")

Do you think this is an infrastructure error or a build failure?
If infrastructure, could you self.fatal("Found none or too many...", exit_code=3) so that the build turns purple?
1 red 2 orange 3 purple 4 retry iirc.

>+    def submit_to_balrog(self):
>+        if not self.query_is_nightly():
>+            self.info("Not a nightly build, skipping balrog submission.")
>+            return

Can you put an

        if not self.config.get("balrog_api_root"):
            self.info("balrog_api_root not set; skipping balrog submission.")
            return

Otherwise, aiui, the other device builds that use releng-private-updates.py will start dying.

Other than that, this patch looks good.  Thanks!

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] from comment #22)
> Comment on attachment 8420206 [details] [diff] [review]
> fully tested mozharness patch
> 
> >     def query_marfile_path(self):
> >         if self.config.get("update_type", "ota") == "fota":
> >             output_dir = self.query_device_outputdir()
> >             return "%s/fota-update.mar" % output_dir
> >         else:
> >-            return "%s/dist/b2g-update/b2g-gecko-update.mar" % self.objdir
> >+            mardir = "%s/dist/b2g-update" % self.objdir
> >+            files = os.listdir(mardir)
> >+            if len(files) != 1:
> >+                self.fatal("Found none or too many marfiles, don't know what to do.")
> 
> Do you think this is an infrastructure error or a build failure?
> If infrastructure, could you self.fatal("Found none or too many...",
> exit_code=3) so that the build turns purple?
> 1 red 2 orange 3 purple 4 retry iirc.

I'm not really sure. None of the automation creates mar files, so RETRY is not going to have any positive effect. If/When the build system provides us a way to reference a specific MAR (a la Gecko's package-name.mk variables: https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk#52) we can do better here.

Do you have a preference for red vs purple?

> >+    def submit_to_balrog(self):
> >+        if not self.query_is_nightly():
> >+            self.info("Not a nightly build, skipping balrog submission.")
> >+            return
> 
> Can you put an
> 
>         if not self.config.get("balrog_api_root"):
>             self.info("balrog_api_root not set; skipping balrog submission.")
>             return

Will do.

> Otherwise, aiui, the other device builds that use releng-private-updates.py
> will start dying.

I think all of the device builds will have the right things set to successfully submit to Balrog (I'm kindof counting on that, so that we can test them out before we change the in-app update URL), but that's probably not something we should assume going forward.

Comment 24

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> I think all of the device builds will have the right things set to
> successfully submit to Balrog (I'm kindof counting on that, so that we can
> test them out before we change the in-app update URL), but that's probably
> not something we should assume going forward.

Oh, I just realized that I'm not setting the Balrog configs for other devices in buildbot-configs....I'm going to update that patch.

Comment 25

[[e-mail] (:bhearsum)]

Attachment: submit to balrog for other devices

I only added this for devices I found in /data/update-channels on update.boot2gecko.org.

Comment 26

[Armen [:armenzg]]

Merged into production and live.

Comment 27

[Aki Sasaki (not active)]

(In reply to Ben Hearsum [:bhearsum] from comment #23)
> > Do you think this is an infrastructure error or a build failure?
> > If infrastructure, could you self.fatal("Found none or too many...",
> > exit_code=3) so that the build turns purple?
> > 1 red 2 orange 3 purple 4 retry iirc.
> 
> I'm not really sure. None of the automation creates mar files, so RETRY is
> not going to have any positive effect. If/When the build system provides us
> a way to reference a specific MAR (a la Gecko's package-name.mk variables:
> https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/
> package-name.mk#52) we can do better here.
> 
> Do you have a preference for red vs purple?

I suppose it's up in the air since the build system creates mars.
A depend build with a renamed mar might get 2 and might need a clobber, but otherwise it wouldn't be an infra problem.  Up to you.

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 8421005 [details] [diff] [review]
submit to balrog for other devices

>     'hamachi': {
>         'mozharness_config': {
>             'script_name': 'scripts/b2g_build.py',
>             # b2g_build.py will checkout gecko from hg and look up a tooltool manifest given by the
>             # --target name below
>             'extra_args': ['--target', 'hamachi', '--config', 'b2g/releng-fota-updates.py',
>                            '--gaia-languages-file', 'locales/languages_dev.json',
>-                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales'],
>+                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales',
>+                           '--config', GLOBAL_VARS['mozharness_configs']['balrog']],
>             'reboot_command': ['bash', '-c', 'sudo reboot; sleep 600'],
>         },
>         'stage_product': 'b2g',
>         'product_name': 'b2g',
>         'base_name': builder_prefix + '_%(branch)s_%(platform)s',
>         'slaves': SLAVES['mock'],
>         'enable_periodic': True,
>         'enable_dep': False,
>@@ -964,17 +966,18 @@ PLATFORM_VARS = {
>     },
>     'tarako': {
>         'mozharness_config': {
>             'script_name': 'scripts/b2g_build.py',
>             # b2g_build.py will checkout gecko from hg and look up a tooltool manifest given by the
>             # --target name below
>             'extra_args': ['--target', 'tarako', '--config', 'b2g/releng-fota-updates.py',
>                            '--gaia-languages-file', 'locales/languages_dev.json',
>-                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales'],
>+                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales',
>+                           '--config', GLOBAL_VARS['mozharness_configs']['balrog']],
>             'reboot_command': ['bash', '-c', 'sudo reboot; sleep 600'],
>         },
>         'stage_product': 'b2g',
>         'product_name': 'b2g',
>         'base_name': builder_prefix + '_%(branch)s_%(platform)s',
>         'slaves': SLAVES['mock'],
>         'enable_periodic': False,
>         'enable_dep': True,

IIRC you only added balrog stuff to releng-private-updates.py, so tarako and hamachi (using releng-fota-updates.py) won't pick that up.

Comment 29

[[e-mail] (:bhearsum)]

Attachment: address final review comments

OK, I think this addresses everything. The only changes since the last patch are:
* Set exit code to 1 if query_marfile fails.
* Skip Balrog submission if balrog_api_root isn't set
* Add Balrog stuff to releng-fota-updates.py, so they can submit data too.

Comment 30

[Aki Sasaki (not active)]

Comment on attachment 8421095 [details] [diff] [review]
address final review comments

>+            mardir = "%s/dist/b2g-update" % self.objdir
>+            files = os.listdir(mardir)
>+            if len(files) != 1:
>+                self.fatal("Found none or too many marfiles, don't know what to do.", 1)

Please use exit_code=1 rather than a bare 1.
The kwargs should be order-independent; we've already hit an issue where we prepended a kwarg and broke a method call that assumed kwarg order.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 8421005 [details] [diff] [review]
submit to balrog for other devices

I landed this (and the mozharness patch) briefly, and then I realized that we need bug 1007180 before we can submit to Balrog in production. I backed them out to make sure they don't accidentally get merged before that's done.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 8421095 [details] [diff] [review]
address final review comments

B2G account changes were made - I landed this. I'm triggering a Flame nightly to verify things.

Comment 33

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #32)
> Comment on attachment 8421095 [details] [diff] [review]
> address final review comments
> 
> B2G account changes were made - I landed this. I'm triggering a Flame
> nightly to verify things.

Oops, no I'm not! This hasn't merged to production.

Comment 34

[Armen [:armenzg]]

Live in production.

Comment 35

[[e-mail] (:bhearsum)]

The submitter ran for a nightly I triggered, but it failed (thankfully, not in a way that burns the job). Turns out I forgot to add b2gbld passwords to BuildSlaves.py. Filed bug 1009657 for this, and it should be fixed before the next scheduled nightlies run.

Comment 36

[[e-mail] (:bhearsum)]

Still not working. Looks like we're trying to submit as ffxbld for some reason:
07:21:32     INFO -  Caught HTTPError: ffxbld is not allowed to access /releases/:name/builds/:platform/:locale by PUT

Comment 37

[[e-mail] (:bhearsum)]

Attachment: override balrog_username for b2g stuff

Oops

Comment 38

[[e-mail] (:bhearsum)]

balrog_username override is in production

Comment 39

[[e-mail] (:bhearsum)]

Updated the wrong thing, needs a Mozharness config change =\

Comment 40

[[e-mail] (:bhearsum)]

Attachment: per-product usernames

I tested this one in staging: http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/b2g_mozilla-central_flame_nightly/builds/17

Comment 41

[[e-mail] (:bhearsum)]

Comment on attachment 8422440 [details] [diff] [review]
override balrog_username for b2g stuff

I backed this out, because it doesn't do anything.

Comment 42

[[e-mail] (:bhearsum)]

Comment on attachment 8423370 [details] [diff] [review]
per-product usernames

I translanted right to production so I can catch the next round of nightlies.

Comment 43

[[e-mail] (:bhearsum)]

Things are submitting now. There will be some follow-ups, but I'll be filing new bugs for them.