Closed Bug 1000208 Opened 6 years ago Closed 6 years ago

make it possible to post b2g mar info to balrog

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: bhearsum)

References

Details

Attachments

(7 files, 6 obsolete files)

1.94 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
1.42 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
2.51 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
491 bytes, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
7.95 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
13.38 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
2.39 KB, patch
aki
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
Probably requires some mozharness work. bug 858797 may have some helpful work in it.
Attached patch add balrog mixinSplinter Review
This patch is step 1 - it just adds a balrog mixin that relies on the existing balrog scripts. It's mostly ripped off from Jordan's work.

I tested it with a quick small script: https://github.com/bhearsum/mozharness/blob/bd57297473259c0abde5e59e2766c7f95129a245/scripts/balrog_test.py

Which created these two blobs:
vhttps://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-20140202020202/data
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-latest/data
Attachment #8411254 - Flags: review?(aki)
Comment on attachment 8411254 [details] [diff] [review]
add balrog mixin

I wonder if we want this under work_dir instead of base_work_dir, since we tend to clobber build/ rather than build/ + all siblings of build/.

I don't think fatal_exit_code is needed if halt_on_failure is False.

I don't think either of these things bother me enough to delay landing.
Attachment #8411254 - Flags: review?(aki) → review+
Comment on attachment 8411254 [details] [diff] [review]
add balrog mixin

(In reply to Aki Sasaki [:aki] from comment #2)
> Comment on attachment 8411254 [details] [diff] [review]
> add balrog mixin
> 
> I wonder if we want this under work_dir instead of base_work_dir, since we
> tend to clobber build/ rather than build/ + all siblings of build/.

Good point, I think that makes sense.

> I don't think fatal_exit_code is needed if halt_on_failure is False.

I'll drop that, too.

Landed as https://hg.mozilla.org/build/mozharness/rev/c48a76ec161a with both points addressed.
Attachment #8411254 - Flags: checked-in+
Attached patch wip patch (obsolete) — Splinter Review
This obviously needs more work but I wanted to get some feedback and maybe help on where to source some of the information we need from.

I noticed that we currently get the mar url from releng-private-updates.py, but it seems like we should be able to get it from post_upload.py since we're uploading to FTP, which is more authoritative.

Any idea on how to read PRODUCT_MODEL from the build system would be appreciated too.
Attachment #8411985 - Flags: feedback?
Comment on attachment 8411985 [details] [diff] [review]
wip patch

Argh, so hard to get the right person!!!!
Attachment #8411985 - Flags: feedback? → feedback?(aki)
Comment on attachment 8411985 [details] [diff] [review]
wip patch

I don't know how to get PRODUCT_MODEL.  That may be in one of the dot files, maybe .config .
Sure, we can get the mar from post_upload.py, though in bug 979554 I actually want to explicitly define what filename/glob is what, rather than guess from a post_upload.py regex.  I think Jordan may have a post_upload regex already for his desktop work.
Comment on attachment 8411985 [details] [diff] [review]
wip patch

Note: we default to default_actions.

You could:

a) add submit-to-balrog to default_actions, and have a way to skip it if it's not an m-c/m-a flame nightly, or

b) add ["--add-action", "submit-to-balrog"] to the flame build command line args (buildbot-configs/mozilla/b2g_config.py), which will add that action to only flame builds on m-c or m-a (you'll need to avoid adding those options to, say, project branches).


We already have things like

        if not self.query_is_nightly():
            self.info("Not a nightly build. Skipping...")
            return
        if not self.config.get('update'):
            self.info("No updates. Skipping...")
            return

in upload_updates().  We could have some flag in flame's in-tree config that specifies enabling balrog.
Attachment #8411985 - Flags: feedback?(aki) → feedback+
Depends on: 1001428
(In reply to Aki Sasaki [:aki] from comment #7)
> Comment on attachment 8411985 [details] [diff] [review]
> wip patch
> 
> I don't know how to get PRODUCT_MODEL.  That may be in one of the dot files,
> maybe .config .

I poked around and AFAICT we can't get it. We already maintain a mapping for Desktop/Android, so it's not the end of the world to do it for b2g, too.
Attachment #8412748 - Flags: review?(aki)
Comment on attachment 8412748 [details] [diff] [review]
add-flame-mapping.diff

> update_platform_map = {
>     'android': ['Android_arm-eabi-gcc3'],
>     'android-armv6': ['Android_arm-eabi-gcc3-armv6'],
>     'android-x86': ['Android_x86-gcc3'],
>     'linux': ['Linux_x86-gcc3'],
>     'linux64': ['Linux_x86_64-gcc3'],
>     'macosx64': ['Darwin_x86-gcc3-u-i386-x86_64', 'Darwin_x86_64-gcc3-u-i386-x86_64'],
>     'win32': ['WINNT_x86-msvc'],
>     'win64': ['WINNT_x86_64-msvc'],
>+    'flame'; ['Flame'],

Maybe this is where your semicolon went ;-)
r=me with a colon.
Attachment #8412748 - Flags: review?(aki) → review+
Comment on attachment 8412748 [details] [diff] [review]
add-flame-mapping.diff

Landed, with the semicolon fixed.
Attachment #8412748 - Flags: checked-in+
Attached patch latest wip for mozharness (obsolete) — Splinter Review
Currently testing this on my dev master. Changes include:
* Create balrog configs for staging/prod.
** Need to decide if we want to use a separate "b2gbld" user for submission. I suspect we do, since we use different keys/accounts on ftp. Probably need to reorg for the config a bit for this.
* Add as an action for anything using releng-private-updates.py. Not sure if this is the right place, it's enough for testing though.
* Let balrog submitter figure out the update platform. I can't find a better way, and this is what we do for desktop/android.
Attachment #8411985 - Attachment is obsolete: true
Attached patch use balrog configs (obsolete) — Splinter Review
Again, may need to change this a bit if we use b2gbld. We might need separate configs per product.

Also need a patch to flip use_credentials_file=True for some/all device builds. Not sure the best way to do that yet - I've hardcoded it for my staging test.
Attached patch latest wip mozharness patch (obsolete) — Splinter Review
fixes a bunch of bugs. right now i'm hitting:
13:07:35     INFO -  "PUT /releases/B2G-mozilla-central-nightly-20140508155931/builds/Flame/en-US HTTP/1.1" 400 64
13:07:35     INFO -  Caught HTTPError: Couldn't update release: Release blob contains forbidden domain.
13:07:35     INFO -  Traceback (most recent call last):
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/scripts/updates/balrog-submitter.py", line 55, in <module>
13:07:35     INFO -      props['completeMarUrl'], **partialKwargs)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/cli.py", line 149, in run
13:07:35     INFO -      buildData=data, copyTo=copyTo, alias=alias)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 157, in update_build
13:07:35     INFO -      url_template_vars=url_template_vars)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 103, in request
13:07:35     INFO -      return self.do_request(url, data, method, url_template_vars)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/balrog/submitter/api.py", line 118, in do_request
13:07:35     INFO -      headers=headers)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/sessions.py", line 203, in request
13:07:35     INFO -      r.send(prefetch=prefetch)
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 585, in send
13:07:35     INFO -      self.response.raise_for_status()
13:07:35     INFO -    File "/builds/slave/b2g_m-cen_flame_ntly-000000000/build/tools/lib/python/vendor/requests-0.10.8/requests/models.py", line 810, in raise_for_status
13:07:35     INFO -      raise http_error
13:07:35     INFO -  requests.exceptions.HTTPError: 400 Client Error


Probably need to start pulling the update URL from the log or something, so we actually get the right one (instead of the update.boot2gecko.org one).
Attachment #8412876 - Attachment is obsolete: true
Comment on attachment 8412882 [details] [diff] [review]
use balrog configs

This, in combination with the stuff I'm about to attach seems to work in staging. Build is here:
http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/b2g_mozilla-central_flame_nightly/builds/8/steps/run_script/logs/stdio

Balrog data is here:
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-20140509122946/data
https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-latest/data

(Ignore the "flame" platform in the second one - that's leftover from some earlier tests.)
Attachment #8412882 - Flags: review?(aki)
This also passes it to a bunch of other builds that don't need it. Meh?
Attachment #8420205 - Flags: review?(aki)
Attached patch fully tested mozharness patch (obsolete) — Splinter Review
Here's a summary of the changes:
* Add "submit-to-balrog" action for anything using releng-private-updates.py. I've only tested flame, the others should just work though, right? :)
* Add mar_base_url for the same set of things. I was originally going to change base_url, but that'll break update.boot2gecko.org updates, and I'd like to land this patch before we're ready to switchover, so we can dogfood for a day or two.
* Add balrog configs for staging/prod. Buildbot-configs handles using the right one.
* Fix some silly bugs in BalrogMixin
* Change query_marfile_path to detect the MAR filename rather than hardcode it. This is needed because I'll be changing the filename to include device name in bug 1000207.
* Implement submit_to_balrog step. Not much to say about this one, it just gathers data and gives it BalrogMixin.
Attachment #8420206 - Flags: review?(aki)
(In reply to Ben Hearsum [:bhearsum] from comment #15)
> Comment on attachment 8412882 [details] [diff] [review]
> use balrog configs
> 
> This, in combination with the stuff I'm about to attach seems to work in
> staging. Build is here:
> http://dev-master1.srv.releng.scl3.mozilla.com:8018/builders/b2g_mozilla-
> central_flame_nightly/builds/8/steps/run_script/logs/stdio
> 
> Balrog data is here:
> https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-
> 20140509122946/data
> https://aus4-admin-dev.allizom.org/releases/B2G-mozilla-central-nightly-
> latest/data
> 
> (Ignore the "flame" platform in the second one - that's leftover from some
> earlier tests.)

Actually, I just realized that we're using PRODUCT_DEVICE in the update URL (instead of PRODUCT_MODEL). This means I'll need to update platforms.py to use that, and we'll end up submitting with a lowercase "flame".
Attachment #8420213 - Flags: review?(aki)
Comment on attachment 8412882 [details] [diff] [review]
use balrog configs

More context in the patch would be nice, but looks right.
Attachment #8412882 - Flags: review?(aki) → review+
Comment on attachment 8420205 [details] [diff] [review]
pass credentials file to device build factories

Probably harmless, since we're already somewhat paranoid about device builds.  If we care, we could add another platform config key that enables/disables this.
Attachment #8420205 - Flags: review?(aki) → review+
Attachment #8420213 - Flags: review?(aki) → review+
Comment on attachment 8420206 [details] [diff] [review]
fully tested mozharness patch

>     def query_marfile_path(self):
>         if self.config.get("update_type", "ota") == "fota":
>             output_dir = self.query_device_outputdir()
>             return "%s/fota-update.mar" % output_dir
>         else:
>-            return "%s/dist/b2g-update/b2g-gecko-update.mar" % self.objdir
>+            mardir = "%s/dist/b2g-update" % self.objdir
>+            files = os.listdir(mardir)
>+            if len(files) != 1:
>+                self.fatal("Found none or too many marfiles, don't know what to do.")

Do you think this is an infrastructure error or a build failure?
If infrastructure, could you self.fatal("Found none or too many...", exit_code=3) so that the build turns purple?
1 red 2 orange 3 purple 4 retry iirc.

>+    def submit_to_balrog(self):
>+        if not self.query_is_nightly():
>+            self.info("Not a nightly build, skipping balrog submission.")
>+            return

Can you put an

        if not self.config.get("balrog_api_root"):
            self.info("balrog_api_root not set; skipping balrog submission.")
            return

Otherwise, aiui, the other device builds that use releng-private-updates.py will start dying.

Other than that, this patch looks good.  Thanks!
Attachment #8420206 - Flags: review?(aki) → review+
(In reply to Aki Sasaki [:aki] from comment #22)
> Comment on attachment 8420206 [details] [diff] [review]
> fully tested mozharness patch
> 
> >     def query_marfile_path(self):
> >         if self.config.get("update_type", "ota") == "fota":
> >             output_dir = self.query_device_outputdir()
> >             return "%s/fota-update.mar" % output_dir
> >         else:
> >-            return "%s/dist/b2g-update/b2g-gecko-update.mar" % self.objdir
> >+            mardir = "%s/dist/b2g-update" % self.objdir
> >+            files = os.listdir(mardir)
> >+            if len(files) != 1:
> >+                self.fatal("Found none or too many marfiles, don't know what to do.")
> 
> Do you think this is an infrastructure error or a build failure?
> If infrastructure, could you self.fatal("Found none or too many...",
> exit_code=3) so that the build turns purple?
> 1 red 2 orange 3 purple 4 retry iirc.

I'm not really sure. None of the automation creates mar files, so RETRY is not going to have any positive effect. If/When the build system provides us a way to reference a specific MAR (a la Gecko's package-name.mk variables: https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/package-name.mk#52) we can do better here.

Do you have a preference for red vs purple?

> >+    def submit_to_balrog(self):
> >+        if not self.query_is_nightly():
> >+            self.info("Not a nightly build, skipping balrog submission.")
> >+            return
> 
> Can you put an
> 
>         if not self.config.get("balrog_api_root"):
>             self.info("balrog_api_root not set; skipping balrog submission.")
>             return

Will do.

> Otherwise, aiui, the other device builds that use releng-private-updates.py
> will start dying.

I think all of the device builds will have the right things set to successfully submit to Balrog (I'm kindof counting on that, so that we can test them out before we change the in-app update URL), but that's probably not something we should assume going forward.
Attachment #8420213 - Flags: checked-in+
Attachment #8420205 - Flags: checked-in+
(In reply to Ben Hearsum [:bhearsum] from comment #23)
> I think all of the device builds will have the right things set to
> successfully submit to Balrog (I'm kindof counting on that, so that we can
> test them out before we change the in-app update URL), but that's probably
> not something we should assume going forward.

Oh, I just realized that I'm not setting the Balrog configs for other devices in buildbot-configs....I'm going to update that patch.
Attachment #8412882 - Attachment is obsolete: true
Attachment #8419634 - Attachment is obsolete: true
I only added this for devices I found in /data/update-channels on update.boot2gecko.org.
Attachment #8421005 - Flags: review?(aki)
Merged into production and live.
(In reply to Ben Hearsum [:bhearsum] from comment #23)
> > Do you think this is an infrastructure error or a build failure?
> > If infrastructure, could you self.fatal("Found none or too many...",
> > exit_code=3) so that the build turns purple?
> > 1 red 2 orange 3 purple 4 retry iirc.
> 
> I'm not really sure. None of the automation creates mar files, so RETRY is
> not going to have any positive effect. If/When the build system provides us
> a way to reference a specific MAR (a la Gecko's package-name.mk variables:
> https://mxr.mozilla.org/mozilla-central/source/toolkit/mozapps/installer/
> package-name.mk#52) we can do better here.
> 
> Do you have a preference for red vs purple?

I suppose it's up in the air since the build system creates mars.
A depend build with a renamed mar might get 2 and might need a clobber, but otherwise it wouldn't be an infra problem.  Up to you.
Comment on attachment 8421005 [details] [diff] [review]
submit to balrog for other devices

>     'hamachi': {
>         'mozharness_config': {
>             'script_name': 'scripts/b2g_build.py',
>             # b2g_build.py will checkout gecko from hg and look up a tooltool manifest given by the
>             # --target name below
>             'extra_args': ['--target', 'hamachi', '--config', 'b2g/releng-fota-updates.py',
>                            '--gaia-languages-file', 'locales/languages_dev.json',
>-                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales'],
>+                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales',
>+                           '--config', GLOBAL_VARS['mozharness_configs']['balrog']],
>             'reboot_command': ['bash', '-c', 'sudo reboot; sleep 600'],
>         },
>         'stage_product': 'b2g',
>         'product_name': 'b2g',
>         'base_name': builder_prefix + '_%(branch)s_%(platform)s',
>         'slaves': SLAVES['mock'],
>         'enable_periodic': True,
>         'enable_dep': False,
>@@ -964,17 +966,18 @@ PLATFORM_VARS = {
>     },
>     'tarako': {
>         'mozharness_config': {
>             'script_name': 'scripts/b2g_build.py',
>             # b2g_build.py will checkout gecko from hg and look up a tooltool manifest given by the
>             # --target name below
>             'extra_args': ['--target', 'tarako', '--config', 'b2g/releng-fota-updates.py',
>                            '--gaia-languages-file', 'locales/languages_dev.json',
>-                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales'],
>+                           '--gecko-languages-file', 'gecko/b2g/locales/all-locales',
>+                           '--config', GLOBAL_VARS['mozharness_configs']['balrog']],
>             'reboot_command': ['bash', '-c', 'sudo reboot; sleep 600'],
>         },
>         'stage_product': 'b2g',
>         'product_name': 'b2g',
>         'base_name': builder_prefix + '_%(branch)s_%(platform)s',
>         'slaves': SLAVES['mock'],
>         'enable_periodic': False,
>         'enable_dep': True,

IIRC you only added balrog stuff to releng-private-updates.py, so tarako and hamachi (using releng-fota-updates.py) won't pick that up.
Attachment #8421005 - Flags: review?(aki) → review+
OK, I think this addresses everything. The only changes since the last patch are:
* Set exit code to 1 if query_marfile fails.
* Skip Balrog submission if balrog_api_root isn't set
* Add Balrog stuff to releng-fota-updates.py, so they can submit data too.
Attachment #8420206 - Attachment is obsolete: true
Attachment #8421095 - Flags: review?(aki)
Comment on attachment 8421095 [details] [diff] [review]
address final review comments

>+            mardir = "%s/dist/b2g-update" % self.objdir
>+            files = os.listdir(mardir)
>+            if len(files) != 1:
>+                self.fatal("Found none or too many marfiles, don't know what to do.", 1)

Please use exit_code=1 rather than a bare 1.
The kwargs should be order-independent; we've already hit an issue where we prepended a kwarg and broke a method call that assumed kwarg order.
Attachment #8421095 - Flags: review?(aki) → review+
Attachment #8421005 - Flags: checked-in+
Depends on: 1007180
Comment on attachment 8421005 [details] [diff] [review]
submit to balrog for other devices

I landed this (and the mozharness patch) briefly, and then I realized that we need bug 1007180 before we can submit to Balrog in production. I backed them out to make sure they don't accidentally get merged before that's done.
Attachment #8421005 - Flags: checked-in+
Comment on attachment 8421095 [details] [diff] [review]
address final review comments

B2G account changes were made - I landed this. I'm triggering a Flame nightly to verify things.
Attachment #8421095 - Flags: checked-in+
Attachment #8421005 - Flags: checked-in+
(In reply to Ben Hearsum [:bhearsum] from comment #32)
> Comment on attachment 8421095 [details] [diff] [review]
> address final review comments
> 
> B2G account changes were made - I landed this. I'm triggering a Flame
> nightly to verify things.

Oops, no I'm not! This hasn't merged to production.
Live in production.
Depends on: 1009657
The submitter ran for a nightly I triggered, but it failed (thankfully, not in a way that burns the job). Turns out I forgot to add b2gbld passwords to BuildSlaves.py. Filed bug 1009657 for this, and it should be fixed before the next scheduled nightlies run.
Still not working. Looks like we're trying to submit as ffxbld for some reason:
07:21:32     INFO -  Caught HTTPError: ffxbld is not allowed to access /releases/:name/builds/:platform/:locale by PUT
Oops
Attachment #8422440 - Flags: review?(aki)
Attachment #8422440 - Flags: review?(aki) → review+
Attachment #8422440 - Flags: checked-in+
balrog_username override is in production
Updated the wrong thing, needs a Mozharness config change =\
Comment on attachment 8422440 [details] [diff] [review]
override balrog_username for b2g stuff

I backed this out, because it doesn't do anything.
Attachment #8422440 - Attachment is obsolete: true
Attachment #8422440 - Flags: checked-in+ → checked-in-
Attachment #8423370 - Flags: review?(aki) → review+
Comment on attachment 8423370 [details] [diff] [review]
per-product usernames

I translanted right to production so I can catch the next round of nightlies.
Attachment #8423370 - Flags: checked-in+
Things are submitting now. There will be some follow-ups, but I'll be filing new bugs for them.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.