Turn on blobber upload for Gaia Unit Tests

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: chmanchester, Assigned: chmanchester)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Assignee)

Description

3 years ago
We'll need this to upload the raw structured logs from these runs.
Looks like BlobUploadMixin will need to be added to the GaiaUnitTest definition in mozharness/scripts/gaia_unit.py, plus associated plumbing. Plus this in buildbot-configs:

diff --git a/mozilla-tests/b2g_config.py b/mozilla-tests/b2g_config.py
--- a/mozilla-tests/b2g_config.py
+++ b/mozilla-tests/b2g_config.py
@@ -661,8 +661,9 @@ GAIA_UNITTESTS = [(
     'gaia-unit', {
         'suite': 'gaia-unit',
         'use_mozharness': True,
         'script_path': 'scripts/gaia_unit.py',
+        'blob_upload': True,
     },
 )]
(Assignee)

Comment 2

3 years ago
Created attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

This is plausible, but requires the buildbot config change above. What's the procedure for modifying those?
(Assignee)

Updated

3 years ago
Assignee: nobody → cmanchester
Status: NEW → ASSIGNED
(Assignee)

Comment 3

3 years ago
Also, perhaps you could review this patch or help me find someone who could? Thank you!
Flags: needinfo?(nthomas)
buildbot-configs is at http://hg.mozilla.org/build/buildbot-configs/; you can submit the patch and after it lands it won't go into production until the next buildbot reconfig (typically a couple of times a week).
(Assignee)

Comment 5

3 years ago
Created attachment 8467239 [details] [diff] [review]
Turn on blobber uploads for gaia unit tests.

Buildbot config patch.
Attachment #8467239 - Flags: review?(nthomas)
Chris, I'm not overly familiar with mozharness, so someone like jlund would be a better person to review. 

Has the mozharness patch had any local testing ? It may be helpful for someone in RelEng to run it through our staging systems. Are any other changes needed to enable structured logging ?
Flags: needinfo?(nthomas)
Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

Review of attachment 8467213 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, let's get jlund's review
Attachment #8467213 - Flags: review?(jlund)
Attachment #8467213 - Flags: feedback+
Comment on attachment 8467239 [details] [diff] [review]
Turn on blobber uploads for gaia unit tests.

I'll pass this over too, so jlund see's the whole picture.
Attachment #8467239 - Flags: review?(nthomas) → review?(jlund)

Comment 9

3 years ago
Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

Review of attachment 8467213 [details] [diff] [review]:
-----------------------------------------------------------------

neat. this will be nice to have. r+ once we try this on ash (see below)

it lgtm. Could we land this on ash-mozharness and trigger a few related jobs? Ash is a project branch that uses its own mozharness repo: ash-mozharness. This is the testing playground. More details here: https://wiki.mozilla.org/ReleaseEngineering/SpecialBranches#Ash

What I would be hoping for is hitting this for gaia-unit jobs http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#95 since the required https://bug1043728.bugzilla.mozilla.org/attachment.cgi?id=8467239 isn't live yet in production.

Then, to complete the test, we could land a patch in ash-mozharness that would force blob_upload_branch so we could see what happens when https://bug1043728.bugzilla.mozilla.org/attachment.cgi?id=8467239 does land. By force I mean adding something like this to gaia_unit_production_config.py:

 "default_blob_upload_servers": ["https://blobupload.elasticbeanstalk.com",],
 "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),
+"blob_upload_branch": "ash",  # XXX hack, simulating --blob-upload-branch {branch} for gaia-unit on ash.

Once all looks well, we can land on our main mozharness repo on default. Does this make sense?

::: configs/b2g/gaia_unit_production_config.py
@@ +42,5 @@
>      ],
> +    "default_blob_upload_servers": [
> +        "https://blobupload.elasticbeanstalk.com",
> +    ],
> +    "blob_uploader_auth_file": os.path.join(os.getcwd(), "oauth.txt"),

oauth.txt is created from buildbot when use_credentials_file is True against the ScriptFactory that runs this mozharness script.

For test jobs we pretty much default this to True plus I double checked it for this specific script and the tl;dr here is I think this is fine to do

::: scripts/gaia_unit.py
@@ +51,5 @@
> +        upload_dir = self.query_abs_dirs()['abs_blob_upload_dir']
> +        if not os.path.isdir(upload_dir):
> +            self.mkdir_p(upload_dir)
> +
> +        env = self.query_env()

I think this is fine. I don't see self.config['env'] in any config file used by this script but as this is a once off and we only care about os.environment + this MOZ_UPLOAD_DIR, it might be better to use partial_env like in this example:

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/testing/talos.py#629

@@ +52,5 @@
> +        if not os.path.isdir(upload_dir):
> +            self.mkdir_p(upload_dir)
> +
> +        env = self.query_env()
> +        env['MOZ_UPLOAD_DIR'] = upload_dir

For other blobber integrated scripts, it looks like we sometimes use a variation of pointing MOZ_UPLOAD_DIR and MINIDUMP_SAVE_PATH to abs_blob_upload_dir.

I am not sure how for this script (gaia_unit) they are swallowed by gaia_unit_test/main.py but I'll assume you know what is needed here and why.
Attachment #8467213 - Flags: review?(jlund) → review+
Comment on attachment 8467239 [details] [diff] [review]
Turn on blobber uploads for gaia unit tests.

Review of attachment 8467239 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. let's hold off landing this until the mozharness patch has been tested on ash.
Attachment #8467239 - Flags: review?(jlund) → review+
(Assignee)

Comment 11

3 years ago
Thanks jlund, sounds like a plan. Here's the push to ash-mozharness:

http://hg.mozilla.org/build/ash-mozharness/rev/5813dc999aae

I retriggered the Gu jobs on this ash push:

https://tbpl.mozilla.org/?tree=Ash&rev=2c97f5469703

I'll check back in the morning.
(Assignee)

Comment 12

3 years ago
We hit the expected "Blob upload gear skipped. Missing cmdline options." in this log:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45302979&full=1&branch=ash

I pushed the "blob_upload_branch" -> "ash" change to ash-mozharness and retriggered again on that same push.
(Assignee)

Comment 13

3 years ago
The log here:

https://tbpl.mozilla.org/php/getParsedLog.php?id=45337804&tree=Ash&full=1

shows this code path is now hit:

http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#43
(In reply to Chris Manchester [:chmanchester] from comment #13)
> The log here:
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=45337804&tree=Ash&full=1
> 
> shows this code path is now hit:
> 
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/
> blob_upload.py#43

OK, so at least this does not seem to break anything. :)

It looks like nothing was put in abs_blob_upload_dir as we returned early: http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/blob_upload.py#59

Is this expected? Does that mean UPLOAD_DIR did nothing? I'm not too familiar with blobber so that is why I ask.
I don't see any code that actually puts a file in that directory.  For instance, the Marionette script does this:

http://hg.mozilla.org/build/mozharness/file/03a994d2ec21/scripts/marionette.py#l415
(Assignee)

Comment 16

3 years ago
As I have it the harness is putting the file in that directory, so without my changes this doesn't result in any uploads. I'll push to ash pointing to my gaia branch to test this out.
(Assignee)

Comment 17

3 years ago
Ok, this resulted in an upload:

https://tbpl.mozilla.org/?tree=Ash&rev=233c0150fc51
https://tbpl.mozilla.org/php/getParsedLog.php?id=45371659&tree=Ash&full=1

Is this good to go? The mozharness changes should land ahead of the config change, right?
Flags: needinfo?(jlund)
yes lgtm. tx for doing the rounds of testing before deploying.

we can land both at the same time at their respective default branch. default buildbot is used by nothing. mozharness default is used by Cedar and Cypress (we should watch those branches to ensure everything is still green). mozharness default merges to production usually when we merge buildbot default -> production but even if mozharness merges to prod long before, this shouldn't turn any jobs orange or red right? I'm basing that off your comment 11-13 just showing logs that blobber step didn't happen but the overall job was still a success.

buildbot default merge to production should never happen without a mozharness merge
Flags: needinfo?(jlund)
(Assignee)

Comment 19

3 years ago
Thanks. Right, I wouldn't expect any issue from mozharness hitting production first. I'll back this out if anything looks weird on Cypress tonight.

mozharness:
https://hg.mozilla.org/build/mozharness/rev/bba9efc62b21

buildbot:
https://hg.mozilla.org/build/buildbot-configs/rev/900f21cfaf08
Comment on attachment 8467213 [details] [diff] [review]
Add blob upload ability to gaia unit test jobs.

in production
Attachment #8467213 - Flags: checked-in+
(Assignee)

Updated

3 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.