1209064 - [TC linux64] builds don't set the 'mozharness' location

Status: RESOLVED DUPLICATE of bug 1171033

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

Came across this when working on bug 1171033. The test jobs fail early because they can't find mozharness:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=408e060ac475

It looks like bug 1188330 recently added a 'download mozharness' step that needs to get called. Digging into it a bit..

1. The tester image downloads mozharness if MOZHARNESS_URL is set:
https://dxr.mozilla.org/mozilla-central/source/testing/docker/tester/bin/entrypoint#14

2. The taskcluster-graph mach command (indirectly) sets this env if there is a 'mozharness' location:
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/mach_commands.py?offset=600#405

3. The 'mozharness' location is defined in b2g_base.yml, but not in the desktop configs:
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/builds/b2g_base.yml#9


I sort of just figured this out while writing up this bug description.. Will do a quick try push and post patch shortly if it works.

Comment 1

[Andrew Halberstadt [:ahal]]

So that triggered the download mozharness step, but there is no 'mozharness.zip' artifact uploaded anywhere:
https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/ZezaGHH0TM69vvaonYMkjQ/0/public/logs/live_backing.log

I see the b2g jobs do this in various scripts like "testing/taskcluster/scripts/builder/build-mulet.sh", but the build-linux.sh script doesn't handle artifacts there.

Dustin, do you know where the linux64 builds handle artifact uploading? I can't seem to find it with dxr.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Looking at the artifacts from a recent job,
  https://tools.taskcluster.net/task-inspector/#ZZ0dwftrQTqK92MnXO2fhg/0
I don't see any mozharness.zip.  Those artifacts are from the `make upload` step.

So one option is to script something in the build process that will zip up `testing/mozharness` from the gecko tree and drop it in artifacts/.  I'd be cool with that happening in build-linux.sh.

However, every build is then going to generate the same mozharness.zip, which ends up eating a lot of storage.  So I think the *right* way to do this is for the decision task to write out mozharness.zip as an artifact, and point all of the downstream test jobs at that artifact.  It could just as well create a subtask to do so, but I don't see the point of making another Gecko checkout just for this.  Either way, this requires a lot more action from the decision task, so it's not a quick fix.

Do you want to try the build-linux.sh approach in this bug, and I'll file a new bug for the other suggestion?

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> Do you want to try the build-linux.sh approach in this bug, and I'll file a
> new bug for the other suggestion?

Sure, sounds good to me.

How about in this bug I'll land the patch that adds the 'mozharness' location (as stated in the title). The tests will still be orange, but a little further along at least.

Then I can hack up the build-linux.sh solution locally and push it to try along with the other patches in the series (this doesn't really need to land until we schedule tests on branches other than try.. hooray in-tree everything!).

In the meantime, maybe the proper solution will be fixed and we never even have to land the build-linux.sh changes in the first place.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: MozReview Request: Bug 1209064 - [taskcluster] Add 'mozharness' location to desktop firefox builds, r=dustin

Bug 1209064 - [taskcluster] Add 'mozharness' location to desktop firefox builds, r=dustin

Comment 5

[Andrew Halberstadt [:ahal]]

I guess I could also just land the above patch locally on try for now too.. Doesn't really matter to me.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I'd prefer that, since it's an incomplete (but correct!) patch as-is.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8666814 [details]
MozReview Request: Bug 1209064 - [taskcluster] Add 'mozharness' location to desktop firefox builds, r=dustin

>https://reviewboard.mozilla.org/r/20603/

Comment 8

[Andrew Halberstadt [:ahal]]

Looks like zipping mozharness already happens as part of the test package step:
https://dxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#433

Probably just a matter of copying it from the objdir to $HOME/artifacts.

Comment 9

[Andrew Halberstadt [:ahal]]

Eh, the mozharness.zip packaging probably doesn't belong in testsuite-targets.mk anyway as it isn't part of the test package. It should happen as part of the build by the installer along with all the other build artifacts. I'll change it to do that, then we get the mozharness.zip artifact for free.

Comment 10

[Andrew Halberstadt [:ahal]]

I ended up doing the mozharness packaging in a different bug. This patch is now so small, I'm just going to roll it up with the main one.