Closed Bug 1185643 Opened 4 years ago Closed 4 years ago

switch over taskcluster jobs to use gecko copy of mozharness

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(firefox43 fixed)

RESOLVED FIXED
Tracking Status
firefox43 --- fixed

People

(Reporter: jlund, Assigned: dustin)

References

Details

(Whiteboard: [build])

Attachments

(2 files)

thanks to progress in https://bugzil.la/1131856, a copy of mozharness is now in the gecko tree and is being used for all buildbot based jobs.

We can now switch over TC to also use the gecko copy.

You have a couple options:

1) if you have a copy of the tree and the rev you are build/testing on the slave:

# just call the script you need from the tree:
`python $gecko_tree/testing/mozharness/scripts/$script # with args` from the same cwd you have been calling mozharness from before. The work_dir (`pwd`/build) can stay the same

2) if you don't have a copy (say maybe for tests):

# you can use Archiver[1] and the client[2] to download a tarball of mozharness based on a specific gecko repo + rev:

`python $BUD/tools_jlund/buildfarm/utils/archiver_client.py mozharness --repo integration/mozilla-inbound --rev 0a1f292ba8cc --destination $CWD/mozharness --debug; python mozharness/scripts/$script # + args`

[1] https://api.pub.build.mozilla.org/docs/usage/archiver/
[2] https://hg.mozilla.org/build/tools/file/default/buildfarm/utils/archiver_client.py
Whiteboard: [build]
Assignee: nobody → dustin
Depends on: 1187706
Wander: testing/taskcluster/tasks/build.yml has references to MOZHARNESS_* which I won't remove yet, on the assumption that b2g builds are still depending on them (bug 1187706)
Bug 1185643: use in-tree mozharness; r?wander r?mrrrgn

This required moving the custom mozharness configs out of the builder image
(which puts them in ~, where mozharness no longer looks) and moves them into
mozharness itself.
Attachment #8639464 - Flags: review?(winter2718)
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

Bug 1185643: use in-tree mozharness; r?wander r?mrrrgn

This required moving the custom mozharness configs out of the builder image
(which puts them in ~, where mozharness no longer looks) and moves them into
mozharness itself.
Attachment #8639464 - Flags: review?(wcosta)
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

Bug 1185643: use in-tree mozharness; r?wander r?mrrrgn

This required moving the custom mozharness configs out of the builder image
(which puts them in ~, where mozharness no longer looks) and moves them into
mozharness itself.
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

The only issue I have with this, is that $WORKSPACE/build/src/testing is hard coded in before $MOZHARNESS_SCRIPT so if someone wants to add a custom path they'll need to add "../../../"

Otherwise, heck yes. :)
Attachment #8639464 - Flags: review?(winter2718) → review+
On second thought, we'll probably have more users wanting to just change scripts from within the tree than we will people who want to test their own MH checkout. +1
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

https://reviewboard.mozilla.org/r/14203/#review12823

Ship It!
Attachment #8639464 - Flags: review?(wcosta) → review+
Depends on: 1188330
I'm game to change $MOZHARNESS_SCRIPT to be relative to ${topsrcdir}, but let's take one step at a time.  Specifically, it'd be great to have gecko, b2g, etc. all using the same approach for that.
seems this broke a lot of stuff like b2g and mulet like https://treeherder.mozilla.org/logviewer.html#?job_id=12197062&repo=mozilla-inbound so backed this out and hopefully this fix the issue and we are able to re-open mozilla-inbound
Ah, the failures I was seeing in mach are from bug 1186003, the landing of which caused a permafail for linux builds in TC.  I'm not sure why the manifestation was different than in bug 1187047, but hopefully the fix to both is the same.
Depends on: 1187047
Duplicate of this bug: 1188330
Depends on: 1189892
This is blocking me for getting Mulet Gu. When can I hope that this is fixed?
Blocks: 1188337
Duplicate of this bug: 1197158
:ted -- we might be able to land this patch more-or-less as-is, even if the desktop builds still don't work (12 days ago, there were 3 or so things causing desktop builds to fail, this among them -- probably more now).  I don't recall why the backout occurred.
Ah, yes -- https://dxr.mozilla.org/mozilla-central/source/testing/docker/tester/bin/entrypoint starts with `test $MOZHARNESS_REPOSITORY`, and due to the mach_commands.py changes in e3c15b9f89b3, that value is empty.
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

Bug 1185643: use in-tree mozharness for builds; r?wander

This required moving the custom mozharness configs out of the builder image
(which puts them in ~, where mozharness no longer looks) and moving them into
mozharness itself.
Attachment #8639464 - Attachment description: MozReview Request: Bug 1185643: use in-tree mozharness; r?wander r?mrrrgn → MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander
Attachment #8639464 - Flags: review+ → review?(winter2718)
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

https://reviewboard.mozilla.org/r/14203/#review14995

::: testing/taskcluster/mach_commands.py:288
(Diff revision 2)
>          mozharness = load_mozharness_info()

Can't this be removed?
No longer blocks: q3-bb-tc-migration
No longer depends on: 1187047, 1189892
Comment on attachment 8639464 [details]
MozReview Request: Bug 1185643: use in-tree mozharness for builds; r?wander

https://reviewboard.mozilla.org/r/14203/#review14997

Ship It!
Attachment #8639464 - Flags: review+
It can't be removed yet, because testing/taskcluster/tester/bin/entrypoint still refers to MOZHARNESS_RE*
Attachment #8639464 - Flags: review?(winter2718)
Attached patch bump.patchSplinter Review
Attachment #8651094 - Flags: review?(wcosta)
Attachment #8651094 - Flags: review?(wcosta) → review+
https://hg.mozilla.org/mozilla-central/rev/be6ff275a904
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Depends on: 1205934
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.