1230657 - Make Mozharness check for testPackagesUrl and packageUrl properties before trying to find TC artifacts

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Armen [:armenzg]]

Alice and I recently discovered that in my work to allow mixing Buildbot jobs, Buildbot bridge jobs and TaskCluster tasks, I completely forgot testing a specific test case scenario:
* Trigger a BBB task based on a Buildbot completed job

In this push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=075a055fff49
https://tools.taskcluster.net/task-graph-inspector/#MiXLYEnVQ2Cyj2cYqOugFg/A_JmY0cFTbahirueJNOoow/

We created two tasks with these properties:
* testUrl
* packageUrl

Unfortunately, passing this as properties does not work the same as when we POST to the trigger abitrary job API with a payload with the two files.

adusca came up for a hack for now, however, it would be great to fix this properly.

In this case, there are two ways that I can fix this with:
A) the build jobs set an 'upload_to_task_id' property in mshal's code [1]
  * We can then use that in Mozci as a way to find where the builds have been uploaded to
B) we teach Mozharness to pay attention to 'testUrl' and 'packageUrl' before trying anything else [2]

For my own reference, the last two changes I made in this area are:
* First pass: https://hg.mozilla.org/integration/mozilla-inbound/rev/7a8fcc716362
* Second pass: https://hg.mozilla.org/mozilla-central/rev/f0e6e0ae362d

[1] https://hg.mozilla.org/mozilla-central/file/f0e6e0ae362d/testing/mozharness/mozharness/mozilla/taskcluster_helper.py#l36
[2] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/testbase.py#312

Comment 1

[Armen [:armenzg]]

Attachment: Support set properties to find installer and package url

That was easy! :)

Comment 2

[Jordan Lund (:jlund)]

Comment on attachment 8696025 [details] [diff] [review]
Support set properties to find installer and package url

Review of attachment 8696025 [details] [diff] [review]:
-----------------------------------------------------------------

wondering if it be worth in the future having a helper class that can do things like: is_buildbot_bridge_job(), is_buildbot_job(), is_tc_job(), and more complicated mixes of those. I guess we will see how many more times we require making the distinction across our mh scripts..

patch lgtm if it works for you :)

it might be a bit fickle if 'taskId' and 'testPackagesUrl' both exist in the tc->bbb->bb case (the lines you are adding) as someone could trip if they extend the conditions

Comment 3

[Armen [:armenzg]]

Anything scheduled through TaskCluster sets a taskId for the task.
For the last two cases we also have taskId but the two properties have more relevance, hence, the order in which I evaluate it.

Thanks!

Comment 4

[Armen [:armenzg]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/627b01cda561566049d1497f47c8d789948496aa
Bug 1230657 - Make Mozharness check for testPackagesUrl and packageUrl properties before trying to find TC artifacts. r=jlund

Comment 5

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/627b01cda561