1471227 - adjust taskcluster backfill with path to use test-verify instead of original job

Status: RESOLVED FIXED

(Testing :: General, enhancement, P1)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

thanks to the work in bug 1465117, it is easier to add a single test name via a custom action in treeherder and retrigger a job on X revisions to run that test.

The intention was to run this in test-verify and I had not thought it through and asked for the wrong thing.  This bug is to do what I originally had intended.

What needs to happen:
1) on retriggering the task when we enter a path in testPaths, we will append it to the env via MOZHARNESS_TEST_PATHS (as it currently does)
2) edit the command to run a generic test in --verify mode
3) adjust the label to be test-verify
4) adjust some minor args for index:rank, and chunks to keep taskcluster organized well
5) adjust the values for treeherder to have the right group/label/code.

there are a lot of changes, I imagine I will get some things wrong here in the first pass.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

an example try run (in progress):
https://treeherder.mozilla.org/#/jobs?repo=try&revision=e189f4997d142c53b7cffd4179e359ea9d6207ed

more interesting there is the code:
https://hg.mozilla.org/try/rev/6ce4c36ca717c42f0d894e38c447fac55b144440

ideally a few tweaks and tests and we can get this landed.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

there seems to be a lot to consider when hacking on a task.  Things like different taskcluster workers treating commands differently, and taskcluster inserting build taskID dynamically into certain parts of the command to execute (installer-url for example).

I did get this working:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c6f3eef1cab55d21609819972ffe2232d27b320

It is hacky, and I suspect many use cases will not work.  When I have tested it on a larger set of tests (plain, gpu, wpt) and configs (linux, osx, windows, android), I will solicit feedback.

meanwhile, if there are any drive by comments on my changes so far, they are welcome:
https://hg.mozilla.org/try/rev/32493a0a36a97ad371d837fa3667d31e96bc365b

I will sneak in a small change to this to fix the --geckoProfile as well- the original patch added the string as an env variable, and it needs to be added to the command line.  I will consider using the env var instead of hacking the cli.

Comment 3

[Phabricator Automation]

Attachment: Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

this is ugly and sloppy, not sure of a cleaner way to do this, but open to suggestions.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

on my recent push, I tested a good bit on try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f02781710d5a049d07d9557405490b28080fd098&filter-tier=1&filter-tier=2&filter-tier=3

for the different tests on all 5 operating systems, I tested:
reftest: layout/reftests/writing-mode/tables/vertical-table-1a.html
xpcshell: chrome/test/unit/test_bug292789.js
wpt: testing/web-platform/tests/WebCryptoAPI/generateKey/failures_AES-CBC.https.any.js
webgl: dom/canvas/test/crash/test_1251091-1.html
mochitest: dom/indexedDB/test/test_count.html
jsreftest: js/src/tests/non262/Array/regress-101964.js
* no test run on android, not sure why
mochitest-media: dom/media/test/test_VideoPlaybackQuality_disabled.html
* not run on android-hw (this is new bitbar, happy to investigate later)

you can see a couple scenarios here are not met, I believe we can tackle those in follow up issues.  I did shift slightly to a model where we define gpu_required by inspecting the test paths instead of a hybrid of task name and test path.  Also I need to test all of this in >1 test provided, although I see that as a different use case altogether.   Open to further thoughts.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Comment 8

[Phabricator Automation]

Comment on attachment 8989080 [details]
Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Geoff Brown [:gbrown] has approved the revision.

https://phabricator.services.mozilla.com/D1908

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8989841 [details]
Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

carry over from previous patch

Comment 11

[Phabricator Automation]

Comment on attachment 8989841 [details]
Bug 1471227 - adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Geoff Brown [:gbrown] has approved the revision.
Joel Maher ( :jmaher ) (UTC-4) has been removed from the revision.

https://phabricator.services.mozilla.com/D1961

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/22764fd6ae90
adjust taskcluster backfill with path to use test-verify instead of original job. r=gbrown

Comment 13

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/22764fd6ae90

Comment 14

[Julien Cristau [:jcristau] (back April 22)]

https://hg.mozilla.org/releases/mozilla-release/rev/f709d5da4e80

Comment 15

[Julien Cristau [:jcristau] (back April 22)]

https://hg.mozilla.org/releases/mozilla-esr60/rev/ed08947f3922