1357551 - Taskcluster enhancement to trigger a job when a test file is modified

Status: RESOLVED WONTFIX

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

"when: file_changed:" is great, but probably not quite sufficient for bug 1357513. I want a similar mechanism that triggers a job when a test file is changed.  "when: file_changed: @TEST@", or "when: test_changed"...something like that.

What is a test file? A file listed in an active test manifest, ideally including support files. It looks like the existing TestResolver might do the job.

https://dxr.mozilla.org/mozilla-central/rev/a374c35469935a874fefe64d3e07003fc5bc8884/python/mozbuild/mozbuild/testing.py#179

A complication is that a test might be skipped on some platforms, but we don't want that to stop test verification on other platforms. Ideally, test_changed would trigger test verification on platforms where the modified test is enabled; I think a reasonable alternative would be to trigger test verification on all platforms, allowing the harness to recognize the platform skip and no-op where appropriate.

Comment 1

[Geoff Brown [:gbrown]]

I've had a look at implementing "when: tests-changed" paralleling the approach for "when: files-changed" and gotten that working down to the level of determining whether a path corresponds to a test file. I had hoped to use TestResolver for this, but attempts to instantiate TestResolver in the decision task fail at:

https://dxr.mozilla.org/mozilla-central/rev/bdb2387396b4a74dfefb7c983733eed3625e906a/python/mozbuild/mozbuild/testing.py#188

[task 2017-06-02T17:07:02.622846Z]   File "/home/worker/checkouts/gecko/taskcluster/mach_commands.py", line 164, in taskgraph_decision
[task 2017-06-02T17:07:02.622900Z]     return taskgraph.decision.taskgraph_decision(options)
[task 2017-06-02T17:07:02.622967Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 136, in taskgraph_decision
[task 2017-06-02T17:07:02.623023Z]     write_artifact('task-graph.json', tgg.morphed_task_graph.to_json())
[task 2017-06-02T17:07:02.623087Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 180, in morphed_task_graph
[task 2017-06-02T17:07:02.623130Z]     return self._run_until('morphed_task_graph')
[task 2017-06-02T17:07:02.623187Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 286, in _run_until
[task 2017-06-02T17:07:02.623229Z]     k, v = self._run.next()
[task 2017-06-02T17:07:02.623288Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 274, in _run
[task 2017-06-02T17:07:02.623329Z]     do_not_optimize)
[task 2017-06-02T17:07:02.623388Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 43, in optimize_task_graph
[task 2017-06-02T17:07:02.623423Z]     existing_tasks=existing_tasks)
[task 2017-06-02T17:07:02.623508Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 128, in annotate_task_graph
[task 2017-06-02T17:07:02.623557Z]     opt_result = optimize_task(task, params)
[task 2017-06-02T17:07:02.623614Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 83, in optimize_task
[task 2017-06-02T17:07:02.623656Z]     opt_result = opt_fn(task, params, *args)
[task 2017-06-02T17:07:02.623710Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/optimize.py", line 262, in opt_tests_changed
[task 2017-06-02T17:07:02.623758Z]     changed = files_changed.check_tests(params, flavor)
[task 2017-06-02T17:07:02.623819Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 94, in check_tests
[task 2017-06-02T17:07:02.623853Z]     if test_match(path, flavor):
[task 2017-06-02T17:07:02.623917Z]   File "/home/worker/checkouts/gecko/taskcluster/taskgraph/files_changed.py", line 75, in test_match
[task 2017-06-02T17:07:02.623959Z]     resolver = build_obj._spawn(TestResolver)
[task 2017-06-02T17:07:02.624008Z]   File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 623, in _spawn
[task 2017-06-02T17:07:02.624045Z]     topobjdir=self.topobjdir)
[task 2017-06-02T17:07:02.624100Z]   File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/testing.py", line 195, in __init__
[task 2017-06-02T17:07:02.624148Z]     self.topsrcdir, 'build', 'gen_test_backend.py'),
[task 2017-06-02T17:07:02.624208Z]   File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 469, in _run_make
[task 2017-06-02T17:07:02.624249Z]     args = self._make_path()
[task 2017-06-02T17:07:02.624300Z]   File "/home/worker/checkouts/gecko/python/mozbuild/mozbuild/base.py", line 600, in _make_path
[task 2017-06-02T17:07:02.624341Z]     raise Exception('Could not find a suitable make implementation.')

TestResolver runs make("backend.TestManifestBackend") to generate pickle representations of the tests dictionary, which it then reads in and searches to satisfy requests.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fcdfcc2d60de3d44850a9f95c43e6ce5994fe750
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2ea767b238014f983a92c548a163aba1f62f5cd

Comment 2

[Geoff Brown [:gbrown]]

The experience in comment 1 reminds me that "proper" examination of test manifests is linked to the build system, and as such cannot be performed pre-build: We can't determine if a file is a test file in the decision task. If instead, the manifest lookup is performed post-build, perhaps in a normal test task, it may be able to use all-tests.pkl, etc generated at build time. Also, post-build, the manifest lookup can be performed in the context of a specific platform, properly accounting for os, debug, etc annotations.

Let's try a different approach...