1412349 - test-verify sometimes does not run when it should

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Geoff Brown [:gbrown]]

In https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=d5be34ec792e30745652c742b08512ad660e9dfc, a .js file was modified and test-verify was run on Windows and OSX, but not on Linux: Why did it not run on Linux?

This might be oddly related to bug 1410911, where TV is run even when no applicable files are modified.

Another troubling possibility is that it was eliminated by SETA.

Comment 1

[Geoff Brown [:gbrown]]

:jmaher - Is there a way to avoid having test-verify eliminated by SETA?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

you could add something here:
https://github.com/mozilla/treeherder/blob/master/treeherder/seta/preseed.json

 {"platform": "*", "buildtype": "*", "testtype": "test-verify", "buildsystem": "taskcluster", "expiration_date": "*", "priority": 1}

Comment 3

[Andrew Halberstadt [:ahal]]

This can also be tweaked in the taskgraph code (which might be cleaner). It would require a minor refactor, but we could also exclude the 'lint' tasks if we did this. Basically we want to make sure these tasks have the "non-seta" optimization:
https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/tests.py#972

Comment 4

[Geoff Brown [:gbrown]]

Thanks :ahal -- that's a good idea. It might be nice to have a SETA item in the yml actually. Is that what you had in mind?

Comment 5

[Andrew Halberstadt [:ahal]]

Yeah something like that. Keep in mind we can already specify 'optimization'/'when' in the .yml. So maybe we just need to specify one of those in the test-verify task and make sure transforms/tests.py doesn't overwrite it.

Comment 6

[Geoff Brown [:gbrown]]

This problem seemed to stop when we changed the scheduling back to when: files-changed, in bug 1410911. I'll leave this open at least until bug 1410911 is resolved.

Comment 8

[Geoff Brown [:gbrown]]

This was noted again in bug 1439589, and is readily apparent on integration branches: test-verify is often not run because it is SETA-optimized.

Comment 9

[Geoff Brown [:gbrown]]

Attachment: do not seta-optimize SCHEDULES.inclusive tasks

Here's a slightly impulsive fix...I'm hopeful but not sure that it is sensible.

Does it make sense to avoid SETA when a task is SCHEDULES.inclusive? Have I implemented that right?

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8952560 [details] [diff] [review]
do not seta-optimize SCHEDULES.inclusive tasks

Review of attachment 8952560 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, that makes sense: SETA is a probabalistic approach to whether a task should be run.  SCHEDULES.exclusive is a more reliable approach, so it should override SETA.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/05a9497b18e7
Avoid SETA optimization of SCHEDULES.inclusive suites, like test-verify; r=dustin

Comment 12

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/05a9497b18e7