1397970 - Verify tests for path/manifest when manifest updated

Status: NEW

(Testing :: General, enhancement, P3)

Description

[Geoff Brown [:gbrown]]

This is a vague enhancement idea...something to consider.

Currently test-verify is triggered when a test file is modified. It could also be triggered when a .ini file is updated, with verification in this case applied to the entire manifest.

This would be useful when a skip is removed or a skip-if is modified, or tests are otherwise added or removed from a manifest *without* modifying the test. (The normal case is probably that a test file is modified *and* the manifest is updated, which today results in only the modified test coming under scrutiny; that is probably the optimal behavior for that case, and I wouldn't want to see that change.)

Comment 1

[Geoff Brown [:gbrown]]

Additionally, if this works, we could then *not* run any task but test-verify when a manifest is updated (bug 1383880 / check with Dustin for implementation details).

Comment 2

[Nick Alexander :nalexander [he/him]]

I just got bitten by this. In https://treeherder.mozilla.org/jobs?repo=try&revision=11e5fa13b9cc87535b541d6d8588ab9040258f07, the macOS TV jobs appear green. But in fact, I only modified the xpcshell.ini definitions, and no jobs ran.

TV would be more useful if it computed a "diff of the test manifest state" or something similar. (And, unrelated, if a TV job that did no work was marked as failing or otherwise gave an indication that it didn't do what was probably expected.)