test-verify should check the manifests of the tests being run
Categories
(Testing :: Mochitest, defect, P2)
Tracking
(Not tracked)
People
(Reporter: Gijs, Unassigned, NeedInfo)
References
Details
See also: bug 1993106.
Trypush: https://treeherder.mozilla.org/jobs?repo=try&revision=70a3bd38007ad7bb8de56e195b99c620d6c0b918
When test-verify ran, it did not flag the toml changes as broken. They only got flagged by the mochitest-browser runs, but ./mach try auto did not run any of those, so then the patch got backed out when it landed instead ( bug 1990922 comment 5 ).
verify jobs should do the same manifest checks as well, for any tests they are asked to verify, and cause failures in the verify jobs.
|
||
Comment 1•2 months ago
|
||
I edited the same file (browser_mixed_content_cert_override.js), and on try, it ran TV as expected. so there must be something specific with that try push and decision task.
please mention this on #devhelp or here in this bug if you observe this again!
|
Reporter | |
Comment 2•2 months ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #1)
I edited the same file (
browser_mixed_content_cert_override.js), and on try, it ran TV as expected. so there must be something specific with that try push and decision task.
This isn't what the issue in comment 0 is about. TV jobs also ran on the trypush linked in comment 0.
TV jobs do not check the manifests of the tests they run. So the mochitest-browser failure:
The 'prefs' key must be set in the DEFAULT section of a manifest. Fix the following manifests: browser/base/content/test/siteIdentity/browser.toml
That got the patch backed out does not appear on the trypush. So this bug is about fixing that: making these warnings fail TV jobs, if any test in that manifest is being run by TV.
How could I update the bug summary to make this more obvious?
Description
•