Open Bug 1488850 Opened 6 years ago Updated 2 years ago

Test verification TV and TVw fail because of unknown test file paths

Categories

(Testing :: Code Coverage, enhancement)

Version 3
enhancement

Tracking

(Not tracked)

People

(Reporter: gabriel-v, Unassigned)

References

(Blocks 1 open bug)

Details

This happens when 'mach try coverage' adds a path for a test that is not registered with TV or TVw. This happens, for example, with paths from the python test suite. This causes instant task failure for TV and TVw.

Should we avoid scheduling TV and TVw tasks when running mach try coverage? Or should we stop failing the task when we find a unknown test file?

Failing try push made with 'mach try coverage': https://treeherder.mozilla.org/#/jobs?repo=try&revision=3429ad812ce4267273bfe7934b4c1737be947cc6&selectedJob=196918179

Error message: 

    Per-test run could not find requested test 'tools/tryselect/test/test_tasks.py'
Here's another example: I added enough paths to 'mach try fuzzy' in such a way that TV doesn't recognize a WPT, and TVw doesn't recognize a reftest, so they both fail.

This try push is based off of central directly, made with 'mach try fuzzy': https://treeherder.mozilla.org/#/jobs?repo=try&revision=4556485e5ad31c0042e24436d2a5e76f6362a1db
Summary: Test verification tasks fail when run by 'mach try coverage' because of unknown test file paths → Test verification TV and TVw fail because of unknown test file paths
This is introduced by bug 1482413, https://hg.mozilla.org/mozilla-central/rev/71af60e17c8b

Geoff: Can we turn those 'self.fatal(...)' lines into warnings? Or is there a smarter way of fixing this?
Flags: needinfo?(gbrown)
Flags: needinfo?(gbrown)
In bug 1482413, we specifically wanted to fail when test-verify is requested for an invalid path. A particular use case we were thinking of was running backfill (TV-bf), manually entering a path, and making a typo. If we s/fatal/warning/, those typos will produce green runs -- which do nothing.
Maybe we could add a flag to indicate a 'mach try coverage' run and behave differently in that case? Or is there a way for 'mach try coverage' to avoid requesting TV for unsupported test types?
Depends on: 1489100
No longer blocks: 1429463
(In reply to Geoff Brown [:gbrown] from comment #4)
> Maybe we could add a flag to indicate a 'mach try coverage' run and behave
> differently in that case? Or is there a way for 'mach try coverage' to avoid
> requesting TV for unsupported test types?

Is there a way to tell which test types are not supported by TV or TVwpt?
Flags: needinfo?(gbrown)
I think there's no file or api providing that information.

Currently TV/TVg support mochitests (including all flavors and subsuites), reftests (including crashtests and jsreftests), and xpcshell-tests. TVw supports all web-platform tests.

This bit of code defines the relationship:

https://dxr.mozilla.org/mozilla-central/rev/d6ac0dc85306b753c1d1d5f3084f73f0d1286b86/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#254-259

I read that as:
 - for TVw, files of interest are found in the wpt manifest
 - for TVg/TV, files of interest are found in the manifest listed at
    https://dxr.mozilla.org/mozilla-central/rev/d6ac0dc85306b753c1d1d5f3084f73f0d1286b86/testing/mozharness/mozharness/mozilla/testing/per_test_base.py#45-50
 - files in 'gpu' or 'webgl' suites should run in TVg, else TV
 - everything else is unsupported
Flags: needinfo?(gbrown)
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.