Open Bug 1178441 Opened 5 years ago Updated 1 year ago

Running tests that match a tag on Try results in "No tests run" failures

Categories

(Firefox Build System :: Try, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: botond, Unassigned)

References

Details

Bug 987360 and bug 978846 introduced the ability to do a Try run where only tests that match a particular tag are run, by adding "--tag <tag>" to the Try syntax.

If such a test run results in jobs that contain no tests matching the tag, those jobs fail with a "No tests run" error (example: https://treeherder.mozilla.org/#/jobs?repo=try&revision=f41ba073d187).

This falsely creates the appearance that things have gone wrong, when they haven't.

It would be nice to avoid giving these errors, so that a successfully Try run remains green even if you're using a tag.
Ideally we would be able to automatically run just the chunks that contain the tagged tests. Chunking depends on the platform, so I guess this is fairly difficult to implement in buildbot.

If the chunks containing the relevant tests are already known, selecting just those chunks in try syntax should have the desired effect (-u mochitest-3 in the provided push).
Can we compute, for each chunk, the number of tests that are expected to run in that chunk (i.e. the number of tests in that chunk that match the tag), and if that number is zero, do not issue the "No tests run" failure?
Yeah, the UX here is pretty bad, please bear with us.

There's a fair amount of uncertainty about what the future will look like (given new tools like taskcluster, mozci, autoland and active data). We're still figuring out how best to fit all these new pieces together, but likely the future will be very different from the current reality. That's to say, I think chunks will be an implementation detail that devs won't need to worry about.
Duplicate of this bug: 1255658
Component: General → Try
Product: Testing → Firefox Build System
You need to log in before you can comment on or make changes to this bug.