1380180 - 'make check' failures cause other test suites not to be run

Status: NEW

(Firefox Build System :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Hitting an intermittent failure in a 'make check' test (such as bug 1359596, bug 1349943, bug 1348513, bug 1352633, etc.) causes the build to turn red.  It's fine that there's a failure indicated (though it should be orange -- there's been a huge amount of regressions in orange vs. red distinction since taskcluster).  However, it causes none of the other test suites to be run.

This breaks the general expectation that intermittent test failures shouldn't cause *other* test suites not to be run.  Developers expect that if a build fails, then the tests that were waiting on that build won't be run.  But having an intermittent test failure cause other tests not to be run is bad because:

 * it makes it take longer to get the test results (since it requires noticing the failure and then retriggering the build)

 * developers may well not notice that this test failure was actually treated like a build failure (especially now that taskcluster has made orange vs. red distinction pretty meaningless) and caused other test suites not to run, and therefore fail to detect failures on try, leading to them landing with test failures.


If there's a failure in 'make check' tests (which are run as part of the build job), the failure shouldn't prevent triggering the other test suites that should have been run off of that build.


Discussion on #developers suggests this is a regression from when builds were switched to taskcluster.

Comment 1

[Chris AtLee [:catlee]]

Not sure if Core:Build Config is the right place for this, or if this should go into one of the TC components.

The problem is that TC does not have the ability to trigger tests before the build task is finished, like we had in buildbot. It's been discussed a few times, but never implemented mostly because we think that 'make check' should contain as few tests as possible.

Given that this feature doesn't exist in TC, I think the correct solution right now is to make 'make check' failures non-fatal for the build as far as TC is concerned, but still turn the job orange in TH.

Comment 2

[Gregory Szorc [:gps]]

Yes, this is a build system bug. This is due to the way `make check` is invoked in buildbot vs TC. In build bot, the build system was invoked, packages were uploaded (triggering other test jobs), then `make check` ran. This was a hack to facilitate starting tests before `make check` finished.

The TC architecture doesn't support this (easily). So the build task in TC runs to completion (including `make check`) before files are uploaded and tests run.

We've been trying to move things out of `make check`. But some things linger. We can either finish eliminating the heavy parts of `make check` (bug 992323) or hack up the build system in automation to not fail if there is a failure during `make check`. I don't think the latter is as trivial as it sounds.