1406204 - Follow-up on test-verify failures that do not predict intermittent failures

Status: RESOLVED WORKSFORME

(Testing :: General, enhancement, P3)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

TV is intended to find intermittent failures in tests before they fail. Perhaps inevitably, it seems to also have false alarms: TV failures in tests which do not appear to fail otherwise.

Let's investigate those cases...perhaps we can eliminate some of these cases / make TV less bothersome.

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Some bugs I am watching:

bug 1405873
bug 1405732
bug 1405448
bug 1405128
bug 1405426

Comment 2

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Bug 1407178 - TV failure in step 1, likely because initial state assumed (only fails with --repeat).

Comment 3

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I haven't found the time to follow-up on these as much as I had hoped, but I think I understand all the important issues. In summary:

* The most frequent cause of test-verify failure that does not predict subsequent intermittent failure is that a test is non-repeatable. The most frequent sub-issue is that the test assumes an initial state (say, an initial pref setting that is modified by the test and not restored). The test runs fine in continuous integration because it gets a more-or-less fresh browser instance each time it runs, but it cannot be repeated in the same browser session (or in some cases, cannot be repeated in separate browser sessions). These cases are not as important/urgent to address as failures associated with intermittent failures, but they are still weaknesses in the test: Non-repeatable tests are harder to debug, frustrate verification, and are often more susceptible to future failures when tests are re-ordered, re-chunked, etc.

* There are some edge cases where tests should be skipped on a platform, but test-verify tries to run them anyway: bug 1406407.

* There are some edge cases where test-verify runs for too long and times out: bug 1431125.

* In theory, I suspect chaos mode may have issues, but in the current configuration, "partial" chaos mode, I have yet to find a single problem associated with chaos mode. I hope to enable more chaos mode categories in future.