Open Bug 1357513 (test-verify) Opened 3 years ago Updated 1 month ago
[meta] New/modified test verification
A major finding from the Stockwell project's triaging experience: Many frequent intermittent test failures arise from the introduction of new tests, or the modification of existing tests. A few random examples: https://bugzilla.mozilla.org/show_bug.cgi?id=1340413#c7 https://bugzilla.mozilla.org/show_bug.cgi?id=1318389#c2 https://bugzilla.mozilla.org/show_bug.cgi?id=1353894#c2 https://bugzilla.mozilla.org/show_bug.cgi?id=1351456#c1 https://bugzilla.mozilla.org/show_bug.cgi?id=1351409#c5 Sometimes a new/modified test fails frequently and obviously on try and the test is improved before check-in to an integration branch. Sometimes a new/modified test fails frequently and obviously on check-in and the changeset is backed out. But sometimes those checks fail and an intermittent test failure is introduced anyway. We can reduce intermittent failures by introducing tools and processes which find these cases faster. The basic strategy here is to notice when tests are being updated and subject those tests to more stringent verification right away. For example, when mochitest test_blah.html is updated in a push to try or an integration branch, a new test-verification job is run and it runs test_blah.html 50 times, in isolation. A similar test-verification mach command might be useful for ad-hoc use in development environments. Not all of the implementation details are clear to me, but some of them are; I'll file dependent bugs.
It feels like there's a fair amount of overlap between this and bug 1322433. Some of the business logic is no doubt different but the basic concept ("run this job N times or until it fails") seems similar. You could consider using action tasks for this, which would have the added bonus of exposing the feature in the treeherder UI: http://gecko.readthedocs.io/en/latest/taskcluster/taskcluster/actions.html
Documentation at: https://developer.mozilla.org/en-US/docs/test_verification
Depends on: 1405369
Depends on: 1405561
Depends on: 1406663
Depends on: 1439591
You need to log in before you can comment on or make changes to this bug.