Open Bug 1418375 Opened 6 years ago Updated 2 years ago

test-verify-wpt improvements for mass file changes / imports

Categories

(Testing :: General, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: gbrown, Unassigned)

References

(Blocks 1 open bug)

Details

One of the challenges of test verification is time: Try to strenuously verify too many tests -- or even just a few long-running tests -- and you find the wall clock has advanced quite a ways. Too long of a run keeps people waiting for results, and risks timeouts (and relaxation of timeouts needs to be weighed against the probability of hangs).

The initial implementation of test-verify favored speed over completeness:
 - if a single test is taking more than about 10 minutes to verify, give up
 - if many tests files change, only verify the first 10
 - if a test verification job is taking longer than an hour, give up

The initial implementation of test-verify-wpt followed the same model. However, web-platform tests are often updated en-masse on mozilla-central, to incorporate upstream modifications, and need to undergo a similar verification process. It would be nice if test-verify-wpt could handle those import scenarios.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1411660#c7 particularly.

Specific enhancements:
 - a way to relax max-time and max-tests, perhaps from try
 - more efficient mozharness calls into the test harness (not one run per affected test)
 - possibly less logging (see --verify-log-full use)
See Also: → 1413729
Priority: -- → P3
Type: defect → enhancement
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.