[wpt-sync] Sync PR 15634 - [testharness.js] Tolerate late tests
Categories
(Testing :: web-platform-tests, enhancement, P4)
Tracking
(firefox69 fixed)
| Tracking | Status | |
|---|---|---|
| firefox69 | --- | fixed |
People
(Reporter: mozilla.org, Unassigned)
References
()
Details
(Whiteboard: [wptsync downstream])
Sync web-platform-tests PR 15634 into mozilla-central (this bug is closed when the sync is complete).
PR: https://github.com/web-platform-tests/wpt/pull/15634
Details from upstream follow.
Mike Pennisi <mike@mikepennisi.com> wrote:
[testharness.js] Tolerate late tests
We might try to signal a harness error whenever this occurs, but a few details eventually convinced me to silently ignore it:
First, correctly reporting this condition would require a large amount of structural change. testharness.js was not designed for a "COMPLETE to COMPLETE" transition, so things like updating the DOM will take some doing. In gh-14533, @mdittmer noted a discrepancy between the reported data and the JSON written to the document--that's a side effect of the harness emitting an array which it later extends.
It's also not clear how this would effect
test_done_callback. It would be tricky to name/document if we allowed for functions registered with that API to be invoked more than once. Alternatively, we could implement some kind of "all clear" period, where the harness sits idle waiting for late tests before reporting completion. I don't know how to balance the desire to guard against this edge case against our interest to minimize the duration of test execution.Finally, we can't detect this situation deterministically. We currently ignore late tests--just in an inconsistent way. If we tried to disallow them, then we might prevent some test bugs, but we might also allow test bugs to detract from stability.
There are a handful of tests which exhibit this behavior today. I've submitted corrections for those in separate patches:
|
Assignee | |
Comment 1•5 years ago
|
||
Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=9fc3486e2fcf1c3e2d284dee14120d4cffd8df00
|
|
Assignee | |
Comment 2•5 years ago
|
||
Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=d0a72d8e4a2075d3e8c392bfe44871ec05b5c132
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/mozilla-inbound/rev/a6b2fe416307 [wpt PR 15634] - [testharness.js] Tolerate late tests, a=testonly
|
||
Comment 4•5 years ago
|
||
| bugherder | ||
Description
•