1587660 - [wpt-sync] Sync PR 19612 - [testharness.js] Don't complete after allowed exc.

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 19612 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/19612
Details from upstream follow.

Mike Pennisi <[e-mail]> wrote:

[testharness.js] Don't complete after allowed exc.

The following tests intentionally produce an uncaught exception and then define
subtests:

• html/semantics/scripting-1/the-script-element/module/error-and-slow-dependency.html
• html/webappapis/scripting/processing-model-2/window-onerror-parse-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error-throw.html

testharness.js immediately transitions to "complete" and ignores the
subsequent subtests.

---

Today, the tests referenced in the commit message are reported as passing single-page tests. Once we've completed the implementation of WPT RFC 28, they will instead be reported as test harness errors.

As an alternative to this patch, we could modify each of the tests. If a subtest is defined before the exception is thrown, then all the subtests are reported as expected. (We could, for example, declare the existing synchronous subtests as asynchronous subtests instead.)

Changing the harness is definitely riskier, but it also seems more correct to me.

This change could interfere with single-page tests that use allow_uncaught_exceptions. My experiments don't indicate that any such tests exist, but it's difficult to demonstrate or summarize that research here. A more blunt heuristic might do. Only three tests use allow_uncaught_exception without invoking functions named test, promise_test, or async_test:

 $ git grep -l allow_uncaught_exception | xargs grep -LE '\b(test|promise_test|async_test)\s*\('
 IndexedDB/fire-error-event-exception.html
 IndexedDB/fire-success-event-exception.html
 custom-elements/upgrading/upgrading-enqueue-reactions.html

All three define subtests indirectly through helper functions.

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=f7bd6362fcc033016742ae563bd7b7bb524020ca

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Result changes from PR not available.

Comment 3

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Result changes from PR not available.

Comment 4

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Attachment: Bug 1587660 [wpt PR 19612] - [testharness.js] Don't complete after allowed exc., a=testonly

Automatic update from web-platform-tests
[testharness.js] Don't complete after allowed exc. (#19612)

The following tests intentionally produce an uncaught exception and then define
subtests:

• html/semantics/scripting-1/the-script-element/module/error-and-slow-dependency.html
• html/webappapis/scripting/processing-model-2/window-onerror-parse-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error-throw.html

testharness.js immediately transitions to "complete" and ignores the
subsequent subtests.

--

wpt-commits: e9465a52d29ca76d30e327db675bb66139f27b8c
wpt-pr: 19612

Depends on D53418

Comment 5

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Attachment: Bug 1587660 [wpt PR 19612] - [testharness.js] Don't complete after allowed exc., a=testonly

Automatic update from web-platform-tests
[testharness.js] Don't complete after allowed exc. (#19612)

The following tests intentionally produce an uncaught exception and then define
subtests:

• html/semantics/scripting-1/the-script-element/module/error-and-slow-dependency.html
• html/webappapis/scripting/processing-model-2/window-onerror-parse-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error-throw.html

testharness.js immediately transitions to "complete" and ignores the
subsequent subtests.

--

wpt-commits: e9465a52d29ca76d30e327db675bb66139f27b8c
wpt-pr: 19612

Depends on D53428

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Attachment: Bug 1587660 [wpt PR 19612] - [testharness.js] Don't complete after allowed exc., a=testonly

Automatic update from web-platform-tests
[testharness.js] Don't complete after allowed exc. (#19612)

The following tests intentionally produce an uncaught exception and then define
subtests:

• html/semantics/scripting-1/the-script-element/module/error-and-slow-dependency.html
• html/webappapis/scripting/processing-model-2/window-onerror-parse-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error.html
• html/webappapis/scripting/processing-model-2/window-onerror-runtime-error-throw.html

testharness.js immediately transitions to "complete" and ignores the
subsequent subtests.

--

wpt-commits: e9465a52d29ca76d30e327db675bb66139f27b8c
wpt-pr: 19612

Comment 7

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Result changes from PR not available.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ff22aa46bca0
[wpt PR 19612] - [testharness.js] Don't complete after allowed exc., a=testonly

Comment 9

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/ff22aa46bca0