Closed Bug 1593077 Opened 1 year ago Closed 1 year ago

[wpt-sync] Sync PR 20036 - [testharness.js] Require single-page test opt-in

Categories

(Testing :: web-platform-tests, task, P4)

task

Tracking

(firefox72 fixed)

RESOLVED FIXED
mozilla72
Tracking Status
firefox72 --- fixed

People

(Reporter: mozilla.org, Unassigned)

References

()

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 20036 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/20036
Details from upstream follow.

Mike Pennisi <mike@mikepennisi.com> wrote:

[testharness.js] Require single-page test opt-in

This enacts the change specified by WPT RFC 28.

https://github.com/web-platform-tests/rfcs/blob/master/rfcs/single_test.md


Overall, this patch appears to be safe to land, though we may want to wait for gh-19904 and gh-20031 before doing so.

In gh-19578, we aimed to extend all of WPT's single-page tests with the new implicit opt-in pattern. Once we completed that, I created this patch and ran it in Chrome Dev and Firefox Nightly.

Most of the discrepancies are due to flaky tests (listed below). The second most common cause comes from references to the API under test outside of a subtest (also listed below). In master today, the resulting ReferenceError causes the test to be reported as a failing single-page test, even though in most cases, it was authored to define multiple sub-tests. The new result is a harness error, and that's more appropriate for these cases.

There are a handful of tests whose results changed for other reasons:

  • these tests are being fixed via https://github.com/web-platform-tests/wpt/pull/19904
    • WebCryptoAPI/derive_bits_keys/
  • these tests rely on new ECMAScript language features that are not yet supported in Firefox. The harness error is a more appropriate status
    • IndexedDB/structured-clone.any.html
    • IndexedDB/structured-clone.any.worker.html
  • these tests includes an unresolvable reference. The harness error is a more appropriate status
    • fetch/metadata/sec-fetch-dest/redirect/multiple-redirect-https-downgrade-upgrade.tentative.sub.html
    • fetch/metadata/sec-fetch-dest/redirect/redirect-http-upgrade.tentative.sub.html
  • this test includes a syntax error. The harness error is a more appropriate status
    • fetch/metadata/sec-fetch-dest/redirect/redirect-https-downgrade.tentative.sub.html -
  • this test references an internal API which is not defined. The harness error is a more appropriate status
    • idle-detection/interceptor.https.html
  • these errors describe the intended effect of the patch
    • infrastructure/expected-fail/uncaught-exception.html
    • infrastructure/expected-fail/unhandled-rejection.html
  • these tests are single-page tests which we failed to identify initially; we're updating them via https://github.com/web-platform-tests/wpt/pull/20031
    • preload/preload-csp.sub.html
    • preload/preload-default-csp.sub.html

<details>
<summary>flaky tests</summary>

  • detected by Chrome
    • background-fetch/
    • content-security-policy/
    • css/ - flaky
    • html/semantics/links/links-created-by-a-and-area-elements/
    • intersection-observer/cross-origin-iframe.sub.html
    • longtask-timing/longtask-in-sibling-iframe-crossorigin.html
    • offscreen-canvas/
    • scroll-to-text-fragment/scroll-to-text-fragment.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/cors-check.https.html
    • webmessaging/broadcastchannel/basics.html
    • webmessaging/broadcastchannel/workers.html
  • detected by Firefox
    • css/css-animations/CSSAnimation-startTime.tentative.html
    • css/css-backgrounds/background-334.html
    • css/css-multicol/animation/column-rule-color-interpolation.html
    • css/css-properties-values-api/property-cascade.html
    • css/css-transitions/events-007.html
    • css/css-writing-modes/text-combine-upright-parsing-valid-001.html
    • html/cross-origin-embedder-policy/require-corp.https.html
    • html/semantics/scripting-1/the-script-element/load-error-events-3.html
    • html/semantics/scripting-1/the-script-element/script-onerror-insertion-point-1.html
    • html/webappapis/dynamic-markup-insertion/opening-the-input-stream/abort-refresh-immediate.window.html
    • offscreen-canvas/the-offscreen-canvas/offscreencanvas.commit.w.html
    • performance-timeline/case-sensitivity.any.html
    • pointerevents/pointerevent_setpointercapture_inactive_button_mouse.html
    • referrer-policy/gen/req.attr/same-origin/img-tag/cross-https.swap-origin.http.html
    • resource-timing/resource_connection_reuse.html
    • resource-timing/resource_timing_buffer_full_eventually.html
    • service-workers/service-worker/ready.https.html
    • upgrade-insecure-requests/gen/worker-classic-data.meta/upgrade/fetch/cross-http-downgrade.downgrade.https.html
    • webaudio/the-audio-api/the-scriptprocessornode-interface/simple-input-output.html
    • webdriver/
    • webvtt/

</details>

<details>
<summary>unsafe API references</summary>

  • detected by Chrome
    • streams/readable-byte-streams/construct-byob-request.any.html
    • streams/readable-byte-streams/construct-byob-request.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.sharedworker.html
    • streams/readable-byte-streams/construct-byob-request.any.worker.html
    • trusted-types/
  • detected by Firefox
    • 2dcontext/wide-gamut-canvas/imageData-colorManagedBehavior.html
    • animation-worklet/worklet-animation-pause.https.html
    • animation-worklet/worklet-animation-without-target.https.html
    • css/css-properties-values-api/conditional-rules.html
    • css/css-properties-values-api/unit-cycles.html
    • css/css-properties-values-api/url-resolution.html
    • css/css-typed-om/
    • custom-elements/form-associated/ElementInternals-accessibility.html
    • custom-elements/form-associated/form-associated-callback.html
    • encoding/streams/decode-utf8.any.serviceworker.html
    • encoding/streams/realms.window.html
    • streams/piping/pipe-through.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.html
    • streams/readable-byte-streams/construct-byob-request.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.worker.html
    • streams/transform-streams/brand-checks.any.html
    • streams/transform-streams/brand-checks.any.serviceworker.html
    • streams/transform-streams/brand-checks.any.worker.html
    • streams/transform-streams/properties.any.serviceworker.html
    • streams/writable-streams/brand-checks.any.html
    • streams/writable-streams/brand-checks.any.serviceworker.html
    • streams/writable-streams/brand-checks.any.worker.html
    • streams/writable-streams/properties.any.serviceworker.html
    • trusted-types/GlobalEventHandlers-onclick.tentative.html
    • trusted-types/Node-multiple-arguments.tentative.html
    • trusted-types/TrustedTypePolicyFactory-metadata.tentative.html
    • trusted-types/WorkerGlobalScope-importScripts.https.html
    • trusted-types/block-Node-multiple-arguments.tentative.html
    • trusted-types/block-eval.tentative.html
    • trusted-types/block-string-assignment-to-Document-write.tentative.html
    • trusted-types/block-string-assignment-to-Element-setAttribute.tentative.html
    • trusted-types/eval-with-permissive-csp.tentative.html
    • trusted-types/trusted-types-eval-reporting-no-unsafe-eval.tentative.https.html
    • trusted-types/trusted-types-eval-reporting-report-only.tentative.https.html
    • trusted-types/trusted-types-eval-reporting.tentative.https.html
    • trusted-types/trusted-types-report-only.tentative.https.html
    • trusted-types/trusted-types-reporting-check-report.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworklet-messageport.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworklet-postmessage-sharedarraybuffer.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletglobalscope-sample-rate.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletglobalscope-timing-info.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletnode-onerror.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletnode-output-channel-count.https.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/cors-check.https.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/no-cors.https.html

</details>

Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Test result changes from PR not available.
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/62491361e839
[wpt PR 20036] - [testharness.js] Require single-page test opt-in, a=testonly
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla72
You need to log in before you can comment on or make changes to this bug.