Closed Bug 1593077 Opened 6 years ago Closed 6 years ago

[wpt-sync] Sync PR 20036 - [testharness.js] Require single-page test opt-in

Categories

(Testing :: web-platform-tests, task, P4)

Product:
Testing
Component:
web-platform-tests
Type:
task

Tracking

(firefox72 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla72
Tracking Flags:
Tracking Status
firefox72 --- fixed

People

(Reporter: wpt-sync, Unassigned)

References

(
URL
)

Details

(Whiteboard: [wptsync downstream])

Sync web-platform-tests PR 20036 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/20036
Details from upstream follow.

Mike Pennisi <mike@mikepennisi.com> wrote:

[testharness.js] Require single-page test opt-in

This enacts the change specified by WPT RFC 28.

https://github.com/web-platform-tests/rfcs/blob/master/rfcs/single_test.md

Overall, this patch appears to be safe to land, though we may want to wait for gh-19904 and gh-20031 before doing so.

In gh-19578, we aimed to extend all of WPT's single-page tests with the new implicit opt-in pattern. Once we completed that, I created this patch and ran it in Chrome Dev and Firefox Nightly.

Most of the discrepancies are due to flaky tests (listed below). The second most common cause comes from references to the API under test outside of a subtest (also listed below). In master today, the resulting ReferenceError causes the test to be reported as a failing single-page test, even though in most cases, it was authored to define multiple sub-tests. The new result is a harness error, and that's more appropriate for these cases.

There are a handful of tests whose results changed for other reasons:

  • these tests are being fixed via https://github.com/web-platform-tests/wpt/pull/19904
    • WebCryptoAPI/derive_bits_keys/
  • these tests rely on new ECMAScript language features that are not yet supported in Firefox. The harness error is a more appropriate status
    • IndexedDB/structured-clone.any.html
    • IndexedDB/structured-clone.any.worker.html
  • these tests includes an unresolvable reference. The harness error is a more appropriate status
    • fetch/metadata/sec-fetch-dest/redirect/multiple-redirect-https-downgrade-upgrade.tentative.sub.html
    • fetch/metadata/sec-fetch-dest/redirect/redirect-http-upgrade.tentative.sub.html
  • this test includes a syntax error. The harness error is a more appropriate status
    • fetch/metadata/sec-fetch-dest/redirect/redirect-https-downgrade.tentative.sub.html -
  • this test references an internal API which is not defined. The harness error is a more appropriate status
    • idle-detection/interceptor.https.html
  • these errors describe the intended effect of the patch
    • infrastructure/expected-fail/uncaught-exception.html
    • infrastructure/expected-fail/unhandled-rejection.html
  • these tests are single-page tests which we failed to identify initially; we're updating them via https://github.com/web-platform-tests/wpt/pull/20031
    • preload/preload-csp.sub.html
    • preload/preload-default-csp.sub.html

<details>
<summary>flaky tests</summary>

  • detected by Chrome
    • background-fetch/
    • content-security-policy/
    • css/ - flaky
    • html/semantics/links/links-created-by-a-and-area-elements/
    • intersection-observer/cross-origin-iframe.sub.html
    • longtask-timing/longtask-in-sibling-iframe-crossorigin.html
    • offscreen-canvas/
    • scroll-to-text-fragment/scroll-to-text-fragment.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/cors-check.https.html
    • webmessaging/broadcastchannel/basics.html
    • webmessaging/broadcastchannel/workers.html
  • detected by Firefox
    • css/css-animations/CSSAnimation-startTime.tentative.html
    • css/css-backgrounds/background-334.html
    • css/css-multicol/animation/column-rule-color-interpolation.html
    • css/css-properties-values-api/property-cascade.html
    • css/css-transitions/events-007.html
    • css/css-writing-modes/text-combine-upright-parsing-valid-001.html
    • html/cross-origin-embedder-policy/require-corp.https.html
    • html/semantics/scripting-1/the-script-element/load-error-events-3.html
    • html/semantics/scripting-1/the-script-element/script-onerror-insertion-point-1.html
    • html/webappapis/dynamic-markup-insertion/opening-the-input-stream/abort-refresh-immediate.window.html
    • offscreen-canvas/the-offscreen-canvas/offscreencanvas.commit.w.html
    • performance-timeline/case-sensitivity.any.html
    • pointerevents/pointerevent_setpointercapture_inactive_button_mouse.html
    • referrer-policy/gen/req.attr/same-origin/img-tag/cross-https.swap-origin.http.html
    • resource-timing/resource_connection_reuse.html
    • resource-timing/resource_timing_buffer_full_eventually.html
    • service-workers/service-worker/ready.https.html
    • upgrade-insecure-requests/gen/worker-classic-data.meta/upgrade/fetch/cross-http-downgrade.downgrade.https.html
    • webaudio/the-audio-api/the-scriptprocessornode-interface/simple-input-output.html
    • webdriver/
    • webvtt/

</details>

<details>
<summary>unsafe API references</summary>

  • detected by Chrome
    • streams/readable-byte-streams/construct-byob-request.any.html
    • streams/readable-byte-streams/construct-byob-request.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.sharedworker.html
    • streams/readable-byte-streams/construct-byob-request.any.worker.html
    • trusted-types/
  • detected by Firefox
    • 2dcontext/wide-gamut-canvas/imageData-colorManagedBehavior.html
    • animation-worklet/worklet-animation-pause.https.html
    • animation-worklet/worklet-animation-without-target.https.html
    • css/css-properties-values-api/conditional-rules.html
    • css/css-properties-values-api/unit-cycles.html
    • css/css-properties-values-api/url-resolution.html
    • css/css-typed-om/
    • custom-elements/form-associated/ElementInternals-accessibility.html
    • custom-elements/form-associated/form-associated-callback.html
    • encoding/streams/decode-utf8.any.serviceworker.html
    • encoding/streams/realms.window.html
    • streams/piping/pipe-through.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.html
    • streams/readable-byte-streams/construct-byob-request.any.serviceworker.html
    • streams/readable-byte-streams/construct-byob-request.any.worker.html
    • streams/transform-streams/brand-checks.any.html
    • streams/transform-streams/brand-checks.any.serviceworker.html
    • streams/transform-streams/brand-checks.any.worker.html
    • streams/transform-streams/properties.any.serviceworker.html
    • streams/writable-streams/brand-checks.any.html
    • streams/writable-streams/brand-checks.any.serviceworker.html
    • streams/writable-streams/brand-checks.any.worker.html
    • streams/writable-streams/properties.any.serviceworker.html
    • trusted-types/GlobalEventHandlers-onclick.tentative.html
    • trusted-types/Node-multiple-arguments.tentative.html
    • trusted-types/TrustedTypePolicyFactory-metadata.tentative.html
    • trusted-types/WorkerGlobalScope-importScripts.https.html
    • trusted-types/block-Node-multiple-arguments.tentative.html
    • trusted-types/block-eval.tentative.html
    • trusted-types/block-string-assignment-to-Document-write.tentative.html
    • trusted-types/block-string-assignment-to-Element-setAttribute.tentative.html
    • trusted-types/eval-with-permissive-csp.tentative.html
    • trusted-types/trusted-types-eval-reporting-no-unsafe-eval.tentative.https.html
    • trusted-types/trusted-types-eval-reporting-report-only.tentative.https.html
    • trusted-types/trusted-types-eval-reporting.tentative.https.html
    • trusted-types/trusted-types-report-only.tentative.https.html
    • trusted-types/trusted-types-reporting-check-report.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworklet-messageport.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworklet-postmessage-sharedarraybuffer.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletglobalscope-sample-rate.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletglobalscope-timing-info.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletnode-onerror.https.html
    • webaudio/the-audio-api/the-audioworklet-interface/audioworkletnode-output-channel-count.https.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/cors-check.https.html
    • webaudio/the-audio-api/the-mediaelementaudiosourcenode-interface/no-cors.https.html

</details>

Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Whiteboard: [wptsync downstream] → [wptsync downstream error]
Whiteboard: [wptsync downstream error] → [wptsync downstream]
Test result changes from PR not available.
Pushed by wptsync@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/62491361e839 [wpt PR 20036] - [testharness.js] Require single-page test opt-in, a=testonly

https://hg.mozilla.org/mozilla-central/rev/62491361e839

Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox72: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla72
You need to log in before you can comment on or make changes to this bug.