1807109 - [wpt-sync] Sync PR 37653 - WIP: Initial support for running tests against WebKitTestRunner

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 37653 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/37653
Details from upstream follow.

Sam Sneddon <[e-mail]> wrote:

WIP: Initial support for running tests against WebKitTestRunner

Obviously this shouldn't actually live in executorcontentshell.py, as there is a fair bit of divergence. It does seem like the content_shell code has problems with leaking files (pipes and semaphores, I think?), as a relatively modest number of restarts will cause macOS to run out of file handles for the process.

But, as evidence this mostly works:

gsnedders@gsnedders-marsha web-platform-tests % ./wpt run --no-restart-on-unexpected --no-manifest-update --log-mach - --log-mach-level info --binary /Users/gsnedders/projects/Safari/OpenSource/WebKitBuild/Release/WebKitTestRunner --manifest MANIFEST.json --metadata infrastructure/metadata wktr infrastructure
...
 1:11.56 INFO No more tests
 1:11.56 INFO Closing logging queue
 1:11.56 INFO queue closed
 1:11.59 INFO Got 5 unexpected results, with 1 unexpected passes
 1:11.59 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 297 checks (154 subtests, 143 tests)
Expected results: 253
Skipped: 39 tests
Unexpected results: 5
  test: 4 (1 fail, 1 ok, 2 timeout)
  subtest: 1 (1 fail)

Unexpected Results
------------------
/infrastructure/expected-fail/unhandled-rejection-following-subtest.html
  UNEXPECTED-OK /infrastructure/expected-fail/unhandled-rejection-following-subtest.html
/infrastructure/server/http2-websocket.sub.h2.any.worker.html
  TIMEOUT /infrastructure/server/http2-websocket.sub.h2.any.worker.html -
/infrastructure/server/http2-websocket.sub.h2.any.html
  TIMEOUT /infrastructure/server/http2-websocket.sub.h2.any.html -
/infrastructure/server/test-pac.html
  FAIL test that PAC metadata is respected - promise_test: Unhandled rejection with value: object "TypeError: Load failed"
get_stack@http://web-platform.test:8000/resources/testharness.js:4516:30
AssertionError@http://web-platform.test:8000/resources/testharness.js:4509:31
assert@http://web-platform.test:8000/resources/testharness.js:4493:37
@http://web-platform.test:8000/resources/testharness.js:759:35
@http://web-platform.test:8000/resources/testharness.js:2591:30
@http://web-platform.test:8000/resources/testharness.js:2638:40
/infrastructure/assumptions/ahem.html
  FAIL /infrastructure/assumptions/ahem.html - /infrastructure/assumptions/ahem.html ['242cbe5affeea2dfb65e72a2a717a4569cdcd018']
/infrastructure/assumptions/ahem-ref.html ['2de41e36cbb8b36868361c3170389cb84fc46ef9']

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Pushed to try https://treeherder.mozilla.org/#/jobs?repo=try&revision=d1f04e4d2e5350e242b18e74676d81cae4b2ba46

Comment 2

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Test result changes from PR not available.

Comment 3

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Test result changes from PR not available.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b74480ca11ed
[wpt PR 37653] - Initial support for running tests against WebKitTestRunner, a=testonly

Comment 5

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/b74480ca11ed