1506352 - Stop running web-platform-tests in jstests.py until we have a working strategy for updates

Status: RESOLVED FIXED

Description

[James Graham [:jgraham]]

We started running some wasm tests from web-platform-tests in the jsshell in Bug 1333800. However we didn't properly consider how things would work when the tests are updated; a wpt update introducing a test that doesn't pass currently breask the build. There are two things that would help:

* Start producing a wptreport.json log and run the jsshell tests as part of the wpt update process, so we are able to update the metadata to match the observed failures on import.

* Don't run the jsshell tests as part of the build, but as a seperate job. IN particular make the wpt part of this tier 2 so that failures can be fixed asynchronously rather than requiring a backout.

Before we can do one or both of those things we need to reconsider running these tests by default.

Comment 1

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Attachment: Patch v1

Comment 2

[James Graham [:jgraham]]

Comment on attachment 9026169 [details] [diff] [review]
Patch v1

Review of attachment 9026169 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/tests/lib/results.py
@@ +56,5 @@
>      CRASH = 'CRASH'
>  
>      """Classified result from a test run."""
>  
> +    def __init__(self, test, result, results, wpt_results=None):

I wonder if it's possible to just pass the formatter in here rather than have the handler abstraction, but I don't have a strong opinion.

@@ +127,5 @@
>                  results.append(test_result)
>                  stdout.append(test_output)
>  
>          output.out = "\n".join(stdout) + "\n"
> +        

Whitespace

::: js/src/tests/lib/wptreport.py
@@ +51,5 @@
> +                            * "status": the actual status of the whole test;
> +                            * "expected": the expected status of the whole test;
> +                            * "subtests": a list of dicts with keys "test",
> +                              "subtest", "status" and "expected".
> +        :param float duration: the runtime of the test 

Also whitespace

Comment 3

[Benjamin Bouvier [:bbouvier]]

Comment on attachment 9026169 [details] [diff] [review]
Patch v1

Review of attachment 9026169 [details] [diff] [review]:
-----------------------------------------------------------------

::: js/src/tests/jstests.py
@@ +202,5 @@
>                           type='choice', choices=['automation', 'none'],
>                           help='Output format. Either automation or none'
>                           ' (default %default).')
> +    output_og.add_option('--log-wptreport', dest='wptreport', action='store',
> +                         help='Path to write a wptreport output file')

nit: Can you explicit `wptreport` into `a Web Platform test report`, so there's no doubt for people who read this for the first time?

::: js/src/tests/lib/results.py
@@ +245,5 @@
>              self.n += 1
>          else:
>              result = TestResult.from_output(output)
> +
> +            if self.wptreport and result.wpt_results:

Check with `is not None`, to avoid surprising results?

@@ +335,5 @@
>              self.slog.suite_end()
>          else:
>              self.list(completed)
>  
> +        if self.wptreport:

ditto?

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5931ccb19e50
Add a --log-wptreport option to jstests; r=bbouvier,jgraham

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/55d9c8696101
Followup: Add missing blank line to placate flake8.

Comment 6

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/5931ccb19e50
https://hg.mozilla.org/mozilla-central/rev/55d9c8696101