Closed Bug 1617827 Opened 5 months ago Closed 4 months ago

Import wasm spec-tests for proposals we implement

Categories

(Core :: Javascript: WebAssembly, task, P2)

task

Tracking

()

RESOLVED FIXED
mozilla76
Tracking Status
firefox76 --- fixed

People

(Reporter: lth, Assigned: rhunt)

References

(Blocks 1 open bug)

Details

Attachments

(4 files, 1 obsolete file)

We want to ensure that we pass the bulk memory spec tests, but we have to import them in a way that separates them from the "standard" spec tests, probably as a separately copied test repo.

I'm looking at this now. I initially tried to use the wasm testsuite repository, but that's not going to be sufficient for our use cases. The big problem is that it only contains wast files, and we require the spec interpreter for each proposal to convert these wast file to js/wpt test formats. We also want other features such as excluding certain tests, and applying a patch to the generated files.

With those things in mind, I've written a small rust program to build us a minimal testsuite of js/wpt files for our implemented proposals. It works by cloning the spec and all the proposals repos, generating the test files using the correct interpreter, and de-duplicating tests between proposals that are the same. The tool does some work to reduce differences between generated files by attempting a merge with the master repo.

I've got test file generation working well, need to work on some more integration so the files can run in jit-tests/ and wpt.

Assignee: lhansen → rhunt

I've got a repo up for the tool now [1]. Try run using the tool [2].

We can now specify the proposals we are testing in a configurable manner, here's the config file I'm using currently [3]. The nice part is this lets us track which tests in each proposal are currently excluded because they're failing, and customize the 'directives.txt' for each proposal so that they don't run if the feature isn't enabled. Each repository specifies its upstream, and the tool will merge and diff with it to generate a minimal amount of tests to include.

[1] https://github.com/eqrion/wasm-generate-testsuite
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=3cc599596462e7a34798eb2cd4e6a41226eb44f4
[3] https://hg.mozilla.org/try/file/8521fe79f73235ec654553a329634770a8e778c9/js/src/jit-test/etc/wasm-config.toml

This is needed so that we can find include the 'harness/sync_index.js' for
each proposal using its directives.txt.

Before this commit, this was done by lib/wasm-testharness.js loading it from a
relative path. We could assume that each proposal has the same sync_index.js,
as it doesn't change often, but that seemed like it could lead to a hard to
discover bug later down the road.

This switches our test import Makefile to use the wasm-generate-testsuite tool
for importing spec-tests. A new config file declaring the proposals to import,
along with tests to skip, is added to jit-test/etc.

There was an ability to apply a .patch file to paper over a back-compat issue
is types.wast.js. It appears that newer spec/tests don't include a types.wast,
so I don't think this patch is necessary anymore. We can add this ability back
later if needed.

Depends on D65194

This commit has the results of generating a new testsuite.

jit-test/wasm/proposals contains the commits of the proposals used for
generating the tests.

Depends on D65195

Attachment #9130558 - Attachment description: Bug 1617827 - Wasm: Generate new spec test-suite for jit-test. r?lth → Bug 1617827 - Wasm: Generate new spec-testsuite for jit-test. r?lth
Attachment #9130560 - Attachment description: Bug 1617827 - Wasm: Generate new spec test-suite for web-platform-test. r?lth → Bug 1617827 - Wasm: Generate new spec-testsuite for web-platform-test. r?lth
Summary: Import wasm proposal bulk memory tests → Import wasm spec-tests for proposals we implement

This is working pretty well now. I've modified my tool to now accept pinned commits and generate a config.lock file which can be used to generate tests in a deterministic manner based off a previous run. We can also exclude proposals from generating WPT's which I think will be required for features that aren't enabled by default.

There are some spec tests we'll need to exclude to land this, but I intend to investigate them all after this lands.

The two blockers left are (1) the soft freeze and (2) lando seems to dislike the size of test update patch. I think there's potential for this change to cause a test failure in beta/release builds and so I don't want to land this during the soft freeze. It's scheduled to end on Monday, so I think that waiting won't be a huge issue. However, I may have to split up the initial test update patch further as there's a lot of churn with the new tests.

Most excellent.

There an override available so that you can self-land patches directly (ie not via lando) using a=whatever, I think? https://wiki.mozilla.org/Tree_Rules. (I had to land a very large patch recently, but weirdly I managed to do so without that workaround.) It seems wrong to workaround tool restrictions with lots of patch-splitting busy-work.

Attachment #9132465 - Attachment is obsolete: true
Pushed by rhunt@eqrion.net:
https://hg.mozilla.org/integration/autoland/rev/2c328ec4ae21
Tests: Allow specifying an 'include' relative to directives.txt in jit-tests. r=lth
https://hg.mozilla.org/integration/autoland/rev/f5b812ee53b2
Wasm: Switch to 'wasm-generate-testsuite' tooling for importing spec test-suite. r=lth
https://hg.mozilla.org/integration/autoland/rev/d97d6138575b
Wasm: Generate new spec-testsuite for jit-test. r=lth
https://hg.mozilla.org/integration/autoland/rev/9b6eb4ab142b
Wasm: Generate new spec-testsuite for web-platform-test. r=lth
Blocks: 1621641
You need to log in before you can comment on or make changes to this bug.