1600247 - Provide a test harness to run web-platform-tests with minimal run-time dependencies

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[Simon Giesecke [:sg] [he/him]]

The test harness for web-platform-tests run via mach wpt has a lot of runtime dependencies.

This has several drawbacks, including

• the set-up and tear-down time for running wpt tests is quite high (on my desktop machine, it is ca. 30 seconds set-up and 6 seconds tear-down for a partly optimized debug build; compared to 0.4 seconds set-up and no measurable tear-down time for xpcshell-test, on my notebook it is significantly longer, it is even longer when running in a debugger)
• during debugging, it may be hard to differentiate calls to the subject code from the actual test vs. other components that are running (e.g. it may require extra work to set breakpoint conditions, or be practically impossible)
• the actual tests may not even run when the changes to be tested cause a fatal error or crash during set-up, which may regular happen during development. Typically, I want to know which tests are still running and which are not to get a quick clue what might be the cause of a problem. It is impossible to get this information when the test harness prevents the tests from being run.

Specifically for IndexedDB, the problem consists of several uses of IndexedDB by the run-time dependencies, specifically remote-settings and ActivityStream. However, similar issues probably exist for other components.

The xpcshell test harness has reduced reduced dependencies. It would be really helpful if a similar shim existed for running web-platform-tests, as an alternative to the existing harness, which remains useful.