Open Bug 1300819 Opened 3 years ago Updated 3 years ago

Notification of changes in expectation values for tests in wpt

Categories

(Testing :: web-platform-tests, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: jgraham, Unassigned)

Details

At the moment it's possible for us to miss important changes particularly when we update web-platform-tests from upstream. For example the following situation can occur:

1) We implement feature X per spec, and pass the tests
2) The spec for X changes
3) Someone updates the tests for X to match the new spec
4) We import the new tests and auto-mark them as expected-fail

Some additional steps are required here:

5) We notify the developer of X that there are some new failures in those tests.
6) We fix the implementation to match the spec and other implemetations of X.

This bug covers step 5 in the above example.

https://github.com/jgraham/wptreport has code for reading the structured log output of tests and dumping the results into a database. We could use this as a basis for the following procedure:

1) For each push to m-c store all the wpt results in a database
2) Provide a UI for subscribing to changes in the results in particular directories
3) For each m-c push, check for regression-type changes to the expectation data and if these are seen notify relevant developers

This is not the only way to achieve the goal here. For example we could run a script against the metadata files to look for expectation changes. But there would still need to be some way of determining who wanted to know about the changes, parsing the expectation files means working with the weird custom format, and looking at the results directly seems more obviously extensible to other problems (e.g. cross-browser comparisons) than looking at the expectation data.
You need to log in before you can comment on or make changes to this bug.