1365311 - We should generate some sort of "new failure report" on wpt import

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[Boris Zbarsky [:bzbarsky]]

Specific things we'd like to know about, because they might be indications of implementation work that needs to happen:

1)  Tests that were added upstream that are failing.
2)  Tests that were modified upstream that are now failing as a result.

We've talked about this a couple of times in the past, but I'm not seeing a bug filed for it, so this is that bug.

Comment 1

[James Graham [:jgraham]]

So the plan is to move to much more frequent upstreaming and downstreaming in the near future; this work ended up bloccked on that. Apologies for not communicating this better. As part of this we intend to start filing bugs in the relevant component when we import tests that used to pass and now fail, so that's (2). For (1) we could just file a bug in the same way but it seems like it could be quite spammy where people don't specifically care about the tests yet (e.g. if Google are working on a testsuite for a feature we don't implement yet). We could do the same but only for a specified list of directories? Does that sound like what you want or would some other way of surfacing the data work better?

Comment 2

[Boris Zbarsky [:bzbarsky]]

More frequent syncs sounds great, and the plan for approaching (2) also sounds great.

For (1) my gut feeling is that we should start treating it as (2) and blacklist directories as needed, but I don't feel super-strongly about it and in particular would be quite open to changing how we handle it if we discover that there's just too much noise.