1383515 - web-platform-test expected crashes don't show nearly enough indications that they are expected

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1383469 got discovered on Saturday, but the crash can already be found in the logs starting Thursday with the web-platform-tests update:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=3c96d3de9c7c7eea1692c1bdbca882644a587660&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=runnable&filter-resultStatus=usercancel&filter-resultStatus=success&filter-searchStr=f55c261883b0f53a23f5bd54af7a1f7a13674613&tochange=4eb7f7fbdd22f080309a9b95c38ed5bcd5e3fc3e

There are also other crashes whose runs are reported as passing/green, e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=115942249&repo=mozilla-inbound

Comment 1

[Ed Morley [:emorley]]

Treeherder just displays the job status provided to it by buildbot/taskcluster, who normally rely solely on the exit code of the harness. As such, this isn't a bug in Treeherder, so moving to web-platform tests.

Comment 2

[James Graham [:jgraham]]

I mean this is by design. web-platform-tests just records the status on import and tells you if that changes. Any other setup wouldn't allow us to import tests at all because we would be blocked on fixing issues (or just disable the tests, which is worse). We are working on importing in smaller batches, which will also allow creating bugs to inform people of the status when tests are imported and don't pass.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Ok, that makes it difficult to see why a test run got set as orange, e.g. if the first error will always be in the log. Fortunately, that's rare at the moment.

Comment 4

[Phil Ringnalda (:philor)]

Well, no, not particularly rare. Consider, for instance, that Mac Stylo wpt10 has been permaorange since last Thursday because bug 1386604 landed a bunch of webrtc tests that we totally fail, but we totally fail them differently on Mac Stylo than we fail them elsewhere, and the annotations failed to reflect that, but that webrtc permaorange has been starred as the completely expected crash bug 1387684 ever since. And after noticing that, I discovered yet another one (which I've since lost track of) where every instance, 32 in the last week, was actually another unfiled failure.