Open Bug 1713878 Opened 5 years ago Updated 5 years ago

Push Health task status indicators can be misleading

Categories

(Tree Management :: Push Health, defect)

Product:
Tree Management
Component:
Push Health
Type:
defect

Tracking

(Not tracked)

People

(Reporter: ahal, Unassigned)

Details

For example:
https://treeherder.mozilla.org/push-health/push?repo=try&revision=cb9f1227ad13802e0ff14176156dde3385ea32d8&testGroup=pr&selectedTest=accessibletestsbrowserfissionbrowsernestediframejs&selectedTaskId=&selectedJobName=

In this group, it looks like browser_nested_iframe.js is perma-fail. However if you drill down into the task logs, it only failed in the second task. The other tasks had tests that failed for different reasons.

I think Push health should show the tasks where it didn't fail as green when looking under this group heading (even if they are red for another path). Or maybe it could be yellow or something to denote that it failed for a different reason.

Otherwise it is easy to misjudge the frequency of a failure.

I run into this often and for my use cases this is why I don't use push health all the time. I imagine that fixing this would help moving things into the known failures section more frequently.

I think this will be tricky to solve as I'm not sure Push Health has access to test level failure data. It should have manifest-level data at least, so we could make sure it only shows red if a test in the same manifest failed (which would still be a big improvement here).

push health should have access to the list of failure summary lines for a given job, and could determine what failed- I spent some time trying to figure it out yesterday, but after 30 minutes I wasn't making much headway; I might dig a bit more today if I am blocked on something

You need to log in before you can comment on or make changes to this bug.