Closed Bug 1059770 Opened 7 years ago Closed 5 years ago

Provide a test-centric (rather than job-centric) view for results

Categories

(Tree Management Graveyard :: Treeherder: Test-based View, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1337488

People

(Reporter: vaibhav1994, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

We need to figure out a way to display intermittent failures that occurs on jobs on treeherder. This feature will be useful for developers to see intermittent failures without having to scan the logs and will also help in getting a step closer to autoland.
With a test-centric (rather than job-centric) view this is trivial to display; something like

= Regressions =

     Test     |   Suite   |    Expected   |   Actual   |   Message   |
[...]

= Intermittents =

     Test     |    Suite    |    Expected   |   Actual   |   Message   | Failure Rate |
---------------------------------------------------------------------------------------
 My new test  |  Mochitests |     PASS      |  PASS(8)   |             |     2/10     |
              |             |               |  FAIL(1)   |             |              |
              |             |               |  CRASH(1)  |             |              |

Of course it would *also* be useful to have a test-centric view where we can see all the results for a given test.
Priority: -- → P4
Priority: P4 → P5
Summary: Display results of intermittent failures on treeherder → Provide a test-centric (rather than job-centric) view for results
(In reply to Jesse Ruderman from comment #2)
> Another idea for how this could look:
> https://bug983017.bugzilla.mozilla.org/attachment.cgi?id=8392514

I like that idea as an end future state - though it requires many many more pieces in place before we could implement it. eg: overhauling the way we identify intermittents so we have a chance of having only one suggestion per failure line, so we can actually auto-identify.
Duplicate of this bug: 1196702
Attached image Sketch
As another idea, perhaps we could have a view like this, where the pushlog exists like it does right now, but instead of showing individual jobs per-platform, we could have jobs grouped by job type (build, mochitest, mochitest-e10s, etc) in some sort of bar graph, split by the number of jobs in each state (pending, running, completed) with completed jobs split further by passing, failing, busted, retried.
When you click on a section of one of those graphs, a new panel opens at the bottom of the page that lists all of individual test names that have failed in this job type with a count of how many times that failure has happened in the push. Below each failing testname, we could list each individual failing instance of that test with the rest of the failure line from that individual failure.


Talking with Ed, we probably already have all of the information needed to generate these bar graphs, though getting the breakdown of test failures will probably need a lot of new work.
See Also: → 1301795
Forward duping to bug 1337488 which has more recent activity.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1337488
Component: Treeherder → Treeherder: Test-based View
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.