1059770 - Provide a test-centric (rather than job-centric) view for results

Status: RESOLVED DUPLICATE of bug 1337488

(Tree Management Graveyard :: Treeherder: Test-based View, defect, P5)

Description

[Vaibhav (:vaibhav1994)]

We need to figure out a way to display intermittent failures that occurs on jobs on treeherder. This feature will be useful for developers to see intermittent failures without having to scan the logs and will also help in getting a step closer to autoland.

Comment 1

[James Graham [:jgraham]]

With a test-centric (rather than job-centric) view this is trivial to display; something like

= Regressions =

     Test     |   Suite   |    Expected   |   Actual   |   Message   |
[...]

= Intermittents =

     Test     |    Suite    |    Expected   |   Actual   |   Message   | Failure Rate |
---------------------------------------------------------------------------------------
 My new test  |  Mochitests |     PASS      |  PASS(8)   |             |     2/10     |
              |             |               |  FAIL(1)   |             |              |
              |             |               |  CRASH(1)  |             |              |

Of course it would *also* be useful to have a test-centric view where we can see all the results for a given test.

Comment 2

[Jesse Ruderman]

Another idea for how this could look:
https://bug983017.bugzilla.mozilla.org/attachment.cgi?id=8392514

Comment 3

[Ed Morley [:emorley]]

(In reply to Jesse Ruderman from comment #2)
> Another idea for how this could look:
> https://bug983017.bugzilla.mozilla.org/attachment.cgi?id=8392514

I like that idea as an end future state - though it requires many many more pieces in place before we could implement it. eg: overhauling the way we identify intermittents so we have a chance of having only one suggestion per failure line, so we can actually auto-identify.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Attachment: Sketch

As another idea, perhaps we could have a view like this, where the pushlog exists like it does right now, but instead of showing individual jobs per-platform, we could have jobs grouped by job type (build, mochitest, mochitest-e10s, etc) in some sort of bar graph, split by the number of jobs in each state (pending, running, completed) with completed jobs split further by passing, failing, busted, retried.
When you click on a section of one of those graphs, a new panel opens at the bottom of the page that lists all of individual test names that have failed in this job type with a count of how many times that failure has happened in the push. Below each failing testname, we could list each individual failing instance of that test with the rest of the failure line from that individual failure.


Talking with Ed, we probably already have all of the information needed to generate these bar graphs, though getting the breakdown of test failures will probably need a lot of new work.

Comment 6

[Ed Morley [:emorley]]

Forward duping to bug 1337488 which has more recent activity.