Provide a test-centric (rather than job-centric) view for results

RESOLVED DUPLICATE of bug 1337488

Status

Tree Management
Treeherder: Test-based View
P5
normal
RESOLVED DUPLICATE of bug 1337488
4 years ago
10 months ago

People

(Reporter: vaibhav1994, Unassigned)

Tracking

(Blocks: 1 bug)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
We need to figure out a way to display intermittent failures that occurs on jobs on treeherder. This feature will be useful for developers to see intermittent failures without having to scan the logs and will also help in getting a step closer to autoland.
With a test-centric (rather than job-centric) view this is trivial to display; something like

= Regressions =

     Test     |   Suite   |    Expected   |   Actual   |   Message   |
[...]

= Intermittents =

     Test     |    Suite    |    Expected   |   Actual   |   Message   | Failure Rate |
---------------------------------------------------------------------------------------
 My new test  |  Mochitests |     PASS      |  PASS(8)   |             |     2/10     |
              |             |               |  FAIL(1)   |             |              |
              |             |               |  CRASH(1)  |             |              |

Of course it would *also* be useful to have a test-centric view where we can see all the results for a given test.
Priority: -- → P4
Priority: P4 → P5
Summary: Display results of intermittent failures on treeherder → Provide a test-centric (rather than job-centric) view for results

Comment 2

3 years ago
Another idea for how this could look:
https://bug983017.bugzilla.mozilla.org/attachment.cgi?id=8392514
(In reply to Jesse Ruderman from comment #2)
> Another idea for how this could look:
> https://bug983017.bugzilla.mozilla.org/attachment.cgi?id=8392514

I like that idea as an end future state - though it requires many many more pieces in place before we could implement it. eg: overhauling the way we identify intermittents so we have a chance of having only one suggestion per failure line, so we can actually auto-identify.
Duplicate of this bug: 1196702
Created attachment 8763216 [details]
Sketch

As another idea, perhaps we could have a view like this, where the pushlog exists like it does right now, but instead of showing individual jobs per-platform, we could have jobs grouped by job type (build, mochitest, mochitest-e10s, etc) in some sort of bar graph, split by the number of jobs in each state (pending, running, completed) with completed jobs split further by passing, failing, busted, retried.
When you click on a section of one of those graphs, a new panel opens at the bottom of the page that lists all of individual test names that have failed in this job type with a count of how many times that failure has happened in the push. Below each failing testname, we could list each individual failing instance of that test with the rest of the failure line from that individual failure.


Talking with Ed, we probably already have all of the information needed to generate these bar graphs, though getting the breakdown of test failures will probably need a lot of new work.
Forward duping to bug 1337488 which has more recent activity.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1337488
Component: Treeherder → Treeherder: Test-based View
You need to log in before you can comment on or make changes to this bug.