[TestGroup UI] Explore if ActiveData can be used to get pass/fail ratios on a per-test basis

NEW
Assigned to

Status

Tree Management
Treeherder
22 days ago
2 days ago

People

(Reporter: camd, Assigned: camd)

Tracking

(Blocks: 1 bug)

Details

(Assignee)

Description

22 days ago
For a failing test, it would be good to know how many times the same test also passed on the same Push.  Treeherder can not hold this amount of data, so we are hopeful ActiveData could be leveraged this way.
(Assignee)

Updated

22 days ago
Assignee: nobody → cdawson
(Assignee)

Updated

22 days ago
Blocks: 1337488
I guess you are concerned about retriggers?  This usually happens on ?try?.  Do you have an example?
(Assignee)

Comment 2

3 days ago
Yeah, this came out of a conversation with Joel Maher (cc'd) wrt searching for intermittent ratios in the TestGroup UI.  He'd like to be able to see, for a given test, how many times it passed and failed in that push.  As I recall being able to see this ratio not just for the push in question, but for a certain number of pushes prior would be ideal.  

Is this something, given a revision and a test name (path), that we could query for in Active Data?  It would be great if I could just build a link to your UI to get this data.
Flags: needinfo?(klahnakoski)
Yes, I can give you an example query. Problem is most pushes run a test just once, if at all. Test results from try pushes are not included in ActiveData right now.  So, you may need some historical data instead, which I can also provide, given a branch, pushdate, and a test name.
Flags: needinfo?(klahnakoski)
(Assignee)

Comment 4

2 days ago
OK, cool.  I think that's fine.  It will just be useful in the event there ARE multiple runs of the same test.  I'll get back to you when I start tackling this.  Thanks!
You need to log in before you can comment on or make changes to this bug.