For a failing test, it would be good to know how many times the same test also passed on the same Push. Treeherder can not hold this amount of data, so we are hopeful ActiveData could be leveraged this way.
I guess you are concerned about retriggers? This usually happens on ?try?. Do you have an example?
Yeah, this came out of a conversation with Joel Maher (cc'd) wrt searching for intermittent ratios in the TestGroup UI. He'd like to be able to see, for a given test, how many times it passed and failed in that push. As I recall being able to see this ratio not just for the push in question, but for a certain number of pushes prior would be ideal. Is this something, given a revision and a test name (path), that we could query for in Active Data? It would be great if I could just build a link to your UI to get this data.
Yes, I can give you an example query. Problem is most pushes run a test just once, if at all. Test results from try pushes are not included in ActiveData right now. So, you may need some historical data instead, which I can also provide, given a branch, pushdate, and a test name.
OK, cool. I think that's fine. It will just be useful in the event there ARE multiple runs of the same test. I'll get back to you when I start tackling this. Thanks!