It would be really useful, I think, if one could query the data dev.tree-management uses to calculate regressions. We could use this, for example, to integrate compare-talos into TBPL. TBPL would query this service to learn the expected range for Talos based on the qparent of the try push, and then color the Talos runs accordingly. I'd imagine the web service would look something like this: Input: hg qparent rev My Talos results (potentially more than one run for each test) Output: For each test & platform in input, output expected mean, variance For each test & platform in input, output whether the input result is significantly different from the expected result.
I think catlee wrote the tool that does this, not sure where the code lives and where it's running right now.
I like this bug. Are we assuming that the parent has finished testing before the current test is done? Could we have input as: branch platform test output could be (from the last 7 runs): expected range expected value expected variance who would do the comparison, talos or buildbot? Talos already talks to the graph server to upload results with all the input information, it would be nice if the graph server could spit back a return code and we could take it from there.
> Are we assuming that the parent has finished testing before the current test is done? If not, there's not much we can do, right? > Could we have input as: > branch > platform > test This would be a lot more queries per run of the Talos suite, and IME, the fewer queries we make to graphserver, the faster we get back our results. If you think we can make each query very fast, then I'm happy to break it up like this. > who would do the comparison, talos or buildbot? Yes, I think the REST API should do the comparison, because it's a non-trivial statistical test.
I believe we would have to query this for every test that finishes. For example linux tests finish much faster than windows tests usually because the builds are completed much faster. If we are going to be doing close to ~80 queries / push, then it makes more sense to do this inside of talos. Maybe I have overlooked something, just trying to understand this fully.
> I believe we would have to query this for every test that finishes. I guess that's true, for the TBPL case. > If we are going to be doing close to ~80 queries / push, then it makes more sense to do this > inside of talos. What do you mean by this, exactly?
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.