736872 - Expose dev.tree-management per-cset expected Talos data as a REST service

Status: RESOLVED INCOMPLETE

(Webtools Graveyard :: Graph Server, defect)

Description

[Justin Lebar (not reading bugmail)]

It would be really useful, I think, if one could query the data dev.tree-management uses to calculate regressions.

We could use this, for example, to integrate compare-talos into TBPL.  TBPL would query this service to learn the expected range for Talos based on the qparent of the try push, and then color the Talos runs accordingly.

I'd imagine the web service would look something like this:

 Input:
   hg qparent rev
   My Talos results (potentially more than one run for each test)

 Output:
   For each test & platform in input, output expected mean, variance
   For each test & platform in input, output whether the input result is significantly different from the expected result.

Comment 1

[Robert Helmer [:rhelmer]]

I think catlee wrote the tool that does this, not sure where the code lives and where it's running right now.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I like this bug.  Are we assuming that the parent has finished testing before the current test is done?

Could we have input as:
branch
platform
test

output could be (from the last 7 runs):
expected range 
expected value
expected variance

who would do the comparison, talos or buildbot?  Talos already talks to the graph server to upload results with all the input information, it would be nice if the graph server could spit back a return code and we could take it from there.

Comment 3

[Justin Lebar (not reading bugmail)]

> Are we assuming that the parent has finished testing before the current test is done?

If not, there's not much we can do, right?

> Could we have input as:
> branch
> platform
> test

This would be a lot more queries per run of the Talos suite, and IME, the fewer queries we make to graphserver, the faster we get back our results.  If you think we can make each query very fast, then I'm happy to break it up like this.

> who would do the comparison, talos or buildbot?

Yes, I think the REST API should do the comparison, because it's a non-trivial statistical test.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I believe we would have to query this for every test that finishes.  For example linux tests finish much faster than windows tests usually because the builds are completed much faster.  If we are going to be doing close to ~80 queries / push, then it makes more sense to do this inside of talos.  Maybe I have overlooked something, just trying to understand this fully.

Comment 5

[Justin Lebar (not reading bugmail)]

> I believe we would have to query this for every test that finishes.

I guess that's true, for the TBPL case.

> If we are going to be doing close to ~80 queries / push, then it makes more sense to do this 
> inside of talos.

What do you mean by this, exactly?