1246174 - figure out why talos tp5 responsiveness posts drastically different numbers for graph server vs perfherder

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

tp5o responsiveness was just turned on yesterday!  This is great to see, and we have more time needed to verify trends, data patterns, and if this is 100% meaningful.

In the meantime, we have much different numbers posted for graph server vs perfherder.  This means our calculations are different and we should look into why this is happening and document it or fix it.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: responsiveness.txt

this is the raw responsiveness data that I see.
* perfherder: 60.60651504852021
* graphserver: 74.21

this is probably an exercise to do the math and determine what each system calculates and ensure we are doing the right thing.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

graph server- using the proper repsonsiveness metric
perfherder - code was taking the mean value of the responsiveness values

pushed a fix to try server:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce22ecada746

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss

Review commit: https://reviewboard.mozilla.org/r/34453/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34453/

Comment 4

[Julien Pagès (:parkouss)]

Comment on attachment 8718180 [details]
MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss

https://reviewboard.mozilla.org/r/34453/#review31171

Hm, it's unfortunate that we can't use the construct_results method for that, it seems that it was designed to handle this case - originally. This code would need a refactoring, maybe that can be done more easily once graphserver is dropped (even though, this won't be an easy task I think).

Note that I still see differences for the cpu number - and for every value that is of type MB (even with a conversion bytes -> megabytes), but responsiveness now looks good.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I agree about how sad it is to not use construct_results.  I think when we remove graph server code, things will get much easier to cleanup and refactor, including how we handle startup tests vs pageloader tests and of course counters!

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

the difference you see in cpu is that the difference of a mean() vs geometric_mean().  We are avoiding more hacking on graph server :)

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4b1c595da956

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4b1c595da956