figure out why talos tp5 responsiveness posts drastically different numbers for graph server vs perfherder

RESOLVED FIXED in Firefox 47

Status

defect
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jmaher, Unassigned)

Tracking

unspecified
mozilla47
Points:
---

Firefox Tracking Flags

(firefox47 fixed)

Details

Attachments

(2 attachments)

tp5o responsiveness was just turned on yesterday!  This is great to see, and we have more time needed to verify trends, data patterns, and if this is 100% meaningful.

In the meantime, we have much different numbers posted for graph server vs perfherder.  This means our calculations are different and we should look into why this is happening and document it or fix it.
Posted file responsiveness.txt
this is the raw responsiveness data that I see.
* perfherder: 60.60651504852021
* graphserver: 74.21

this is probably an exercise to do the math and determine what each system calculates and ensure we are doing the right thing.
Flags: needinfo?(jmaher)
graph server- using the proper repsonsiveness metric
perfherder - code was taking the mean value of the responsiveness values

pushed a fix to try server:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce22ecada746
Flags: needinfo?(jmaher)
Attachment #8718180 - Flags: review?(j.parkouss) → review+
Comment on attachment 8718180 [details]
MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss

https://reviewboard.mozilla.org/r/34453/#review31171

Hm, it's unfortunate that we can't use the construct_results method for that, it seems that it was designed to handle this case - originally. This code would need a refactoring, maybe that can be done more easily once graphserver is dropped (even though, this won't be an easy task I think).

Note that I still see differences for the cpu number - and for every value that is of type MB (even with a conversion bytes -> megabytes), but responsiveness now looks good.
I agree about how sad it is to not use construct_results.  I think when we remove graph server code, things will get much easier to cleanup and refactor, including how we handle startup tests vs pageloader tests and of course counters!
the difference you see in cpu is that the difference of a mean() vs geometric_mean().  We are avoiding more hacking on graph server :)
https://hg.mozilla.org/mozilla-central/rev/4b1c595da956
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
You need to log in before you can comment on or make changes to this bug.