tp5o responsiveness was just turned on yesterday! This is great to see, and we have more time needed to verify trends, data patterns, and if this is 100% meaningful. In the meantime, we have much different numbers posted for graph server vs perfherder. This means our calculations are different and we should look into why this is happening and document it or fix it.
Created attachment 8716471 [details] responsiveness.txt this is the raw responsiveness data that I see. * perfherder: 60.60651504852021 * graphserver: 74.21 this is probably an exercise to do the math and determine what each system calculates and ensure we are doing the right thing.
graph server- using the proper repsonsiveness metric perfherder - code was taking the mean value of the responsiveness values pushed a fix to try server: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce22ecada746
Created attachment 8718180 [details] MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss Review commit: https://reviewboard.mozilla.org/r/34453/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/34453/
Attachment #8718180 - Flags: review?(j.parkouss)
Attachment #8718180 - Flags: review?(j.parkouss) → review+
Comment on attachment 8718180 [details] MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss https://reviewboard.mozilla.org/r/34453/#review31171 Hm, it's unfortunate that we can't use the construct_results method for that, it seems that it was designed to handle this case - originally. This code would need a refactoring, maybe that can be done more easily once graphserver is dropped (even though, this won't be an easy task I think). Note that I still see differences for the cpu number - and for every value that is of type MB (even with a conversion bytes -> megabytes), but responsiveness now looks good.
I agree about how sad it is to not use construct_results. I think when we remove graph server code, things will get much easier to cleanup and refactor, including how we handle startup tests vs pageloader tests and of course counters!
the difference you see in cpu is that the difference of a mean() vs geometric_mean(). We are avoiding more hacking on graph server :)
Status: NEW → RESOLVED
Last Resolved: 3 years ago
status-firefox47: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
You need to log in before you can comment on or make changes to this bug.