figure out why talos tp5 responsiveness posts drastically different numbers for graph server vs perfherder

RESOLVED FIXED in Firefox 47

Status

Testing
Talos
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jmaher, Unassigned)

Tracking

unspecified
mozilla47
Points:
---

Firefox Tracking Flags

(firefox47 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

(Reporter)

Description

2 years ago
tp5o responsiveness was just turned on yesterday!  This is great to see, and we have more time needed to verify trends, data patterns, and if this is 100% meaningful.

In the meantime, we have much different numbers posted for graph server vs perfherder.  This means our calculations are different and we should look into why this is happening and document it or fix it.
(Reporter)

Comment 1

2 years ago
Created attachment 8716471 [details]
responsiveness.txt

this is the raw responsiveness data that I see.
* perfherder: 60.60651504852021
* graphserver: 74.21

this is probably an exercise to do the math and determine what each system calculates and ensure we are doing the right thing.
Flags: needinfo?(jmaher)
(Reporter)

Comment 2

2 years ago
graph server- using the proper repsonsiveness metric
perfherder - code was taking the mean value of the responsiveness values

pushed a fix to try server:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ce22ecada746
Flags: needinfo?(jmaher)
(Reporter)

Comment 3

2 years ago
Created attachment 8718180 [details]
MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss

Review commit: https://reviewboard.mozilla.org/r/34453/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34453/
Attachment #8718180 - Flags: review?(j.parkouss)
Attachment #8718180 - Flags: review?(j.parkouss) → review+
Comment on attachment 8718180 [details]
MozReview Request: Bug 1246174 - fix tp5o responsiveness calculation for reporting to perfherder. r?parkouss

https://reviewboard.mozilla.org/r/34453/#review31171

Hm, it's unfortunate that we can't use the construct_results method for that, it seems that it was designed to handle this case - originally. This code would need a refactoring, maybe that can be done more easily once graphserver is dropped (even though, this won't be an easy task I think).

Note that I still see differences for the cpu number - and for every value that is of type MB (even with a conversion bytes -> megabytes), but responsiveness now looks good.
(Reporter)

Comment 5

2 years ago
I agree about how sad it is to not use construct_results.  I think when we remove graph server code, things will get much easier to cleanup and refactor, including how we handle startup tests vs pageloader tests and of course counters!
(Reporter)

Comment 6

2 years ago
the difference you see in cpu is that the difference of a mean() vs geometric_mean().  We are avoiding more hacking on graph server :)

Comment 7

2 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/4b1c595da956

Comment 8

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/4b1c595da956
Status: NEW → RESOLVED
Last Resolved: 2 years ago
status-firefox47: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
You need to log in before you can comment on or make changes to this bug.