Sometimes Delta(%) seems to contradict the absolute measures
Categories
(Testing :: PerfCompare, defect, P3)
Tracking
(Not tracked)
People
(Reporter: jstutte, Assigned: aesanu)
References
Details
(Whiteboard: [pcf])
Attachments
(2 files)
See the Linux measurements in the attachment.
Updated•21 days ago
|
Comment 1•21 days ago
|
||
I think this comes from the fact we (still) incorrectly show the median and not the mean in the row, but the delta is the delta of the means.
Reporter | ||
Comment 2•21 days ago
|
||
(In reply to Julien Wajsberg [:julienw] from comment #1)
I think this comes from the fact we (still) incorrectly show the median and not the mean in the row, but the delta is the delta of the means.
Interesting.
Do we exclude outliers from that mean ? I occasionally see outliers (often on Linux, like once any 10-20 runs) that are way faster, maybe because the underlying host was idle except for that single test and able to use more turbo-boost or what not.
Comment 3•21 days ago
|
||
Ah I'm not sure.
I know there are some work to improve this in sparky's team, so I'm gonna NI sparky here because I don't know the current status.
Comment 4•21 days ago
|
||
There's some bugs open about reducing interfering processes on windows, but nothing on linux.
We don't do outlier removal during tests, and there's nothing major in progress related to outlier removal atm (we have some things we're exploring though).
Comment 6•7 days ago
|
||
The severity field is not set for this bug.
:carla, could you have a look please?
For more information, please visit BugBot documentation.
Updated•4 days ago
|
Updated•4 days ago
|
Comment 7•4 days ago
|
||
This will be fixed by https://github.com/mozilla/perfcompare/pull/798
Comment 8•4 days ago
|
||
Updated•4 days ago
|
Description
•