Open Bug 1926213 Opened 21 days ago Updated 4 days ago

Sometimes Delta(%) seems to contradict the absolute measures

Categories

(Testing :: PerfCompare, defect, P3)

defect

Tracking

(Not tracked)

People

(Reporter: jstutte, Assigned: aesanu)

References

Details

(Whiteboard: [pcf])

Attachments

(2 files)

Attached image Weird delta.png

See the Linux measurements in the attachment.

This is the comparison.

I think this comes from the fact we (still) incorrectly show the median and not the mean in the row, but the delta is the delta of the means.

(In reply to Julien Wajsberg [:julienw] from comment #1)

I think this comes from the fact we (still) incorrectly show the median and not the mean in the row, but the delta is the delta of the means.

Interesting.
Do we exclude outliers from that mean ? I occasionally see outliers (often on Linux, like once any 10-20 runs) that are way faster, maybe because the underlying host was idle except for that single test and able to use more turbo-boost or what not.

Ah I'm not sure.
I know there are some work to improve this in sparky's team, so I'm gonna NI sparky here because I don't know the current status.

Flags: needinfo?(gmierz2)

There's some bugs open about reducing interfering processes on windows, but nothing on linux.

We don't do outlier removal during tests, and there's nothing major in progress related to outlier removal atm (we have some things we're exploring though).

Flags: needinfo?(gmierz2)
Duplicate of this bug: 1927493

The severity field is not set for this bug.
:carla, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(csevere)
Severity: -- → S3
Priority: -- → P3
Flags: needinfo?(csevere)
Attached file GitHub Pull Request
Assignee: nobody → felash
Assignee: felash → aesanu
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: