Closed Bug 1927493 Opened 5 days ago Closed 5 days ago

Delta and Score appear to disagree across platforms

Categories

(Testing :: PerfCompare, defect)

defect

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1926213

People

(Reporter: mgaudet, Unassigned)

Details

Using this perfCompre as an example:

This is the same benchmark, on Linux vs Windows

  • Windows Base has a higher score Windows New
  • Linux Base has a lower score than Linux New

Yet both report a negative Delta(%). One of those presumably has a sign error? Unfortunately, I can't really reverse-engineer the math on the delta percent either? E.g. for windows, the Delta(%) is reported as -0.57%, but...

  • (100*20.51/20.42)-100 = 0.4407443683
  • (100*20.42/20.51)-100 = -0.4388103364

So not sure where the sign error is here.

I think this comes from the fact that we currently show the Median score, not the Mean score, but the delta is the delta of Means. This is clearly confusing to the users.
We will change it shortly to show the Mean (like the old tool) instead of the Median. And in the future we'll gather feedback about which one is the most interesting.

Status: NEW → RESOLVED
Closed: 5 days ago
Duplicate of bug: 1926213
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.