Open Bug 1500528 Opened 2 years ago Updated 9 months ago

Make perfherder comparisons consistent across benchmarks

Categories

(Tree Management :: Perfherder, enhancement, P3)

enhancement

Tracking

(Not tracked)

People

(Reporter: mgaudet, Unassigned)

Details

(Keywords: good-first-bug)

Looking at the compare view for a try push [1] it is really irksome that a negative delta means both "benchmark got worse" and "benchmark got better" on the same page

To see what I mean, look at  the hover tooltip over Dromaeo (negative delta is bad) and tp6 amazon (negative delta is good!). 

The columns should have a consistent meaning across all rows. 

[1]: https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-central&newProject=try&newRevision=ac631e7771a1&framework=1&showOnlyComparable=1&selectedTimeRange=172800

Yes, I see your point. For Dromaeo higher values are better, while for Kraken the lower ones are.
We should revisit the way we're presenting this data.

From what I know, values with high confidence get highlighted in either red (bad) or green (good).
Low confidence values don't have that benefit right from the start.

My proposal: remove the "-" minus prefix from the Delta column. Then provide a dimmed colored background of red/green around that delta for the low confidence values.

Keywords: good-first-bug
Priority: -- → P3
Blocks: 1520720

Dave, do you have another opinion or require any other info on this?

Flags: needinfo?(dave.hunt)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #2)

Dave, do you have another opinion or require any other info on this?

Comment 0 mentioned dromeo and tp6 amazon, however I only see dromeo and kraken in the linked comparison view. I think the "-" on the delta is fine, but I would agree that having a clear indication of if the change indicates an improvement for the specific test based on whether higher is better or worse. Colour would be a good option but I think we should also include text or an icon to indicate if the change is an improvement. Perhaps a thumbs up/down?

Flags: needinfo?(dave.hunt)

(I really like the thumbs up / warning icons I've seen now!)

Hey, I am new to this project and interested to work on this issue. Can someone help me to get started, I will be really thankful if someone does.

Hi Farhaan, you'll need to look at our docs to get started: https://treeherder.readthedocs.io/installation.html
You'll only need to concern yourself with UI development for this bug. Please also read through the Testing and Code Style sections. If you have questions about this bug Ionut (igoldan) is a good person to ask. You can check the "request information" box (below) to be sure he see's your question.

Type: defect → enhancement
No longer blocks: 1520720

(In reply to Matthew Gaudet (he/him) [:mgaudet] from comment #4)

(I really like the thumbs up / warning icons I've seen now!)

So can we close this bug as fixed, then?

Flags: needinfo?(gaudet)
Flags: needinfo?(gaudet) → needinfo?(mgaudet)

As a casual user of Perfherder, I think I'd still prefer it if the deltas were uniformly computed.

However, I'm not a day-to-day user, so I'd be willing to grant that perhaps it's correct that somethings have negative deltas meaning improvements, and other things have positive deltas meaning improvement.

The icons help quite a lot as a casual user though.

Flags: needinfo?(mgaudet)
You need to log in before you can comment on or make changes to this bug.