1170305 - Improve display of bar charts visualization in compare view

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[William Lachance (:wlach)]

We currently present a colored bar if there's a regression, but we don't provide any idea of what their scale is (we just blank stuff out if we don't have any data), so it's not always obvious even what the column is for in many cases.

We should probably at least provide a faded out scale so people can have a better idea of what's being measured. Maybe something like a health meter from a video game (just going in both directions):

http://en.wikipedia.org/wiki/Health_%28gaming%29

Comment 1

[William Lachance (:wlach)]

Attachment: Screen Shot 2015-07-29 at 6.28.15 PM.png

Finally started working on this. Here's a proof of concept, let me know if the graphical bars in this comparison view are more clear than what we had before, and whether there's anything you'd like to see that would further improve them.

In addition to providing a scale, I also modified the behaviour to show a bar graph if we notice a change at all (>1% change), regardless of our confidence in it.

(note I'm CC'ing a lot of people here because I've heard lots of opinions on the bar charts and want to capture a good sample of feedback-- don't feel obligated to say anything beyond "I agree with XXX")

Comment 2

[Avi Halachmi (:avih)]

Comment on attachment 8640770 [details]
Screen Shot 2015-07-29 at 6.28.15 PM.png

1. I don't think this solves the "unknown scale" thingy since it's still not clear to me what the muted-color is (looks like 100%, but it doesn't say so anywhere).

2. Most regressions are 2-10%, and on a scale of 0-100 they may be hard to tell apart.

3. Regressions can be more than 100% (e.g. a value changes from 10 to 50 -> 400% regression), how would that show with a 0-100 range?

IMO the existing bars are OK - they make the higher regressions "pop" and puts the regressions into perspective of which is bigger than which. They're just a visual tool to complement the % delta, but I don't think they're worth putting time into making them really meaningful.

> In addition to providing a scale, I also modified the behaviour to show a bar
> graph if we notice a change at all (>1% change), regardless of our confidence
> in it.

I'm not seeing this at the screenshot. I think it's OK to display bars for all/most values, as long that it's in very muted colors. The actual regressions and improvements should clearly pop, and the cases where the confidence is low or the diff is low should get out of the way IMO.

Comment 3

[William Lachance (:wlach)]

(In reply to Avi Halachmi (:avih) from comment #2)
> Comment on attachment 8640770 [details]
> Screen Shot 2015-07-29 at 6.28.15 PM.png
> 
> 1. I don't think this solves the "unknown scale" thingy since it's still not
> clear to me what the muted-color is (looks like 100%, but it doesn't say so
> anywhere).

The delta gives a percentage right beside it. :)

> 2. Most regressions are 2-10%, and on a scale of 0-100 they may be hard to
> tell apart.

2 and 10 should be fairly obvious to someone with decent eyesight. 1 and 2% are harder to tell apart, yes, but I think that's ok. The only intention here is to give the user a hint that there's something that may be worth paying attention to.

> 3. Regressions can be more than 100% (e.g. a value changes from 10 to 50 ->
> 400% regression), how would that show with a 0-100 range?

We'd have to cap it at displaying 100%. Not a big deal IMO, first such situations are quite rare. Second, as you say the intent isn't to precisely represent the data but to provide a hint to the user about what's going on.

> IMO the existing bars are OK - they make the higher regressions "pop" and
> puts the regressions into perspective of which is bigger than which. They're
> just a visual tool to complement the % delta, but I don't think they're
> worth putting time into making them really meaningful.

The problem is that for someone who's not already intimately familiar with this view, the existing bars just look like blobs of color. For someone who's intimately familiar with the details of how this thing works, that's fine (I guess?), but I think we can provide something more intuitive without that much effort.

Comment 4

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #3)
> The delta gives a percentage right beside it. :)

Indeed, so what does the "scale" add here? The only thing it _may_ add is if you can figure the percentage by looking at the bar only (e.g. it's quarter of the full width and the full width is 100%, so a quarter is ~25%). But since it doesn't say anywhere that it's 100%, it doesn't add a thing IMO.

I'm not saying this or another approach is bad, just that IMO this specific tweak is not worth spending time on it.

Comment 5

[Mike Hommey [:glandium]]

Looks better to me. That doesn't solve the problem of knowing if variations that are not colored are good or bad, though, but I forget if that was a separate bug.

Comment 6

[Blake Winton (:bwinton) (:☕️)]

Comment on attachment 8640770 [details]
Screen Shot 2015-07-29 at 6.28.15 PM.png

Seems easier to read.  If you're not going to have a grey bar/outline from -100% to 100%, then I would suggest halving the space you take up for the graph, and have the red and green bars line up.  (They'll still be distinguishable by their colour, or for the colour-blind among us, by the location of the darker shade.  :)

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8640770 [details]
Screen Shot 2015-07-29 at 6.28.15 PM.png

avi brings up a valid point that we are trying to assume everything fits into a 0-100 scale and we don't always fit into that.  

I know if we hit 100% or more, it will be a large green or large red line, that in general is useful and the raw data can indicate what is going on.

What this change does is help show the magnitude visually of the change.  The reason we have this bar graph in the first place it to help visually pick out the areas to look at further, doing this helps add context and explanation to the data point.

I am not sure if we want to have green/red going in the same direction.  It would save real estate on the webpage, but I think it would be harder to read and understand.

Comment 8

[Blake Winton (:bwinton) (:☕️)]

(In reply to Joel Maher (:jmaher) from comment #7)
> I am not sure if we want to have green/red going in the same direction.  It
> would save real estate on the webpage, but I think it would be harder to
> read and understand.

I made a quick mockup to show what I was thinking, and to help us figure out how much harder (or not ;) it would be to read:
https://dl.dropboxusercontent.com/u/2301433/Firefox/Perfherder1.png

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

oh, that is nice.  I had assumed that we would go the same direction, but retaining the original direction of the improvement vs regression makes it more understandable.

Comment 10

[Benoit Girard (:BenWa)]

I don't have anything to add. The ideas suggested here all sound like a good improvement over the current design.

Comment 11

[William Lachance (:wlach)]

Attachment: Screen Shot 2015-08-04 at 4.45.27 PM.png

bwinton and I chatted a bit more about this on Friday and discussed maybe removing the delta column altogether and merging it with the percentage difference, by putting the value to the right/left of the bar chart depending on whether it was a regression or improvement. Tried this, not really satisfied with the results (see picture). I think I'm just going to propose a patch which goes along with his last suggestion. I think that's easy enough to interpret and there are other things I think are more important things to improve on atm.

Comment 12

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #11)
> bwinton and I chatted a bit more about this on Friday and discussed maybe
> removing the delta column altogether and merging it with the percentage
> difference, by putting the value to the right/left of the bar chart
> depending on whether it was a regression or improvement.

Main issue with that IMO is that smaller than threshold deltas disappear with this scheme. Also, if that was to be kept, I'd have argued that putting the deltas on the same side of the bar (but still with different colors) could have been faster to scan...

Comment 13

[William Lachance (:wlach)]

(In reply to Avi Halachmi (:avih) from comment #12)
> (In reply to William Lachance (:wlach) from comment #11)
> > bwinton and I chatted a bit more about this on Friday and discussed maybe
> > removing the delta column altogether and merging it with the percentage
> > difference, by putting the value to the right/left of the bar chart
> > depending on whether it was a regression or improvement.
> 
> Main issue with that IMO is that smaller than threshold deltas disappear
> with this scheme. Also, if that was to be kept, I'd have argued that putting
> the deltas on the same side of the bar (but still with different colors)
> could have been faster to scan...

Yes, at that point we're basically back to where we started though. :)

I'm pretty happy with the level of improvement with my current patch. Let's see if others agree, if so, let's get an incremental win and move onto other stuff... can always circle back to this later.

Comment 15

[William Lachance (:wlach)]

Generalizing this bug a bit.

Comment 16

[William Lachance (:wlach)]

Attachment: Proposed PR

Ok, so here's what I have. Changes:

* Show scale as proposed by :bwinton
* Fix the amount of space taken up by bar graphs so there's generally more room for the test name (as they can be rather long)
* Show bar charts for results that we're less confident in, in grey (so they're still visible)

I've uploaded an instance of treeherder/perfherder with this change enabled to stage. See for example: 

https://treeherder.allizom.org/perf.html#/compare?originalProject=try&originalRevision=fa358692aeaf&newProject=try&newRevision=54483e48d958

(ignore the fact that there are many invalid results in there -- just choosing it because it shows a nice combination of good/bad/less-confident results)

Comment 17

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #16)
> * Show bar charts for results that we're less confident in, in grey (so
> they're still visible)

This is inconsistent with this:

> https://treeherder.allizom.org/perf.html#/compare?originalProject=try&originalRevision=fa358692aeaf&newProject=try&newRevision=54483e48d958

You're showing gray bars for every delta value below the threshold (1.5%) regardless of their confidence levels (and many of the gray bars actually have high confidence in the delta), and some of the low-confidence deltas don't have bars.

IMO either show the gray bars for everything (which isn't green/red), or don't show them at all. If you're adventurous, of everything which isn't red/green, use black for deltas with high confidence and gray for deltas with med/low confidence.

Comment 18

[Joel Maher ( :jmaher ) (UTC -8)]

I don't like the new look on staging, the bars don't add anything useful to that view, and the test names are wrapped so much.  The concept of showing things that we trust are real changes vs possible changes and highlighting them differently appeals to me, it just doesn't look good in this view.  I think a 5% difference looks no different than a 2% difference, it is rare that we have >20% difference, and for the points where we have >20% (maybe 10% of the time) it is just as likely to have a point at 20% or 150%.

Comment 19

[William Lachance (:wlach)]

Comment on attachment 8643310 [details] [review]
Proposed PR

Ok went back to the drawing board on this one. 

Updated look so that we just show magnitude of regression on a scale of 0-20%. This is more similar in spirit to what we had before, where we just wanted a visual indicator that "something happened". Also added a small tooltip indicating what that means. Showing greyed our bars for all regressions of a certain size, regardless of confidence. Hopefully this will make everyone happy. :)

Before:

http://people.mozilla.org/~wlachance/Screen%20Shot%202015-08-05%20at%204.23.57%20PM.png

After:

http://people.mozilla.org/~wlachance/Screen%20Shot%202015-08-05%20at%204.20.26%20PM.png

Comment 20

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #19)
> Comment on attachment 8643310 [details] [review]
> Proposed PR

> Before:
> 
> http://people.mozilla.org/~wlachance/Screen%20Shot%202015-08-05%20at%204.23.
> 57%20PM.png
> 
> After:
> 
> http://people.mozilla.org/~wlachance/Screen%20Shot%202015-08-05%20at%204.20.
> 26%20PM.png

Slightly updated this to further reduce the size of the bar graphs to 120px. Still just as readable, but plays better on screens that are horizontally challenged.

http://people.mozilla.org/~wlachance/Screen%20Shot%202015-08-05%20at%206.18.49%20PM.png

Comment 21

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8643310 [details] [review]
Proposed PR

one comment in the PR regarding code maintainability, otherwise I think this gets us a few steps closer to a useful compare view :)

Comment 22

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/867448737a96fe7aae9c8dbced7c6a299455b828
Bug 1170305 - Improve display of bar graphs in perfherder

* Allocate a smaller, fixed space for the bar graph (gives more
  space for the test labels, which are often quite long)
* Make bar graph on a scale from 0-20% (with 20%+ regressions /
  improvements consuming the full width)
* Add a tooltip documenting that we are only showing the relative
  magnitude of the regression/improvement
* Show differences even if unsure (just show a greyed out bar chart
  for those)

Comment 23

[William Lachance (:wlach)]

Ok, gonna resolve this one.

Comment 24

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d8d34e6c301d22e2dcc6174d8271a32359e3a72a
Bug 1170305 - Fix case where some tables were formatted incorrectly

Comment 25

[Mike Hommey [:glandium]]

Okay, now that this landed and I've actually had to use results with it, the light gray colored bars, even if they have parts on the right or left depending on whether it's good or bad, still need brain processing for me. Would it be possible to make them light-green/red?