1164891 - perfherder compare view shows averaged numbers + stddev over runs, we would like to visualize the distribution

Status: RESOLVED FIXED

(Tree Management :: Perfherder, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

for tests with a good amount of noise compared to regressions (ts_paint, paint, sessionrestore), it is really hard to determine if you have a regression or not, despite pushing two changes to try and doing a lot of retriggers.

One thing that will help is a way to visualize the 'range' of numbers reported from each time the test was run.  To make this useful having it side by side to see how much overlap and shift in the range there is.

This could be specific for a given test/platform (signature in the backend of perfherder) and either show up in the subtest view, or have a different link/mouseover with a popup that shows this distribution side by side.

Comment 1

[William Lachance (:wlach)]

Attachment: fivethirtyeight's comparison between candidates

I really like the way fivethirtyeight does this (see attachment). It's for polling results, but we could easily use the same style of visualization for perfherder, with some modifications. 

I think we could do something like this in a popover (http://angular-ui.github.io/bootstrap/#/popover), probably when the user hovers over delta in the compare view. See this for example:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=9915ff6e7bf7&newProject=mozilla-inbound&newRevision=952eae508967&framework=1&showOnlyImportant=0

In this case, we would just display two sets of bar charts on top of each other. The x axis would be meaningless. The y axis would be the actual results from each of the replicates. To make it easier to compare results, sort the results by value (Y) from least to highest, and make sure to use the same scale for both the left and right graphs.

You can see an example of using flot (the JS charting we're library we're using) to create bar charts here:

http://people.mozilla.org/~wlachance/eideticker-startup-test/20-results/camera-startup-20/

(obviously the charts would be a lot smaller in our implementation, I could also be convinced not to use flot, probably)

Obviously this would be a front end change. I believe most of the changes you'd need to make would be in:

ui/directives/perf/compare.js
ui/js/controller/perf/compare.js
ui/partials/perf/comparetable.html

Comment 2

[Avi Halachmi (:avih)]

Not sure I understand the intended visualization.

At your sample attachment, the x-axis is the value, and the y is the density of results for each x, which hopefully has normal distribution and hence the x-value of the tip of the curve is the projected result.

But we typically have much less values, and probably not enough to decide whether or not the distribution is normal, and not enough values to determine density either.

Also, from your explanation I can imagine a one dimensional chart[s], but not sure how that translate into a bar-chart or anything else 2-d.

For instance, if we have 5 results for rev A, and 5 results for rev B, then I'd imagine something like this could be useful (or set it vertically to match the orientation of the values on the graph):

A [10]------------X-XX---X--X--------[20]
                  [16 +/- 2]
                  
B [10]------X--X-----XX----X---------[20]
              [14 +/- 3]

I'd appreciate if you could explain your intended visualization again.

Comment 3

[William Lachance (:wlach)]

I was thinking of just having two bar charts, stacked on top of each other. I wasn't really thinking in terms of visualizing the distribution, per se. I think I actually like your approach a lot better though, let's do that instead -- it's more space efficient and I think more intuitive. I'm pretty sure we could hack up flot to do that style of visualization fairly easily.

Comment 4

[Vibhor Sehgal]

Hi,I'd like to work on this issue. I have previously worked with Joel during the Summer of Contribution and would like to keep contributing this summer too. The comments above highlight what needs to be done but I'd still request someone to guide me through this. I have knowledge of Angular JS and Django , though I want to improve it. Looking forward to work on this . Thanks !

Comment 5

[Avi Halachmi (:avih)]

(In reply to William Lachance (:wlach) from comment #3)
> let's do that instead ...

Well.. if you take this approach, try to see if you can account for overlapping values somehow. At the graph when Joel retriggers - it's not always easy to notice if two runs end up with the same/near-enough value (no biggie though).

Comment 6

[Vibhor Sehgal]

(In reply to Avi Halachmi (:avih) from comment #5)
> (In reply to William Lachance (:wlach) from comment #3)
> > let's do that instead ...
> 
> Well.. if you take this approach, try to see if you can account for
> overlapping values somehow. At the graph when Joel retriggers - it's not
> always easy to notice if two runs end up with the same/near-enough value (no
> biggie though).

How's this :
http://www.flotcharts.org/flot/examples/stacking/index.html ( Select the 'Without Stacking' and 'Lines' option)

Comment 7

[Avi Halachmi (:avih)]

(In reply to Vibhor Sehgal from comment #6)
> How's this :
> http://www.flotcharts.org/flot/examples/stacking/index.html ( Select the
> 'Without Stacking' and 'Lines' option)

Not sure I get it. What are the X and Y values, and what do the 3 colors of each bar mean?

Comment 8

[William Lachance (:wlach)]

Hi Vibhor! Great that you're interested in working on this, I've assigned the bug to you. However, I don't think the stacking plugin really matches what we're trying to do here. I suspect this error bars example is probably a better example to work from if you want to reproduce something like avi's visualization: http://www.flotcharts.org/flot/examples/series-errorbars/index.html
It would also be ok to drop down to using canvas for this case, though I think it would be easier using flot.

Comment 9

[Vibhor Sehgal]

(In reply to William Lachance (:wlach) from comment #8)
> Hi Vibhor! Great that you're interested in working on this, I've assigned
> the bug to you. However, I don't think the stacking plugin really matches
> what we're trying to do here. I suspect this error bars example is probably
> a better example to work from if you want to reproduce something like avi's
> visualization:
> http://www.flotcharts.org/flot/examples/series-errorbars/index.html
> It would also be ok to drop down to using canvas for this case, though I
> think it would be easier using flot.

Hi, Thanks !
Before I proceed, Can you please elaborate on what exactly is required? What the input data is ? What exactly is Delta and the Standard deviation? I'll try using Flot only.
Thanks.

Comment 10

[William Lachance (:wlach)]

So "delta" means "difference". If we have two means for example, 4.5 (base) and 5.5 (new), the delta would be 1. Standard deviation would mean the difference between the components that make up the mean. It's a standard concept in stats which you can look up in Wikipedia: http://en.wikipedia.org/wiki/Standard_deviation

The input would be the different means for the seperate test runs, try hovering over the base/new numbers in the example I previously gave:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=mozilla-inbound&originalRevision=9915ff6e7bf7&newProject=mozilla-inbound&newRevision=952eae508967&framework=1&showOnlyImportant=0

Comment 11

[William Lachance (:wlach)]

:Pike did up a visualization using perfherder data similar to what was described in this bug:

https://pike.github.io/talos-compare/?revision=5b91982f5583&revision=9c8319a80059&revision=035ec6b3cbe4&revision=3728d4d4866d

Comment 12

[Axel Hecht [:Pike]]

Oy.

A bit of detail on this visualiation:

This is for talos runs with --rebuild 20, so we actually have 21 talos runs. We're not able to dig in to the individual results of a single talos run, so the numbers are still not as raw as I personally would like them to be.

More on to the design of the view:

I use one scale for all measurements, but with different offsets. That means that a ms is some amount of pixels across the view.

I used some opacity to visualize densitiy of test results. This is visually hard, and different folks have different attitude towards the black background.

I used the first six characters of the revision hash to color the graph. That's interesting as you can use different links to graphs, and colors between graphs would match. It's also makes the question about contrast harder, 'cause you don't control the color range.

I like that we can show/hide individual platforms. Makes it a lot easier to rationalize, as different platforms are differently noisy.

I focused on absolute time and not %. That's probably something that needs to align with organizational goals.

My use of d3 in this view is rather silly, tbh. It would be nice if it would animate the scales if one changes the selection of platforms, for example.

I did entertain the idea of doing a underlay of the graph that indicated median and error margin, though what I'd be really interested in is "is this result noisy, or are there things that push stuff out of bounds".

CCing gandalf, as he's looking in to actually generating a view of a subset of our results that just show the full raw data as it's in PERFHERDER_DATA in the logs.

Comment 13

[William Lachance (:wlach)]

Resetting assignee (Vibhor, let me know if you want to work on this, assuming you don't have time) and adding dependant bug.

Comment 14

[Mukul Jain]

I would like to take this as my GSOC project, I have worked with graphs(using d3.js) and angular(1.0) before to create comparison tool for an Indian Startup, so this will be good for me, I will use angular 2 for this one.

Before moving I would like to know what exact we have to represent? 

From the above conservation this what I got:

1. Input will be mean of different separate test runs.
2. In the given link (https://pike.github.io/talos-compare/?revision=5b91982f5583&revision=9c8319a80059&revision=035ec6b3cbe4&revision=3728d4d4866d) some parameters are compared for a different test.

things are still puzzled to me, please could you elaborate on more

1. What we are comparing
2. What things need to be compared.
3. On what basis comparison is done.
4. what are the endpoints for input data or we have to create?

Comment 15

[William Lachance (:wlach)]

This bug morphed into something else, sorry about the confusion everyone. I filed a new bug 1350384, to deal with the question of how to deal with replicates. This bug (as originally filed) is just about getting an easier-to-understand view of the existing test/subtest summaries that perfherder already displays. Let's keep its scope there (most of the original discussion is still valid, it's only in the last few points that it went off the rails). :)

Comment 16

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1164891-canvas > mozilla:master

Comment 17

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

I have rendered the graphs to the tooltip. Please take a look and let me know if any changes have to be made.

I'm not able to import numeral.js correctly. What changes should I make to do so?

Comment 18

[William Lachance (:wlach)]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

Cancelling feedback for now.

Comment 19

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

Now I have used HTML table to create the tabular part of the table and to write the minValue and maxValue. The distribution is being rendered by Canvas API.

I have found a solution for the problem wherein sometimes the visualisation didn't render. As discussed on the call, it was due to the fact that the "id" of the canvas element was kept same. To solve this issue, I have changed the "id" after creating the context. This seems to work.

Please take a look at it and let me know if any more changes are required.

Comment 20

[William Lachance (:wlach)]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

Left some comments in the PR

Comment 21

[Robert Wood [:rwood]]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

:wlach already had a look, don't think my feedback is also required

Comment 22

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

I have made the changes which you'd asked for in the PR. Hope it's good now.

Comment 23

[William Lachance (:wlach)]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

Just a few minor nits left from my end. Joel, do you have any comments on this before it goes in? 

Here's a small screenshot of what the current visualization looks like:

https://github.com/mozilla/treeherder/pull/2530#discussion_r121427668

Comment 24

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

overall this is really great work- I like seeing the list of raw data and a visualization of it.  The noise level is important.  In the compare view we have colored visualizations as well (grey, green, yellow, red)- I am not sure if that would apply here, but it does help.

My general thoughts are to get this in, see how this looks over a few dozens scenarios and then work on any tweaks.  In general, we typically push to try and look not at a single data point, but all of the tests compared to recent historical data, so some method to make this easier would be nice.

Comment 25

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8874208 [details] [review]
[treeherder] SJasoria:1164891-canvas > mozilla:master

I have addressed all the requests on the pull request. I have also reduced a size of tooltip. Below this size, the distribution visualisation for tests with a large number of runs looked cramped up.

Comment 26

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/eeaa505a2f18fc858cdbcef2362c498786119d37
Bug 1164891- Visualize the distribution of performance test results using Canvas API (#2530)

Comment 27

[Ed Morley [:emorley]]

The newly added 'numeral' package is in the main index/perfherder webpack bundle rather than the separate vendor bundle:

$ curl -sSf https://treeherder.mozilla.org/index.bb24f9660c639bb7eeed.bundle.js | grep n umeral.js
 * numeral.js

I don't suppose you could add it to the list here, so it's put in the vendor chunk?
https://github.com/mozilla/treeherder/blob/ca9de94d3ca4e5a399a6fa8a71321653d89da03a/neutrino-custom/base.js#L35-L41

Comment 28

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1164891-canvas > mozilla:master

Comment 29

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1164891-followup > mozilla:master

Comment 30

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8878578 [details] [review]
[treeherder] SJasoria:1164891-followup > mozilla:master

I have made the changes. Hope it's good now.

Comment 31

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/92ac4905ba1fe332d0ec2f985c1174ff85345a42
Bug 1164891 - Move the numeral library to the vendor chunk (#2569)

Comment 32

[Ed Morley [:emorley]]

Comment on attachment 8878578 [details] [review]
[treeherder] SJasoria:1164891-followup > mozilla:master

Many thanks!

Comment 33

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1164891-followup > mozilla:master

Comment 34

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8889386 [details] [review]
[treeherder] SJasoria:1164891-followup > mozilla:master

It seems to work fine locally

Comment 35

[Ed Morley [:emorley]]

Comment on attachment 8889386 [details] [review]
[treeherder] SJasoria:1164891-followup > mozilla:master

Many thanks!

Comment 36

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/0f060a4773b366473fe71e945b5e6f3f6ca5c76b
Bug 1164891 - Remove directly included numeral library (#2651)

Since numeral is being included using an angular fixture it is not required to add
it in entry-perf.js