1350384 - Add ability to see a visualization of the test replicates in Perfherder

Status: RESOLVED FIXED

(Tree Management :: Perfherder, enhancement)

Description

[William Lachance (:wlach)]

Perfherder's main views are intended to show *summaries* of different types of test data (or in many cases, summaries of summaries). For example, the tp5o summary benchmark is a summarization of a set of tp5o subtests each of which has many replicates.

However, it could be very useful to be able to display a visualization of these test replicates, either on their own or in a comparison view (when comparing the results from two revisions). :rwood made talos jobs emit the raw data for these replicates in bug 1316692, so it should just be a matter of adding the right view to perfherder so we can actually visualize this.

Shruti and I have been chatting about this, and we think a histogram view probably makes sense. There are probably two cases which we'd want to cover. 

The most common would be viewing two histograms of two runs side-by-side (similar to the current subtest compare view, but for the individual replicates rather than a subtest summary)

The other (really a variant, could reuse much of the logic) would be viewing the replicate distribution for each a particular job or a particular push.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1350384 > mozilla:master

Comment 2

[Shruti Jasoria [:ShrutiJ]]

So far views are being shown for the case where two specific revisions are being compared.

Test with one run on both original and new project:
http://localhost:5000/perf.html#/comparesubtestdistribution?newProject=try&newRevision=dd1d5ae6cd67837004b114351305a6acdb28e9af&newSubtestSignature=81bf4785259fbf48745b4675d422617c46ee3dc8&originalProject=mozilla-inbound&originalRevision=2e1ae8b575704094ae318e8c82e690a1552227b9&originalSubtestSignature=81bf4785259fbf48745b4675d422617c46ee3dc8

Test with three runs on original, none on the new one(would throw an error on console):
http://localhost:5000/perf.html#/comparesubtestdistribution?newProject=mozilla-inbound&newRevision=5223ac33f4b4c5cf64ac275824f8ded2cc77ba40&originalProject=mozilla-central&originalRevision=8f80d594c08d5c7a112e5d4b9eb44ffca717eb7b&originalSubtestSignature=4f9c102bb65f2215c0557144596103daedc0c9b7

Changes I'll be making next:
1. Make it work for baseline comparison.
2. Improve the page's style and add other relevant information.
3. Add a check for new signature, similar to Bug 1376837.
4. Add a $scope.dataLoading variable which will help to display the page once everything is loaded. Would show the cat otherwise.

Comment 3

[Shruti Jasoria [:ShrutiJ]]

I have left a comment in the PR with a few screenshots of the view which we have obtained so far. Please take a look at it.

I think it's good time to ask jmaher and sguha for suggestions.

Comment 4

[William Lachance (:wlach)]

We've discussed this a bit. Let's get the bugs out of this, land, and then iterate. :)

Comment 5

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8878881 [details] [review]
[treeherder] SJasoria:1350384 > mozilla:master

Apart from the changes which I have made so far the following has to be implemented:

1. min_y_from_data for y axis which would be helpful in the cases where the range of replicate values is small.

2. A button to view the data in a sorted manner.

Comment 6

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/dcac420965a8fcc0a7acbb9b5d42ee6eebf83014
Bug 1350384 - Add ability to see a visualization of the test replicates in Perfherder (#2571)

Comment 7

[GitHub Autolander Bot]

Attachment: [treeherder] SJasoria:1350384-followup > mozilla:master

Comment 8

[Shruti Jasoria [:ShrutiJ]]

Comment on attachment 8889371 [details] [review]
[treeherder] SJasoria:1350384-followup > mozilla:master

I have made the changes. Hope it's good now.

Comment 9

[Ed Morley [:emorley]]

Comment on attachment 8889371 [details] [review]
[treeherder] SJasoria:1350384-followup > mozilla:master

Thank you :-)

Comment 10

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/fd5fc36ea830d7320d4418e11355d23e03b28c7a
Bug 1350384 - [Follow Up] Remove redundant dependencies (#2650)

metrics-graphics is built over d3. So there is no need to add its pacakage separately.
Direct include of metrics-graphics is redundant with the angular fixture.