913726 - [META] Datazilla Firefox OS Improvements

Status: RESOLVED INCOMPLETE

(Datazilla Graveyard :: User interface, defect, P2)

Description

[Jonathan Eads ( :jeads )]

This bug is meant to consolidate/track several requests for new functionality in the datazilla UI. Some of these listed features may be split out into individual bugs in the future. Not sure what the prioritization of these features should be. Working on scheduling for Q4.

UI Features
-----------
1.) Support linking to external documentation 

Add support for linking tests to external documentation. This request was made in github issues https://github.com/mozilla/datazilla/issues/98. b2g performance test documentation can be found here https://wiki.mozilla.org/B2G/Datazilla#Current_Tests. We could add a "?" icon where the test name gets set dynamically when a user selects a particular test. The "?" icon could be displayed to the right of the Tests menu label and would be set to whatever the currently selected test is. The icon would link to corresponding HTML anchor found in b2g wiki.

2.) Plot by the median in addition to the mean (boxplots?)

Add a feature to enable the user to plot data by the median in addition to the mean. This request was made in github issues https://github.com/mozilla/datazilla/issues/102. This could provide some additional insight into test time trends that are hidden by applications that have a lot of noise in their replicates. In addition, bkelly also suggested displaying box plots for each data point. This might be the way to go but could yield a messy visualization. I will need to play around a bit and see what works best, either way, enabling a visualization of trends in the median would be useful.

3.) Plot error bars

If box plots prove to be unwieldy, displaying error bars would be very useful. Some applications exhibit dramatic increases in the standard deviation due to noise in the replicates. To see this information now, a user needs to hover over each data point and examine the standard deviation displayed. This is not ideal, displaying error bars would be more effective.

4.) Visualize test time distribution as a histogram

Another idea suggested by bkelly in https://github.com/mozilla/datazilla/issues/102 is to use a histogram to visualize the distribution of test times. This could be a useful way of filtering out some of the noise and getting a better idea of what the most common occurring test times are.

5.) Add zoom in/out and change graph widths to be elastic

To support 2, 3, and 4 above and also to improve the visualization of the 30-60-90 day time intervals that produce very dense data, zoom in/out would be useful. In addition, it would be a good idea to switch from static graph widths to elastic, so the graphs will increase in size as a user increases the browser size. Users with larger monitors would be able to get more out of dense data displays.

6.) Filter out reference data that is no longer valid

Several methods of doing this were proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=892302#c5 but no choice was made as to what would be the best way forward. This bug has been closed so maybe this is no longer an issue. However, there remain many "dead end" test/branch combinations available in the UI that have no data associated with them. One easy temporary fix proposed was to just exclude the test/branch combinations that are no longer valid. A branch/app/test/device type data distribution summary is available here https://bug892302.bugzilla.mozilla.org/attachment.cgi?id=775023

7.) Modify the display of tests to be hierarchical 

This has been discussed several times. It's particularly relevant to the display of the endurance tests. It would also involve adding some additional properties to the JSON object to specify what the hierarchical organization of a particular test should be.

Comment 1

[cmtalbert]

8. Allow us to change the label for the y axis on the graphs as well now that we're starting to store both memory and framewrates in there as well as milliseconds (the current default)

Comment 2

[Dave Huseby [:huseby]]

tagging this up so that the b2g perf team can track this.

Comment 3

[Jonathan Eads ( :jeads )]

Adding a few more UI tasks here

9.) Persist the date range.

The date range displayed is hard coded to 7, 30, 60, 90 day ranges relative to the current date. This is a problem for historical links identifying particular regressions because they eventually go out of range. We need to persist the start/end dates for the range in the url so that the data point of interest is always visible and selected.

10.) Hard code expected date range so that missing data is more obvious

The date range displayed on the x-axis of the main performance chart displayed in https://datazilla.mozilla.org/b2g is set to the date range of the data retrieved for the specified time interval. This has the unwanted side effect of adjusting the data points to fit the axis size, making it hard to determine when data has not been received recently. We would like to hard code the x-axis range to the date range selected, regardless of the date rage of the data retrieved.

11.) Order by gecko or gaia repository push dates

Test data in the b2g user interface is currently ordered by the date that it was received. It would be useful to be able to order test data by the gaia or gecko repository push order. In addition to ordering, the revision sha changes relative to each data point need to be visualized in some way for both repositories. This would be a relatively large project and require a number of changes to the data model.

12.) Display new json property, generated_by, in the ui

A new property, generated_by, has been added to the json structure submitted by b2g perf. It contains the b2g perf test runner version, this needs to be displayed in the UI when a user selects a data point.

Comment 4

[William Lachance (:wlach)]

Note that datazilla is going to be replaced with treeherder performance in the next few quarters, it might be more valuable to direct resources there. I'm the lead developer on that, please contact either me or jgriffin (project manager) for more information.

Comment 5

[Jonathan Griffin (:jgriffin)]

B2G perf data is now being visualized at http://raptor.mozilla.org