686036 - Consider using a scatter chart for build times

Status: RESOLVED INCOMPLETE

(Testing Graveyard :: GoFaster, defect)

Description

[William Lachance (:wlach)]

From jgriffin:

"For the build times chart (http://brasstacks.mozilla.com/gofaster/#/executiontime/build), I wonder if a scatter chart wouldn't be more useful?  The current line chart can be deceiving; if there are few builds on a particular day (i.e., weekends, holidays) and they happen to be extra long, it looks like our build times have been increasing, when in reality that's not the case, on average.  A scatter chart (possibly with a trend line), might help us determine whether there has been any real overall trend vs some oddball outliers. ), I wonder if a scatter chart wouldn't be more useful?  The current line chart can be deceiving; if there are few builds on a particular day (i.e., weekends, holidays) and they happen to be extra long, it looks like our build times have been increasing, when in reality that's not the case, on average.  A scatter chart (possibly with a trend line), might help us determine whether there has been any real overall trend vs some oddball outliers."

Comment 1

[William Lachance (:wlach)]

Note that we now provide an option (default: on) to exclude outliers, which makes this perhaps a bit less necessary (though probably still desirable).

Comment 2

[William Lachance (:wlach)]

(this may also be useful for other graphs, like setup/teardown times)

Comment 3

[Simon de Vlieger]

In the case of implementing the scatterplot what would you like to see on the x and y axis of the plot.

My suggestion would be to run with the build times on the y axis and the platforms on the x axis so you can more easily see which platforms have the longest build times.

Comment 4

[Simon de Vlieger]

Also in the implementation case, would we have checkbox on pages which currently feature a line graph with which to toggle to scatterplot? If that would be desirable UI feature then my previously suggested solution might obsolete this ticket: https://bugzilla.mozilla.org/show_bug.cgi?id=687600 since you would already have a more clear visual representation of which platforms are holding back.

Comment 5

[William Lachance (:wlach)]

For reference, I believe we were talking about this graph: 

http://brasstacks.mozilla.com/gofaster/#/endtoend/average

I think the idea here was to keep the X axis and Y axis the same (X=Date, Y=time to build), but instead of a line graph showing the averages on each day, you'd display a bunch of points on the graph corresponding to the actual build times (where build times == end-to-end build and test for all platforms). In fact though, I don't see any reason to remove the line graph in the short term: we could continue to show the average along with the scatter plot of individual build times (we could also consider creating a trend-line, as jgriffin suggests, but I'm not sure how complicated that would be).

The algorithm to generate this data would look something like:

for each date:
   for each relevant build/test cycle on that date:
       plot a point corresponding to the total time for that build

I'm not exactly sure how to incorporate platform-specific data into this graph, except maybe to reveal the last job for that build when you hover over a point.

The relevant code to do this would be here:

https://github.com/mozilla/gofaster_dashboard/blob/master/src/dashboard/server/handlers.py#L166

Feel free to join us during the week on irc.mozilla.org #ateam if you want additional clarification (I'm usually around from 9-5pm EST, jgriffin from 12-7 PST).

Comment 6

[Jonathan Griffin (:jgriffin)]

> 
> I'm not exactly sure how to incorporate platform-specific data into this
> graph, except maybe to reveal the last job for that build when you hover
> over a point.
>
We could possibly color the points on the scatter chart differently, depending on which platform the last job occurred on.