699670 - Add sum/average/stddev to telemetry dashboard

Status: RESOLVED FIXED

(Mozilla Metrics :: Frontend Reports, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

When it's a log scale, it's hard to guess this data. I need this for GC_MS, GC_MARK_MS, and GC_SWEEP_MS. But it would probably be useful to have it for everything.

Also, it would be nice if you could ask it what percentage of the samples are above or below a certain value. Even better would be if, along with the histogram, it showed a CDF.

Comment 1

[Lawrence Mandel [:lmandel] (use needinfo)]

Pedro - I cced you as you're involved in a numbe of the Telemetry related bugs. Can you please take a look at this request?

Comment 2

[Justin Lebar (not reading bugmail)]

> Even better would be if, along with the histogram, it showed a CDF.

This.

Comment 3

[Annie Elliott]

Migrated to be tracked in JIRA with the rest of metrics work. Internal folks see METRICS-774 for replacement.

Comment 4

[Annie Elliott]

Work now being tracked as part of: https://metrics.mozilla.com/projects/browse/METRICS-774

Comment 5

[Annie Elliott]

updating whiteboard

Comment 6

[Annie Elliott]

updating status and assignee

Comment 7

[Annie Elliott]

specific JIRA: https://metrics.mozilla.com/projects/browse/METRICS-836

Comment 8

[Annie Elliott]

New JIRA migration identifier - 55 BZ issues being considered in current planning for Q3

Comment 9

[Pedro Alves]

Reviweing this bugs. This was filled brfore telemetry evolution, and that view has that info, even if it's a different view. Is this still relevant?

Comment 10

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I'd still like to see a CDF.

Comment 11

[(dormant account)]

Bill, does https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=telemetryEvolution2.wcdf work for your needs?

Comment 12

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

(In reply to Taras Glek (:taras) from comment #11)
> Bill, does
> https://metrics.mozilla.com/data/content/pentaho-cdf-dd/
> Render?solution=metrics2&path=%2Ftelemetry&file=telemetryEvolution2.wcdf
> work for your needs?

It helps, although it raises more questions than it answers. Why does the number of submissions vary so much? It seems like we should have more submissions for released versions of FF. What counts as a submission?

Comment 13

[(dormant account)]

Overall this is a big improvement over existing histogram views.

My observations:
* Why is 99th percentile special? Will be great to get a view of fully backprocessed data

* Need a better way to zoom 'percentiles(sec)' chart...possibly by letting us click at one release a time.
Nightly gets frozen during zooming. This aspect we can fix in a followup. Note Firefox nightly ships with a great profiler to fix perf issues like this.

* Seems we could combine all of the percentiles + mean...and use a list of checkboxes to toggle them off.

Comment 14

[Justin Lebar (not reading bugmail)]

I agree with Taras and Bill; this is a big improvement, and it still has room for a lot of improvement.

Comment 15

[Paulo Pires]

This is still a WIP but no longer shows a CDF, it's according to the first recommendations from Joy and shows a Quantile Plot.

When using a linear scale the differences where barely unnoticeable, so the Quantile Plot has now a logarithmic scale.
When we have 0 values it is showing Math.log(0.9)/Math.log(10) so it won't decrease much the Y-axis space.
The values on the tooltip are not logarithmic, as also the labels in the Y-axis.

The dashboard can be seen at https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=telemetryHistogram2.wcdf

There are a few more recommendations made by Joy to implement:
1. When you have zero values, don't show them, instead mention "4% of the values were zero, this is a quantile plot of the non zero values" and then proceed to show the 1%,2%,..., 99% percentiles.
2. Use an aspect ratio of 1. This is make easier to perceive slopes in the curves or to compare the orientation of the curves.
3. Show 1%,..99% of the data , as opposed to the 0% to 100%. This wil drop of the extremes. Have an export button to export /all/ the percentiles.

Comment 16

[Justin Lebar (not reading bugmail)]

Maybe this is a stupid question, but I don't understand how this plot is different from a CDF, except that in a CDF we'd traditionally switch the X and Y axes.

Comment 17

[Pedro Alves]

We'll let joy answer that :D

Comment 18

[Justin Lebar (not reading bugmail)]

I guess this Q-Q plot is discrete, as opposed to a CDF, which is continuous.  If that's the only difference, I don't care.  Although I would prefer if we flipped the axes so this looked like a CDF.

Perhaps instead of assuming that all plots need to be log-scale, we could default to log-scale only for log-scale histograms.  Or does that not work?

Comment 19

["Saptarshi Guha[:joy]"]

Justin, 

This is a quantile plot, see http://www.stata.com/support/faqs/graphics/gph/graphdocs/quantile-plot/index.html .

It is really a CDF turned around. But the goals for most people looking at CDF is what is the X percentile of the data?

A: look at the x-axis and join a line.

This has nothing to do with QQ plots except that one could say this a QQ plot vs the uniform distribution.

I completely agree you want the log transformation only if the histogram is a exponential one.

Comment 20

[Justin Lebar (not reading bugmail)]

I don't really want to fuss about this, but I think doing something that looks more like a CDF would be helpful, since that's something we all know how to read.  It would also let us align the CDF with the plot underneath it, which would be helpful, I think.

Comment 21

["Saptarshi Guha[:joy]"]

For the objectives mentioned, i feel the quantile plot is appropriate. it doesn't take long to get used to it.

it is a well known plot and nothing new to the statistical field.

I still recommend an aspect ratio of 1 for the quantile plot.

I dont see the benefit of aligning the quantile plot (or CDF ) with a histogram  - what analytical value is there? Moreover, i would also expect the X-labels would be same, which i doubt will be the case in the dashboard.


On another note, since CYCLE_COLLECTOR is exponential, the histogram should be that of log(CYCLE_COLLECTOR) with possible labels on base 10 scale.

Comment 22

[Justin Lebar (not reading bugmail)]

Okay.  Anything we get here will be an improvement over what we currently have, so I'm not inclined to fight about it.

Just to be clear, this sort of disagreement that is an important part of the reason why so many people in engineering want to build their own dashboard.  I'm not saying that your design sense is right or wrong, but many of us are tired of fighting to get the UI that we want out of the Telemetry dashboards.

Comment 23

["Saptarshi Guha[:joy]"]

Hi,

I'm sorry. It isn't really a question of design - but what is analytically correct for the purpose. The Quantile plot is appropiate and what should be used for the uses mentioned.

The reason why i disagree are comments like 
"what is my median time to ready for HTTP vs HTTPs
connections "

How can medians be compared without confidence intervals (i.e. any notion of the variance in the median estimator)?
I have worked on bugs where medians are different, even the tails are seemingly very different, but if one accounts for variance, they are *not*. Are the proposers aware of these issues?

So i'm for all public access to the data, but i wonder how they will be consumed in the right analytically useful way?

So in summary: i'm not fighting, but suggesting what is the appropriate way to consume the data. I dont believe sticking to something 'we're used to' is sufficient reason to stay with it.

Comment 24

["Saptarshi Guha[:joy]"]

Paulo,

I think the quantile plot needs labels

a) the measurement unit on the Y-axis (ms,bytes?)
b) "Proportion" on the x-axis

and (c) give it a caption called Quantile Plot.

and make the aspect ratio 1 (i.e a square)

Comment 25

[Annie Elliott]

Yes -

Please, every single axis everywhere should be labeled. There should be no doubt what the units are.

Comment 26

[Paulo Pires]

Can we safely assume that the end of the string for each measure represents the measurement unit?

For example, measures that end with _MS are ms?
_BYTES are bytes?
_KB are Kbytes?
_TIME are seconds?
_COUNT are #?

But we have many others that don't follow up to this, so it would be great to have a list of units for each measurement

Comment 27

[Daniel Einspanjer [:dre] [:deinspanjer]]

Harsha, I believe that the units and scale are things found in histograms.json, right?  Could you point Paulo at where they can retrieve the json file?

Comment 28

[Harsha [:harsha]]

Hi Paulo,
     Please take a look at this file http://hg.mozilla.org/mozilla-central/file/b1a08130fae6/toolkit/components/telemetry/Histograms.json In description they mention the units. 
-Harsha

Comment 29

[Paulo Pires]

Can we commit these changes on the public server?
We need to git pull and create branches for each repository

Comment 30

[Daniel Einspanjer [:dre] [:deinspanjer]]

Bill, could you please review this version and give us the go-ahead to replace the existing dashboard with this new version?

https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=telemetryHistogram2.wcdf

Comment 31

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Thanks very much. This looks great!