1253259 - The sum from DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS histogram is not correct

Status: RESOLVED INVALID

(Toolkit :: Telemetry, defect)

Description

[Mihai Boldan, Desktop QA [:mboldan]]

[Note]:
- Logged based on Bug 1242057 Comment 13

[Affected versions]:
- Firefox 47.0a1 (2016-03-02)

[Affected platforms]:
- Windows 10 x86

[Steps to reproduce]:
1. Launch Firefox.
2. From about:config, enable the devtools.responsive.html.enabled pref.
3. Open RDM and close it after a few seconds.
4. Repeat step 3.
5. Go to about:telemetry.
6. Search for DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS probe in Histograms and observe the sum.

[Expected result]:
- The 'sum' is equal with the sum of the seconds displayed under each counter.

[Actual result]:
- The sum is not equal with the sum of the seconds displayed under each counter.

[Regression range]:
- This is not a regression since this is a new implementation.

[Additional notes]:
- Note that if the RDM is opened under 15 seconds, the correct no. of seconds is added to the sum.
- Also, if the sum is not correct, the average time is also not correct.

Comment 1

[Chris H-C :chutten]

Would you happen to have a screenshot handy?

Comment 2

[Mihai Boldan, Desktop QA [:mboldan]]

Attachment: sum.png

Here is a screenshot with the issue described above.

Comment 3

[Chris H-C :chutten]

Ah, I see the confusion.

To make an infinitely-precise continuous distribution of measurements (like the number of seconds the devtools are active) able to be stored and transported using finite resources, Telemetry "buckets" values as they are accumulated. DEVTOOLS_RESPONSIVE_TIME_ACTIVE_SECONDS is an exponential histogram with a low of 0, a high of 10000000 and 100 buckets. This means we will store at most 102 numbers for this histogram: the number of values that fall into the 100 buckets, the average of all values, and the sum of all values.

This is very space-efficient, but loses information. We have 100 buckets for all values from 0 to ten million seconds. If this were a linear histogram, each bucket would be one hundred thousand seconds (over a day) wide. Since it is exponential, the buckets at the lower end are narrower and the buckets at the higher end are wider.

Here's an example: say you recorded active seconds of 26, 110, 80, 75, and 42. Then the histogram would have a 5 in the bucket labelled "25" (since all five values fall in the range 25-124), the sum would be 333, and the average would be 66.6.

I think that this Telemetry probe was added without a proper data review back in May, 2013 for bug 866642. Having the max value be ten million seconds is a bit high, since that equates to roughly a third of a calendar year. I think this probe should be retired and replaced by another that is better suited to the measurements you wish to take.

In fact, I think all of the... good grief, there are, what, 60 of these probes added in bug 866642? These should probably all be reviewed for utility.

Comment 4

[Mihai Boldan, Desktop QA [:mboldan]]

(In reply to Chris H-C :chutten from comment #3)
>  
> Here's an example: say you recorded active seconds of 26, 110, 80, 75, and
> 42. Then the histogram would have a 5 in the bucket labelled "25" (since all
> five values fall in the range 25-124), the sum would be 333, and the average
> would be 66.6.
 
Based on your explanation I'm not sure if this is an invalid bug.
Since the sum of the numbers from your example is 333, than the sum from the histograms displayed in the screenshot should be 125 instead of 138(firs histogram) and 148 instead of 164(the second histogram).

Can you confirm that this is correct, or am I missing something?

Comment 5

[Chris H-C :chutten]

The first histogram represents a population with one data point: 138. Its sum is 138, its average is 138. 

The second histogram represents a population with two data points: 138 and 26. Their sum is 164, their average is 82.

To store these two points, we place them in buckets: the 138 data point goes in the [125,...) bucket. The 26 goes into the [23, 125) bucket. The numbers below the bars in the histogram are not the data points' values, but are the inclusive lower bound of the bucket. The numbers above the bars are the counts of how many data points are of values that fall into the bucket.

Comment 6

[Mihai Boldan, Desktop QA [:mboldan]]

Thanks for clarifying this. It make sense now.