1172543 - Empty histograms should be submitted

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

When a histogram is empty, it is submitted only if it's a count, flag or keyed histogram. Ideally we should submit all histograms even when they are empty, or to the very least adopt a consistent behaviour across all histogram types.

During analysis it's not clear if a histogram is missing because it was empty or because of other reasons (e.g. bug, histogram not applicable for client configuration, etc.).

Comment 1

[Mark Reid [:mreid]]

I would like to know what impact this will have on payload size.

Comment 2

[Georg Fritzsche [:gfritzsche]]

As i understand it, we submit count & flag histograms because they have a defined value even if never .add()ing to them (0 & false respectively).
For the other types there is no specific value there; we could actually limit keyed histogram submissions to flag & count types too.

I don't understand what exactly submitting all of them helps us with, do you have a specific scenario in mind?
E.g. submitting empty histograms still doesn't tell us about the presence or absence of bugs.

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #2)

> I don't understand what exactly submitting all of them helps us with, do you
> have a specific scenario in mind?
> E.g. submitting empty histograms still doesn't tell us about the presence or
> absence of bugs.

Yes, validation and consistency. We are going to do a lot more of the fomer in the next quarter hopefully. It's also confusing to treat keyed histograms and non-keyed histograms differently. 

Say the histogram for the number of Flash crashes was not a keyed histogram and you would want to count the proportion of session with crashes. 

To compute the ratio you would need to know how many session didn't have a crash, so that number is important. With keyed histograms I know that the histogram is always there, even when it's empty. With "normal" histograms though, I don't know if the histogram is missing in a session because:

a) histogram is empty
b) metric doesn't apply to that configuration
c) the payload comes from an older build that didn't have that histogram
d) bug
e) (did I forget anything?)

For c), given the build revision, one could get the histogram definition file and see if that histogram is defined and in that case, if the histogram is missing, assume that it's empty. That's going to be slow though, even with a dedicated service.

The current approach could easily yield wrong conclusions; ideally I would like the data to be in a format that is easy to use in the right way and hard to use it in the wrong way. It would be nice to have a quick way to assert that e.g. histogram A was defined in the build that generated the payload and the histogram is empty.

Comment 4

[Georg Fritzsche [:gfritzsche]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #3)
> Say the histogram for the number of Flash crashes was not a keyed histogram
> and you would want to count the proportion of session with crashes. 
> 
> To compute the ratio you would need to know how many session didn't have a
> crash, so that number is important. With keyed histograms I know that the
> histogram is always there, even when it's empty. With "normal" histograms
> though, I don't know if the histogram is missing in a session because:
> 
> a) histogram is empty
> b) metric doesn't apply to that configuration
> c) the payload comes from an older build that didn't have that histogram
> d) bug
> e) (did I forget anything?)

I think the histogram being empty & present is not the best signal here.
For investigating that we should look at the releases and configurations that we want to analyse.

For b) we should be filtering by OS and other environment data etc. Including the empty histogram wouldn't help unless we would only have it present in the configuration you are looking for.
For c) we can constrain/filter the pings by Fx versions, buildids, ...
d) Bugs can happen either way.

I don't see how including empty histograms improves things much while it definitely increases payload size.
Am i missing something?

Comment 5

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #4)
> I don't see how including empty histograms improves things much while it
> definitely increases payload size.
> Am i missing something?

How should we discern in analyses between a measurement that is not available (for whatever reason) and one that is zero?

Comment 6

[Georg Fritzsche [:gfritzsche]]

We talked about this off-bug and came to the conclusion that we can't signal the different states ("histogram doesn't apply to this configuration", "... wasn't recorded", "... is not valid for this version") at a reasonable cost.

We will keep not submitting empty histograms (except "flag" & "count" histograms, which have default values though and are always submitted already).
In the future the validator (server-side) might write default values for other histogram types.