1288444 - Add categorical histograms to the longitudinal dataset

Status: RESOLVED WONTFIX

(Data Platform and Tools Graveyard :: Datasets: Longitudinal, defect, P3)

Description

[Georg Fritzsche [:gfritzsche]]

From IRC:

> categorical histograms could be stored in structs which is what we are already doing in main_summary:
> https://github.com/mozilla/telemetry-batch-view/blob/master/src/main/scala/com/mozilla/telemetry/views/MainSummaryView.scala#L387
> Probably using maps is the better choice as it allows to use cross join unnest

Comment 1

[Ryan Harter [:harter]]

I need some clarification on this. The histograms.json [0] file only lists two example categorical histograms. Are there histograms missing from this set or are we expecting more in the future? Is there an easy way to enumerate these besides filtering past pings?

[0] https://hg.mozilla.org/releases/mozilla-release/raw-file/tip/toolkit/components/telemetry/Histograms.json

Comment 2

[Georg Fritzsche [:gfritzsche]]

(In reply to Ryan Harter [:harter] from comment #1)
> I need some clarification on this. The histograms.json [0] file only lists
> two example categorical histograms. Are there histograms missing from this
> set or are we expecting more in the future? Is there an easy way to
> enumerate these besides filtering past pings?

We are expecting more in the future. AFAIU the plan is that once the tooling is in place, data collection reviewers can point engineers to using them.

Comment 3

[Georg Fritzsche [:gfritzsche]]

(In reply to Ryan Harter [:harter] from comment #1)
> Is there an easy way to enumerate these besides filtering past pings?

Not right now, although i'm thinking about how to fix this.

Comment 4

[Ryan Harter [:harter]]

I tried to find some example pings from the test histograms, but they don't
seem to be reporting [0].

Do we have examples or a spec for these histograms payloads?  The docs
mention these will be like enumerated histograms with labels, but the format's
still unclear to me.  Will they payload be an array, as in a enumerated
histogram, or a map? Thanks!

[0] https://gist.github.com/harterrt/ed44bdd4e07b75be652154f5d706eef6

Comment 5

[Georg Fritzsche [:gfritzsche]]

(In reply to Ryan Harter [:harter] from comment #4)
> Will they payload be an array, as in a enumerated
> histogram, or a map? Thanks!

Yes, in the payload they are just like enumerated histograms.
The labels are only used in the client API, internally it is just like an enumerated histogram.

E.g. for TELEMETRY_TEST_CATEGORICAL, adding 1 to the labels "CommonLabel" & "Label2", <ping>/payload/histograms/TELEMETRY_TEST_CATEGORICAL looks like this:
> {"range":[1,3],"bucket_count":4,"histogram_type":1,"values":{"0":1,"1":1,"2":0},"sum":1}

Comment 6

[Georg Fritzsche [:gfritzsche]]

Note that we are changing the bucket logic in bug 1312806 to reduce user error.
Bug 1324478 was filed to deal with this from telemetry-batch-view.

Comment 7

[Ryan Harter [:harter]]

Moving to P3. I do not plan on addressing this in Q2.

Comment 8

[Ryan Harter [:harter]]

Moving to P2, due to Bug 1376596

Comment 10

[Ryan Harter [:harter]]

I will not be able to take care of this in the next month. Moving to P3.

Comment 11

[Arkadiusz Komarzewski [:akomar]]

Longitudinal has been decommissioned per Bug 1572033.