746319 - Modification of Telemetry Packet to include extra information

Status: RESOLVED DUPLICATE of bug 736189

(Toolkit :: Telemetry, defect)

Description

["Saptarshi Guha[:joy]"]

Hello,

We need to modify the telemetry module to support some extra meta
information:

- measurement type: ordinal (there is order in the variable but no
- scale), categorical (no order), and continuous
- units for a metric - is it in bytes? or seconds?
- validation rules  - so data can be verified when pushed into hbase
- description  - what does the metric do
- is it multidimensional - are we capturing joint distributions of multiple variables
- uptime for measurement collected since uptime (IIRC) does not have much meaning anymore.

Comment 1

[Lawrence Mandel [:lmandel] (use needinfo)]

I agree that this type of information would be a useful addition to Telemetry. In terms of UX, a human readable name for the probe and a description would be of benefit. 

How will adding this information to the ping (what I understand this request to be) affect the size of the daily ping? Can we contribute this information via another mechanism?

Comment 2

["Saptarshi Guha[:joy]"]

True, can this be automatically parsed from the header files? i.e when a metric is added, can the macro take a definition and this can be parsed from header files. This can then be sent over to metrics.

I agree providing definitions in the telemetry packet is wasteful since it's not changing with time. But then, assuming the buckets etc are not changing for a metric over time, sending that too is wasteful.

Would compression solve everything?

Comment 3

[Justin Lebar (not reading bugmail)]

> True, can this be automatically parsed from the header files?

Yes, that's what I recall we decided to do in the meeting we had about this.  Not all of the data is in the header file, but we can add what's missing.

> Would compression solve everything?

No.  :)  You still pay for the extra data, so we just shouldn't send it, imo.

Comment 4

[(dormant account)]

I'm with Justin.

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

Yep. in the meeting, we decided to file this bug to have a metadata section added to the declaration of the probes in the source code.  We can then use either the buildbot infrastructure or some other mechanism to import the metadata and make use of it.

Some other things we also discussed possibly putting here:

SimpleMeasure bucket definition - When we aggregate simple measures into histograms, what should we use for bucket intervals?

Classification - a way to logically group different types of measures together.  For instance: startup; X_feature; garbage collection; memory usage; stability; SQL execution
  This could be either a single hierarchy or possibly a tag system.

Something to indicate whether a lower value is better or worse than a higher value.  Having this as metadata could allow us to provide more intuitive charts in the dashboards.

Comment 6

[(dormant account)]

Note most of the data is already there. See http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetryHistograms.h this is enough to get started. We can extend this with units, labels, etc(by changing the comment field)

Comment 7

[(dormant account)]

Validation data will provide all our metadata.