2017318 - Investigate size of main_v5's additional_properties

Status: RESOLVED FIXED

(Toolkit :: Telemetry, task, P1)

Description

[Chris H-C :chutten]

The most expensive "probe" in main_v5 is additional_properties, the column that accepts all data that doesn't otherwise fit in a column.

What's in there? Why is it so big?

Comment 1

[Chris H-C :chutten]

"main" pings have a lot of data defaulting to additional_properties. Basically all of them have something in there. Some common themes I've found so far:

1. simpleMeasurements missing columns: all simpleMeasurements aside from active_ticks, blank_window_shown, first_paint, main, session_restored, and total_time are not manually added as columns and so end up in additional_properties. (There are a lot of other simple measurements. All of which are being collected via Glean since bug 1972836)
2. "extension" process missing columns. The "extension" process is a special kind of "content" process: its process type is GeckoProcessType_Content. But Telemetry is process-aware and wanted to keep "extension" process data away from the rest of web content. So when data recorded on a "content" process reaches IPC, it uses the process' mRemoteType to detect whether it's a "true" "content" process, or whether it's really an "extension" process. HOWEVER, this doesn't go all the way through Telemetry as there is no way in a probe definition to specify "content" vs "extension". So, though a probe says something like "record_in_processes": ["main", "content"], what that means is that the probe is recorded in processes of types "main", "content", and "extension". However, mozilla-schema-generator doesn't know that and hasn't in the 7 years since we migrated away from AWS. So the only column in payload.processes.extension.histograms is uptake_remote_content_result_1.
   • This is by far the biggest contributor to the size of additional_properties that I've found thus far.
3. "utility" process missing columns. When "utility" process support was added in bug 1794409, there was no corresponding change in mozilla-schema-generator. So any and all Legacy Telemetry probes recorded on that process have never had columns.

Comment 2

[Chris H-C :chutten]

Attachment: Bug 2017318 - Remove simpleMeasurements that the pipeline doesn't have columns for r?#toolkit-telemetry-reviewers-rotation

Comment 3

[Chris H-C :chutten]

Attachment: Bug 2017318 - Don't report Legacy extension-process telemetry since the pipeline doesn't have columns for it r?#toolkit-telemetry-reviewers-rotation

This doesn't stop recording Legacy Telemetry in extension processes,
just the reporting of it.

Comment 4

[Chris H-C :chutten]

Attachment: Bug 2017318 - Don't report Legacy utility-process telemetry since the pipeline doesn't have columns for it r?#toolkit-telemetry-reviewers-rotation

This doesn't stop recording Legacy Telemetry in utility processes, just the
reporting of it.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c99f38fb7b43
https://hg.mozilla.org/integration/autoland/rev/60d69cf1a517
Remove simpleMeasurements that the pipeline doesn't have columns for r=extension-reviewers,sessionstore-reviewers,TravisLong,rpl,nsharpley
https://github.com/mozilla-firefox/firefox/commit/a292048b69f0
https://hg.mozilla.org/integration/autoland/rev/b7bfde1fdec0
Don't report Legacy extension-process telemetry since the pipeline doesn't have columns for it r=TravisLong
https://github.com/mozilla-firefox/firefox/commit/46b064817276
https://hg.mozilla.org/integration/autoland/rev/516a72ddcfc5
Don't report Legacy utility-process telemetry since the pipeline doesn't have columns for it r=TravisLong

Comment 6

[rperta]

https://hg.mozilla.org/mozilla-central/rev/60d69cf1a517
https://hg.mozilla.org/mozilla-central/rev/b7bfde1fdec0
https://hg.mozilla.org/mozilla-central/rev/516a72ddcfc5