1614416 - Missing common ping environment columns in the telemetry namespace

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Anthony Miyaguchi [:amiyaguchi]]

See https://datastudio.google.com/reporting/1GnT6U5c9Z3D4p7DxnECcWU7f_IGGPZgA/page/HyRCB

Missing fields include:

• environment.system.gfx.adapters
• environment.addons.theme.foreignInstall
• environment.settings.userPrefs
• environment.addons.activeAddons

Comment 1

[Anthony Miyaguchi [:amiyaguchi]]

This would be fixed by updating the mozilla-schema-generator to propagate changes for schemas outside of the main ping to those that include the common ping format.

Comment 2

[Anna Scholtz [:ascholtz]]

I'm not sure if mozilla-schema-generator can fix those missing columns. The missing columns, for example environment.system.gfx.adapters.adapterDriver just get added when the ping is sent (see here https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/app/TelemetryEnvironment.jsm#509) but it's not defined in any file that is scraped by probe-scraper.

The only fix I can see here is to add those missing fields to the schema definitions.

Comment 3

[Anthony Miyaguchi [:amiyaguchi]]

I must have been thinking about environment.settings.userPrefs, which is a casted type that's specified in the schema generator. It would be ideal if those columns could be added directly to the raw schemas (via the environment include), but often there's a mismatch between the validation schema and the schema used for table generation. On concrete example in the schema generator repository is the metadata_merge which updates the schema specifically for the BigQuery tables.

There are probably quite a few of these exceptions that we'd want to add along with these changes, an a somewhat generic solution that can modify json paths across a set of pings in a namespace could be handy.

Comment 4

[William Lachance (:wlach)]

Hi Anna, it looks like you already did at least some of this in https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/511/? If you can remember, try to title pull requests with the bug number (e.g. "Bug XXX - ")-- that way the bugzilla pr linker will automatically make a mention of any pull requests filed to address a bug.

I proposed an update to the README to make this more clear: https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/514

Comment 5

[William Lachance (:wlach)]

Hi Anna, there are still a fair number of data in environment (beyond gfx adaptors) that we're not processing. Did you want to do this rest of this? Or should we file a bunch of new bugs?

Comment 6

[Anna Scholtz [:ascholtz]]

I'll do the rest of it for those that can be handled in schema generator. There are some fields, like activeGMPlugins that needs some special handling (see https://bugzilla.mozilla.org/show_bug.cgi?id=1611027).

Comment 7

[Anna Scholtz [:ascholtz]]

I created a PR for the remaining common ping columns that I could identify: https://github.com/mozilla/mozilla-schema-generator/pull/122
I'm not sure if there are more that should get handled? I guess we can create separate bugs for those or list them here.

Comment 8

[William Lachance (:wlach)]

(In reply to Anna Scholtz from comment #7)

I created a PR for the remaining common ping columns that I could identify: https://github.com/mozilla/mozilla-schema-generator/pull/122
I'm not sure if there are more that should get handled? I guess we can create separate bugs for those or list them here.

Aside from GMPPlugins which you already noted, there's also activeAddons and userPrefs (these might have similar issues as GMPPlugins?).
It might be best to file new bugs for these ones, I think it'll be less confusing than just making this a catch-all.

Comment 9

[Anna Scholtz [:ascholtz]]

Those are already handled in schema-generator: https://github.com/mozilla/mozilla-schema-generator/blob/863aa6ead293e73ab5af812d44467d06c8ddd137/mozilla_schema_generator/common_ping.py#L49

The reason why they still show as missing seems to be that schemas haven't been deployed in quite a while.

Comment 10

[Anna Scholtz [:ascholtz]]

Attachment: GitHub Pull Request

Comment 11

[Anna Scholtz [:ascholtz]]

Attachment: GitHub Pull Request