1624617 - I don't know where to find the gfx.os_compositor scalar in moz-fx-data-shared-prod.telemetry.main

Status: RESOLVED FIXED

(Data Platform and Tools :: General, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

SELECT 
       DATE(submission_timestamp) AS submission_date,
       TO_JSON_STRING(payload.processes.parent.scalars.gfx_os_compositor) as compositor
FROM `moz-fx-data-shared-prod.telemetry.main`
WHERE DATE(submission_timestamp) = DATE "2020-03-22"
  AND normalized_channel = 'nightly'
  AND sample_id = 42
ORDER BY submission_date limit 5

fails with:

Error running query: Field name gfx_os_compositor does not exist in STRUCT<a11y_indicator_acted_on BOOL, a11y_instantiators STRING, aushelper_websense_reg_version STRING, ...>; Did you mean gfx_compositor? at [3:56]

Comment 1

[Jeff Muizelaar [:jrmuizel]]

I added this scalar in https://hg.mozilla.org/mozilla-central/rev/f985871b3e62

Comment 2

[Jeff Klukas [:klukas] (UTC-4)]

As this is a new probe in FF 76, we likely have not yet picked up and deployed schema changes to accept the new probe. We generally do a schema deploy once per day, but we have been fighting pipeline stability issues over the past week that and we have been reticent to make any unnecessary changes until we are back to stability.

In the meantime, the value of the new probe should be making its way to the additional_properties JSON blob, so you can at least verify client behavior that way.

Can you comment on the level of impact for not having this probe available to query as a field? If not a major issue, it may be a week before we get back to normal schema deploys.

Comment 3

[Jeff Muizelaar [:jrmuizel]]

This probe is also in 75 and we're looking to verify the values in beta before we go to release so there's some urgency here. In the mean time I'll see if I can get the information I need to from additional_properties.

Comment 4

[Jeff Muizelaar [:jrmuizel]]

We were able to get the info we needed from additional_properties so there's no urgency from us on this anymore.

Comment 5

[Jeff Klukas [:klukas] (UTC-4)]

I'm glad you were able to get what you need. I'm going to go ahead and close this, then. The field should show up by early next week. Note that once the field exists, new pings flowing in will have the field populated and it will no longer appear in additional_properties. You'll need to do a COALESCE(payload.processes.parent.scalars.gfx_os_compositor, JSON_EXTRACT(additional_properties, '$.payload....')) if you want continuity across the boundary.