1775216 - Add Glean data to smoothed dataset

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement)

Description

[Romain Testard [:RT]]

Per discussion with Alessio the Glean data is currently not reported in the smoothed dataset, even though the source data is there in nondesktop client (Product = 'Other', app_name = 'Focus Android Baseline'/'Focus iOS Baseline').
Data from product 'Other' is filtered out here: https://github.com/mozilla/bigquery-etl/blob/98549e3cb8b4f81d03a08a33aedaf6687ea5f[…]ared-prod/telemetry_derived/smoot_usage_nondesktop_v2/query.sql

I need this to work out new and existing user retention on Focus given legacy telemetry is not reliable there.

Comment 1

[Alessio Placitelli [:Dexter]]

Note: this bug was filed as a follow up of this slack discussion and the smoothed dataset we were talking about was prod.telemetry.smoot_usage_day_13.

Comment 2

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/3040

Comment 3

[Anna Scholtz [:ascholtz]]

https://github.com/mozilla/bigquery-etl/pull/3040 adds Klar and Focus Glean data to the smoot tables. The app_name for this data will include "Baseline", so for example it will be Focus Android Baseline.

We'll need to backfill the smoot_ tables for this. @romain how far should I backfill?

Comment 4

[Romain Testard [:RT]]

Can we go back to 2018 included? One use case is understanding if the binary size shift Focus saw in 2018 had an impact on new profiles and newprofile retention to inform further increases.?

Comment 5

[Anna Scholtz [:ascholtz]]

Alright, I'll backfill telemetry_derived.smoot_usage_nondesktop_compressed_v2 and telemetry_derived.smoot_usage_nondesktop_v2 starting 2018-01-01. I did a quick cost check for the backfill and it's pretty low.

Comment 6

[Anna Scholtz [:ascholtz]]

The backfill has completed and data should be available now, e.g:

select * from telemetry.smoot_usage_day_13 
where app_name like "%Baseline%" and date = '2021-06-21' 

Comment 7

[Romain Testard [:RT]]

Thank you!