1353105 - Automatically Add All Scalars to main_summary

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Frank Bertsch [:frank]]

To support heavy user analysis, they need all engagement scalars. We might as well add *all* scalars automatically, to reduce data engineering workload down the line.

Comment 1

[Frank Bertsch [:frank]]

Scalars are per-process, and per-namespace. Current scalars in main_summary are just the scalar name: e.g. "max_concurrent_tab_count". I propose making them like longitudinal, where the name is: <scalar_prefix>_<process>_<namespace>_<name>, e.g. "scalars_parent_browser_engagement_max_concurrent_tab_count". Downside is the previous names won't be available, so we'd probably have to version main_summary so that people can still access them in the historical data (I'm assuming backfill is out of the question). Thoughts, Mark?

Comment 2

[Mark Reid [:mreid]]

I think this is a good idea. We'll likely want to keep adding all the new scalars as they arrive, so making the process as easy as possible makes sense.

When we version to v4, we could rewrite the v3 main_summary data into v4 using the new column names (rather than backfilling from the raw data, which is quite time-consuming), then users of main_summary would not have to query both tables.

Comment 3

[Frank Bertsch [:frank]]

Will be backfilled with bug 1362161