1603639 - Discuss adding `struct` and `struct_list` metric types

Status: RESOLVED INCOMPLETE

(Data Platform and Tools :: Glean: SDK, enhancement, P3)

Description

[Lina Butler [:lina]]

This is probably the biggest pain point we ran into while integrating Glean and Sync. It's discussed more in depth in the feedback doc, but, the long and short of it is, the lack of struct types means we end up with a conceptual mismatch between how we collect and report data, and how Sync actually works.

Some of the problems we've run into include duplicating an identical, or mostly-identical, ping schema for every data type we sync, sending pings immediately after each sync to clear out data from the last engine, losing a level of per-engine granularity because we smush per-engine and global data into the same ping, sending too many pings (one per engine per sync, rather than one per sync or one per multiple syncs), and a cumbersome workflow to add new engines, engine steps, and validation problems.

We don't believe that this approach will scale as we add new synced data types, and eventually transition the Desktop Sync ping to Glean. That's the only hard deadline we can think of for this.

These challenges are further explained in the doc. Additionally, there's a tongue-in-cheek comment in a-c hinting at some of these challenges.

We propose two new data types to support these use cases:

• A struct has one or more fields, whose names and types are declared in metrics.yaml. This is similar to how the labeled_* metrics work today.
• Unlike labels, there's no default __other__ label. It's a compile-time error to use a struct field that's not defined in metrics.yaml.
• Structs can nest, so a struct can have another struct or struct_list as a field.
• A struct_list is composed of zero or more structs of that type. Lists are homogeneous; it's not possible to have different types of structs in the same list.
• It would also be helpful to have a convenient syntax for declaring types or type aliases. That way, a struct type like, say, named_count (with name and count fields), could be reused multiple times in the same ping or data type.

(I know there isn't a process for suggesting new metric types yet, but, if you're open to it, maybe we could use it as a test case? 😁)

Comment 1

[Alessio Placitelli [:Dexter]]

(In reply to :Lina Cambridge from comment #0)

This is probably the biggest pain point we ran into while integrating Glean and Sync. It's discussed more in depth in the feedback doc, but, the long and short of it is, the lack of struct types means we end up with a conceptual mismatch between how we collect and report data, and how Sync actually works.

As additional context, here's where the decision was made.

These challenges are further explained in the doc. Additionally, there's a tongue-in-cheek comment in a-c hinting at some of these challenges.

I've scheduled a meeting for going through the document with your team.

(I know there isn't a process for suggesting new metric types yet, but, if you're open to it, maybe we could use it as a test case? 😁)

Once the process is finalized, we'll definitely be able to get through it for this one as well, if you feel like it.

Comment 2

[Alessio Placitelli [:Dexter]]

Lina, we now have a process for adding new metric types or changing existing ones.

I'm closing this as a bug needs to be filed in another component and there's a template to do so. Please follow the linked guide to do so, if this is still relevant. Thank you for your patience.