1472781 - Make in-progress telemetry available to filter expressions

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement, P1)

Description

[Michael Cooper [:mythmon]]

Filter expressions have access to Telemetry data. Currently this is based on archived telemetry pings.

The choice to use archived pings was out of concern for "incomplete" telemetry being harder to work with. In practice, incomplete telemetry is not an issue. 

What is an issue is access to Telemetry during Normandy's "isFirstRun" executions. These executions happen the first time Normandy runs on a new profile, which is too early for an archived Telemetry ping to be reliably available.

This bug covers reading telemetry data from the in-progress ping when an archived ping is not available.

Comment 2

[Rosanne Scholl]

If I understand the proposed change correctly, this is very useful to be able to reach first-run users more precisely. And it has an even more intriguing use case when it comes to research purposes: being able to offer surveys or other interventions at more precise points in time.... for example, a survey to ask users what they thought of protection-report right after they saw it; or to ask about notification fatigue after a variety of different notification types to see which one is most interruptive. TL;dr: research case is not just new users, but instead all live events (IIUC)

Comment 3

[Michael Cooper [:mythmon]]

Since events are in a different physical ping now, access to live events won't be included in this change. This is something we can explore in another bug if that will be helpful.

Comment 4

[Michael Cooper [:mythmon]]

Attachment: Bug 1472781 - Make in-progress telemetry available to filter expressions as environment.liveTelemetry r=Gijs!

Comment 5

[Chris H-C :chutten]

As Lead for Firefox Telemetry, I have opinions about reading data from Telemetry. Is this the right time and place to voice those opinions, or have I missed the design pass?

Comment 6

[Michael Cooper [:mythmon]]

Normandy has a long history of reading things from archived telemetry pings. It has been a part of our targeting capabilities since very early in the project. The specific problem we're working on here is when that Telemetry is available, since we can't read from archived pings before any pings have been archived, tautologically.

The first thing, reading from Telemetry at all, is going to be pretty tough to change, but we can take a look.

The second thing, having access to Telemetry (or at least the kind of information available in Telemetry) is definitely still up for design, and I'd be happy to get your opinion about this. Sorry if you feel left out of the loop here, I probably should have reached out sooner :)

As a bit of context, I think (but should verify) the things we use from Telemetry are mostly from the Environment section of the ping. If that change in scope makes any changes to your opinions, then we could also explore a more limited set of access.

Comment 7

[Chris H-C :chutten]

Hey, no worries. My job's usually to make things easier for people, not to get in the way, so I'm working against my own grain here : )

Generally speaking, Telemetry is designed to be write-only. Outside of tests, we don't support any mechanism for retrieving data that's been accumulated (whether it's been sent or not). That the Archive and getCurrentPingData (and the testing APIs like getSnapshot) are fairly stable is a lucky accident more than evidence of ongoing maintenance.

Put another way, Telemetry is not and does not want to become a local data broker for storing and serving a model of the interesting points of a Firefox profile. For a concrete example of why: in the not-too-distant future we intend to replace Telemetry with the Glean SDK in Firefox Desktop. We hadn't even considered maintaining API compatibility as the Glean SDK is designed in many ways to be unable to support some of them.

This doesn't close the door on a local data capability that does serve this function with a stable API and lovely documentation and tooling, but that's something that hasn't even yet been proposed, let alone designed and resourced.

Where that leaves us is a bit of a pickle. We all have work to do, so let's get you unblocked first, and then we can think about intended design and future plans later. If all you need is the Environment, then we're in luck that the Environment is only a thin organization placed atop direct APIs you can call. If we know what the specific points are that are of interest, we can get this exposed quickly and confidently. Would that serve the purposes here? Or is the environment itself needed because the organization of the data is of value too?

Comment 8

[Michael Cooper [:mythmon]]

If all you need is the Environment, then we're in luck that the Environment is only a thin organization placed atop direct APIs you can call. If we know what the specific points are that are of interest, we can get this exposed quickly and confidently. Would that serve the purposes here? Or is the environment itself needed because the organization of the data is of value too?

Using Telemetry directly gave us a couple benefits

1. People running experiments are often familiar with Telemetry, and so it is familiar territory.
2. As Telemetry gets better we get improvements for free.
3. It gave us a lot of interesting properties of the browser which we knew were "safe" in regards to letting Mozilla use this data.

Looking more closely at the recipes that exist in Normandy's history, I think I misestimated how much we use non-environment telemetry. I got these stats for the number of accesses to Telemetry in filter expressions. This is the number of times each type of access happened, not the number of recipes it appeared in.

• Accessing only the environment: 231 (75%)
• Accessing histograms: 51 (16%)
• Accessing scalars: 12 (4%)
• Checking if a main ping has been archived: 11 (4%)
• Accessing keyed scalars: 3 (1%)
• Accessing the crash ping: 2 (1%)

I happen to know that the use case for live telemetry could be satisfied today from the environment, so maybe we should do that in this bug. Additionally being more specific about the data that we pull in could be useful, instead of blindly delegating to telemetry.

Josh, Rosanne, what do you think about all this?

Mat, do you see any of this affecting those that use filter expressions in Remote Settings?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/58c8c1819122
Make in-progress telemetry available to filter expressions as environment.liveTelemetry r=Gijs,chutten

Comment 10

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/58c8c1819122