1284652 - Collect telemetry from system add-on for out-of-date notifications

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Description

[Molly Howell (she/her) (no longer active)]

As part of bug 1280378 we're adding a new unified telemetry ping type that needs to be ingested and made available for analysis. A custom ping is needed because this is part of a system add-on which will be installed in old browser versions, so just adding a new histogram is not possible.

To summarize, with the current code the pings are sent using submitExternalPing() with type "outofdate-notifications-system-addon", clientID enabled, and the payload:

{
  shown: <bool>,
  clicked: <bool>
}

See [1] for the documentation and code. bsmedberg has reviewed those materials, so I think the data collection policy requirements are satisfied. I've never done this before; not sure what else is needed.

[1] https://github.com/mozilla/outofdate-notifications-system-addon

Comment 1

[Katie Parlante]

Hi Matt,

Here are a few steps to consider...

- Its probably a good idea to get a review of the new custom ping from one of the "Firefox Telemetry" owner/peers. I'll ?gfritzsche to take a look here.

- The pings will show up in the "other" bucket by default. If the volume is going to be high enough, we may want some pipeline config/setup to move the pings to their own bucket.

- If you want to analyze the pings in re:dash, you'll need a job to make them available. (If/while the volume of pings is small, you can look at them in Spark fairly easily).

Katie

Comment 2

[Georg Fritzsche [:gfritzsche]]

How often will the ping be sent for each user session?
1-3 times per session (start, shown, clicked)?
Or can the dialog be shown more than once per session?

Do we have an estimate of how many users will send this (protected or public)?

Could you actually ever get a `{shown:false,clicked:true}` submission?
It seems like the natural form would be more like `{event:"started"|"shown"|"clicked"}`?

In general, we should get pings out of the "other" bucket to efficiently access them. :mreid can help with that.

Comment 3

[Molly Howell (she/her) (no longer active)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> How often will the ping be sent for each user session?
> 1-3 times per session (start, shown, clicked)?
This is correct; the dialog is only shown on startup.

> Do we have an estimate of how many users will send this (protected or
> public)?
Redirecting this question to spohl; he can get those numbers easier than I can.

> Could you actually ever get a `{shown:false,clicked:true}` submission?
> It seems like the natural form would be more like
> `{event:"started"|"shown"|"clicked"}`?
Right, {shown:false,clicked:true} wouldn't happen. I'd rather not go back and change it now unless there would be a big advantage to doing so, but I'll keep this in mind for the future (again, never had to add new pings before).

Comment 4

[Stephen A Pohl [:spohl]]

Georg, just adding an n-i to get your eyes on the additional feedback below. Please let me know if there are any concerns due to the number of pings that we're going to send (or anything else).

(In reply to Matt Howell [:mhowell] from comment #3)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > How often will the ping be sent for each user session?
> > 1-3 times per session (start, shown, clicked)?
> This is correct; the dialog is only shown on startup.

mhowell and I discussed this and it actually turns out that the notification is shown on every newly opened tab and window, so the frequency can be significantly higher.

> > Do we have an estimate of how many users will send this (protected or
> > public)?
> Redirecting this question to spohl; he can get those numbers easier than I
> can.

The most recent analysis using the 1% longitudinal dataset shows a total number of 27,873 users on Firefox 44 with updates enabled, so it would be reasonable to expect about 2,787,300 users to download this system addon and send pings.

> > Could you actually ever get a `{shown:false,clicked:true}` submission?
> > It seems like the natural form would be more like
> > `{event:"started"|"shown"|"clicked"}`?
> Right, {shown:false,clicked:true} wouldn't happen. I'd rather not go back
> and change it now unless there would be a big advantage to doing so, but
> I'll keep this in mind for the future (again, never had to add new pings
> before).

We have to respin the addon for unrelated reasons, so I'll go ahead and make this change and ask for your review once it's ready.

(In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> In general, we should get pings out of the "other" bucket to efficiently
> access them. :mreid can help with that.

:mreid, mind telling me what we should do here? As mentioned above, I'll be changing the telemetry code to address Georg's feedback, but here's a link[1] to what we have now. I'd like to address everything at once.

[1] https://github.com/mozilla/outofdate-notifications-system-addon/commit/0f9abf4542a4bf87161ed701bdb51293915174a8

Comment 5

[Georg Fritzsche [:gfritzsche]]

(In reply to Stephen A Pohl [:spohl] from comment #4)
> (In reply to Matt Howell [:mhowell] from comment #3)
> > (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > > How often will the ping be sent for each user session?
> > > 1-3 times per session (start, shown, clicked)?
> > This is correct; the dialog is only shown on startup.
> 
> mhowell and I discussed this and it actually turns out that the notification
> is shown on every newly opened tab and window, so the frequency can be
> significantly higher.

This sounds like a potentially rather big volume. In general, unbounded event-driven ping submission is not something we should do.
Is there any need to submit more than one "shown" or "clicked" event per a session?
Or are there other submission limits you can apply here?

A related question - what kind of analysis are you planning to do on the data?
Will you compare overall "started"/"shown"/"clicked" numbers?
Or will you ask questions like "how many of the clients that got the notification shown clicked it"?
Or are you planning to look at event sequences in more detail?
We don't necessarily receive pings on the server-side in the order they were generated on the client-side.

> > > Do we have an estimate of how many users will send this (protected or
> > > public)?
> > Redirecting this question to spohl; he can get those numbers easier than I
> > can.
> 
> The most recent analysis using the 1% longitudinal dataset shows a total
> number of 27,873 users on Firefox 44 with updates enabled, so it would be
> reasonable to expect about 2,787,300 users to download this system addon and
> send pings.

Given the findings on ping triggers above, can we estimate how many pings each of the users will send?

> (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > In general, we should get pings out of the "other" bucket to efficiently
> > access them. :mreid can help with that.
> 
> :mreid, mind telling me what we should do here? As mentioned above, I'll be
> changing the telemetry code to address Georg's feedback, but here's a
> link[1] to what we have now. I'd like to address everything at once.
> 
> [1]
> https://github.com/mozilla/outofdate-notifications-system-addon/commit/
> 0f9abf4542a4bf87161ed701bdb51293915174a8

There is pipeline work to do which is relatively simple, but we should have a volume estimate for the pings first and settle the other questions.

Comment 6

[Stephen A Pohl [:spohl]]

Attachment: Fix telemetry

(In reply to Georg Fritzsche [:gfritzsche] from comment #5)
> (In reply to Stephen A Pohl [:spohl] from comment #4)
> > (In reply to Matt Howell [:mhowell] from comment #3)
> > > (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > > > How often will the ping be sent for each user session?
> > > > 1-3 times per session (start, shown, clicked)?
> > > This is correct; the dialog is only shown on startup.
> > 
> > mhowell and I discussed this and it actually turns out that the notification
> > is shown on every newly opened tab and window, so the frequency can be
> > significantly higher.
> 
> This sounds like a potentially rather big volume. In general, unbounded
> event-driven ping submission is not something we should do.
> Is there any need to submit more than one "shown" or "clicked" event per a
> session?
> Or are there other submission limits you can apply here?

I've tried to change this so that we only send each event once per session at most. I'm sending a string ("started"|"shown"|"clicked") as ping payload now, rather than an object in the form of { event: "started" }. Is this ok, or do we need to send this as an object?

> A related question - what kind of analysis are you planning to do on the
> data?
> Will you compare overall "started"/"shown"/"clicked" numbers?
> Or will you ask questions like "how many of the clients that got the
> notification shown clicked it"?
> Or are you planning to look at event sequences in more detail?
> We don't necessarily receive pings on the server-side in the order they were
> generated on the client-side.

I think there are different things we can analyze, but off the top of my head I'd like to see:
1. How many times does a user click the button but still has subsequent sessions where the addon is shown (i.e. even though the "Get Firefox" button was clicked, the user either didn't download the new version or keeps using the old version even though a new one was installed).
2. Do users become used to seeing the notification and stop clicking the button?

None of these require that we send a "shown" ping for every opened tab or window, so I think it is fine to switch to once per session pings for each event.
 
> > > > Do we have an estimate of how many users will send this (protected or
> > > > public)?
> > > Redirecting this question to spohl; he can get those numbers easier than I
> > > can.
> > 
> > The most recent analysis using the 1% longitudinal dataset shows a total
> > number of 27,873 users on Firefox 44 with updates enabled, so it would be
> > reasonable to expect about 2,787,300 users to download this system addon and
> > send pings.
> 
> Given the findings on ping triggers above, can we estimate how many pings
> each of the users will send?

I could go back and see how many sessions users on Firefox 44 report for a particular period of time. By switching to once per session submissions, do you still want me to run this? If yes, what period of time should I choose? 1 week?
 
> > (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > > In general, we should get pings out of the "other" bucket to efficiently
> > > access them. :mreid can help with that.
> > 
> > :mreid, mind telling me what we should do here? As mentioned above, I'll be
> > changing the telemetry code to address Georg's feedback, but here's a
> > link[1] to what we have now. I'd like to address everything at once.
> > 
> > [1]
> > https://github.com/mozilla/outofdate-notifications-system-addon/commit/
> > 0f9abf4542a4bf87161ed701bdb51293915174a8
> 
> There is pipeline work to do which is relatively simple, but we should have
> a volume estimate for the pings first and settle the other questions.

Where does this pipeline work occur? Anything I need to do here?

Comment 7

[Georg Fritzsche [:gfritzsche]]

(In reply to Stephen A Pohl [:spohl] from comment #6)
> Created attachment 8777867 [details] [review]
> Fix telemetry
> 
> (In reply to Georg Fritzsche [:gfritzsche] from comment #5)
> > (In reply to Stephen A Pohl [:spohl] from comment #4)
> > > (In reply to Matt Howell [:mhowell] from comment #3)
> > > > (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > > > > How often will the ping be sent for each user session?
> > > > > 1-3 times per session (start, shown, clicked)?
> > > > This is correct; the dialog is only shown on startup.
> > > 
> > > mhowell and I discussed this and it actually turns out that the notification
> > > is shown on every newly opened tab and window, so the frequency can be
> > > significantly higher.
> > 
> > This sounds like a potentially rather big volume. In general, unbounded
> > event-driven ping submission is not something we should do.
> > Is there any need to submit more than one "shown" or "clicked" event per a
> > session?
> > Or are there other submission limits you can apply here?
> 
> I've tried to change this so that we only send each event once per session
> at most. I'm sending a string ("started"|"shown"|"clicked") as ping payload
> now, rather than an object in the form of { event: "started" }. Is this ok,
> or do we need to send this as an object?

Great, with "at most once per session for each of the 3 events" and the discussed payload and population, i don't see a volume issue.
The payload needs to be a JSON style object - if we accept non-object payloads we'll have to add hardening here.

> > A related question - what kind of analysis are you planning to do on the
> > data?
> > Will you compare overall "started"/"shown"/"clicked" numbers?
> > Or will you ask questions like "how many of the clients that got the
> > notification shown clicked it"?
> > Or are you planning to look at event sequences in more detail?
> > We don't necessarily receive pings on the server-side in the order they were
> > generated on the client-side.
> 
> I think there are different things we can analyze, but off the top of my
> head I'd like to see:
> 1. How many times does a user click the button but still has subsequent
> sessions where the addon is shown (i.e. even though the "Get Firefox" button
> was clicked, the user either didn't download the new version or keeps using
> the old version even though a new one was installed).
> 2. Do users become used to seeing the notification and stop clicking the
> button?
> 
> None of these require that we send a "shown" ping for every opened tab or
> window, so I think it is fine to switch to once per session pings for each
> event.

Ok, between submission data and the pings creation date, those should be fine to answer from the data.

> > > > > Do we have an estimate of how many users will send this (protected or
> > > > > public)?
> > > > Redirecting this question to spohl; he can get those numbers easier than I
> > > > can.
> > > 
> > > The most recent analysis using the 1% longitudinal dataset shows a total
> > > number of 27,873 users on Firefox 44 with updates enabled, so it would be
> > > reasonable to expect about 2,787,300 users to download this system addon and
> > > send pings.
> > 
> > Given the findings on ping triggers above, can we estimate how many pings
> > each of the users will send?
> 
> I could go back and see how many sessions users on Firefox 44 report for a
> particular period of time. By switching to once per session submissions, do
> you still want me to run this? If yes, what period of time should I choose?
> 1 week?

I think the volume will be fine with the new limitations, unless mreid needs more concrete numbers.

> > > (In reply to Georg Fritzsche [:gfritzsche] from comment #2)
> > > > In general, we should get pings out of the "other" bucket to efficiently
> > > > access them. :mreid can help with that.
> > > 
> > > :mreid, mind telling me what we should do here? As mentioned above, I'll be
> > > changing the telemetry code to address Georg's feedback, but here's a
> > > link[1] to what we have now. I'd like to address everything at once.
> > > 
> > > [1]
> > > https://github.com/mozilla/outofdate-notifications-system-addon/commit/
> > > 0f9abf4542a4bf87161ed701bdb51293915174a8
> > 
> > There is pipeline work to do which is relatively simple, but we should have
> > a volume estimate for the pings first and settle the other questions.
> 
> Where does this pipeline work occur? Anything I need to do here?

mreid can whitelist the ping type here, this can probably just happen in this bug.

Comment 8

[Stephen A Pohl [:spohl]]

Comment on attachment 8777867 [details] [review]
Fix telemetry

Thanks, Georg! Addressed feedback and updated PR. Will rebase before merging, but left it as individual commits to see diffs more easily.

Comment 9

[Georg Fritzsche [:gfritzsche]]

Comment on attachment 8777867 [details] [review]
Fix telemetry

This looks good to me from the Telemetry ping submission perspective.
I haven't taken a detailed look and might be missing context on the addon, hence f+.

Comment 10

[Stephen A Pohl [:spohl]]

Discussed with Georg via IRC. The other parts of the pull request have already been reviewed in bug 1291634, so Georg's f+ is an r+ for the telemetry parts and we can merge the PR. Thanks again!

Comment 11

[Stephen A Pohl [:spohl]]

https://github.com/mozilla/outofdate-notifications-system-addon/pull/8 has been merged. New addon is being signed and uploaded in bug 1292243.

Comment 12

[Stephen A Pohl [:spohl]]

I see that seemingly one .jsonlz4 file is created for each event, i.e. three maximum per session. They get created again when I start a new session. Opening new tabs etc. does not create new files, which I think shows that we're abiding by our rule of only generating three pings maximum per session.

I haven't been able to decompress the .jsonlz4 files however, and about:telemetry does not show any pings for our system addon. I'm going to attach the three files from one session. Benjamin, do you know how to decompress these to confirm that they correspond to each of our three events?

Comment 13

[Stephen A Pohl [:spohl]]

Attachment: First event

Comment 14

[Stephen A Pohl [:spohl]]

Attachment: Second event

Comment 15

[Stephen A Pohl [:spohl]]

Attachment: Third event

Comment 16

[Stephen A Pohl [:spohl]]

I added the expected event to the title of each uploaded file, but the timestamps for the three files may not be precise enough to accurately put them in sequence of "started" -> "shown" -> "clicked". I think as long as all three events are present (one per file), I believe we've confirmed that telemetry for the addon is working as expected.

Comment 17

[Georg Fritzsche [:gfritzsche]]

(In reply to Stephen A Pohl [:spohl] from comment #12)
> I haven't been able to decompress the .jsonlz4 files however, and
> about:telemetry does not show any pings for our system addon. I'm going to
> attach the three files from one session. Benjamin, do you know how to
> decompress these to confirm that they correspond to each of our three events?

You can e.g. use Avis tool: https://github.com/avih/dejsonlz4

about:telemetry should show your pings though - switch to "Archived ping data" and choose "week" and "ping" appropriately.

Comment 18

[Georg Fritzsche [:gfritzsche]]

I'm moving the whitelisting of the ping type to a new bug as this one was closed and had some different discussions.

Comment 19

[Georg Fritzsche [:gfritzsche]]

The documentation here needs updating for the changed payload:
https://github.com/mozilla/outofdate-notifications-system-addon

You should also add the new ping trigger conditions to the documentation.

Comment 20

[Stephen A Pohl [:spohl]]

Comment on attachment 8778034 [details]
First event

{"type":"outofdate-notifications-system-addon","id":"09ba3a72-7289-0a48-ab14-216557eae784","creationDate":"2016-08-05T00:41:11.516Z","version":4,"application":{"architecture":"x86-64","buildId":"20160210153822","name":"Firefox","version":"44.0.2","vendor":"Mozilla","platformVersion":"44.0.2","xpcomAbi":"x86_64-gcc3","channel":"release"},"payload":{"event":"shown"},"clientId":"6d9a2e7e-0bb6-9145-a83f-abe803f142dc"}

Comment 21

[Stephen A Pohl [:spohl]]

Comment on attachment 8778035 [details]
Second event

{"type":"outofdate-notifications-system-addon","id":"4d26a2c1-e899-2a48-95d0-e997028fa434","creationDate":"2016-08-05T00:41:10.630Z","version":4,"application":{"architecture":"x86-64","buildId":"20160210153822","name":"Firefox","version":"44.0.2","vendor":"Mozilla","platformVersion":"44.0.2","xpcomAbi":"x86_64-gcc3","channel":"release"},"payload":{"event":"started"},"clientId":null}

Comment 22

[Stephen A Pohl [:spohl]]

Comment on attachment 8778036 [details]
Third event

{"type":"outofdate-notifications-system-addon","id":"afc7af34-9f09-6a4a-a3ab-0695afb4b543","creationDate":"2016-08-05T00:41:15.976Z","version":4,"application":{"architecture":"x86-64","buildId":"20160210153822","name":"Firefox","version":"44.0.2","vendor":"Mozilla","platformVersion":"44.0.2","xpcomAbi":"x86_64-gcc3","channel":"release"},"payload":{"event":"clicked"},"clientId":"6d9a2e7e-0bb6-9145-a83f-abe803f142dc"}

Comment 23

[Stephen A Pohl [:spohl]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #17)
> (In reply to Stephen A Pohl [:spohl] from comment #12)
> > I haven't been able to decompress the .jsonlz4 files however, and
> > about:telemetry does not show any pings for our system addon. I'm going to
> > attach the three files from one session. Benjamin, do you know how to
> > decompress these to confirm that they correspond to each of our three events?
> 
> You can e.g. use Avis tool: https://github.com/avih/dejsonlz4

Thank you, that worked!

> about:telemetry should show your pings though - switch to "Archived ping
> data" and choose "week" and "ping" appropriately.

This does not seem to show these pings in Firefox 44.0.2. I can only select main or saved-session pings.

Comment 24

[Stephen A Pohl [:spohl]]

(In reply to Georg Fritzsche [:gfritzsche] from comment #19)
> The documentation here needs updating for the changed payload:
> https://github.com/mozilla/outofdate-notifications-system-addon
> 
> You should also add the new ping trigger conditions to the documentation.

Will update today. Keeping n-i set for this. Thanks!

Comment 25

[Georg Fritzsche [:gfritzsche]]

(In reply to Stephen A Pohl [:spohl] from comment #23)
> > about:telemetry should show your pings though - switch to "Archived ping
> > data" and choose "week" and "ping" appropriately.
> 
> This does not seem to show these pings in Firefox 44.0.2. I can only select
> main or saved-session pings.

I see, showing them was only added in Firefox 47+ (bug 1232917 et al).

Comment 26

[waxmigs2902]

[Tracking Requested - why for this release]:

Comment 27

[Stephen A Pohl [:spohl]]

Attachment: Update README.md

Comment 28

[Stephen A Pohl [:spohl]]

(In reply to waxmigs2902 from comment #26)
> [Tracking Requested - why for this release]:

Don't set bugzilla flags at random.

Comment 29

[Stephen A Pohl [:spohl]]

Comment on attachment 8778476 [details] [review]
Update README.md

Updated PR based on feedback. Sending back for final review to make sure I understood your feedback correctly. Thanks!