1690390 - Add telemetry to track when a notification bar is shown

Status: RESOLVED FIXED

(Firefox :: Messaging System, task, P3)

Description

[Mark Striemer [:mstriemer]]

We'd like to know how often users see a notification bar, which type of notification bar it is, and what action the users takes on the notification bar.

It looks like this telemetry could all be added to the widget itself in toolkit/content/widgets/notificationbox.js.

Comment 1

[Blake Winton (:bwinton) (:☕️)]

As per guidance from Vicky, for tracking, we're marking all the bugs that people are working on as P1.

Comment 2

[Neil Deakin]

I have a mostly working implmentation using a set of scalars. But what specific information is needed here. What

Currently, I have a set of scalar telemetry keys that can be added for each infobar:

• a count of how many times that infobar is shown to the user
• a count of how many times a particular button is pressed
• a count of how many times a particular notification is dismissed with the close (X) button

Telemetry keys would be added for each notification type separately, and each of the three types may be supplied, but optional.

Or, do we want to measure something more specific about how the infobar is used?

Who is going to be analyzing this data -- the telemetry requires a email address.

Comment 3

[Mark Striemer [:mstriemer]]

That sounds right to me.

Romain, does this seem like it will meet your needs? Are you going to be the one monitoring this data?

Comment 4

[Neil Deakin]

Just to clarify as well, one or more separate telemetry keys will be needed for each infobar, and that will need to go through a data review ( https://github.com/mozilla/data-review/blob/main/request.md ) before being added.

If you want a simple counts (listed in comment 2), then the patch I will attach soon will handle this, but if you want more detail, such as the time between events, I suggest working with someone on the data collection team to clarify what is needed.

Comment 5

[Romain Testard [:RT]]

(In reply to Mark Striemer [:mstriemer] from comment #3)

That sounds right to me.

Romain, does this seem like it will meet your needs? Are you going to be the one monitoring this data?

Yes this captures the need.
You can provide my email regarding data analysis yes.

Comment 6

[Neil Deakin]

Attachment: Bug 1690390, change appendNotification to take a smaller number of arguments: the string type value, a dictionary containing the notification details, and the button list, r=mstriemer

Comment 7

[Neil Deakin]

Attachment: Bug 1690390, add ability to perform telemetry when a notification bar is shown, a button is pressed, or is dismissed. Use this for the subframe crash notification bar which currently does this a different way, r=mstriemer

Depends on D107755

Comment 8

[Neil Deakin]

The patches above add a general mechanism for notification bars to handle telemetry. It does not add any new telemetry, apart from a minor change to the existing subframe crash one to change to be a keyed scalar. Each bar would need to add telemetry separately by adding a 'telemetry' property when calling appendNotification.

An alternative approach I considered was to instead add a mechanism to have a single keyed scalar for all notification bars with a key for each type of bar. This would mean that telemetry would be add for all bars without extra work, but I opted not to do this way because a) each bar has different buttons, and it didn't seem possible to add telemetry for that without handling it for each notification bar, and b) I thought that we should be more precise in specifying what data we collect from users.

Comment 9

[Romain Testard [:RT]]

Marking as P2. Per experience review we agreed to mark as P1 bugs only bugs that will block MR1.

Comment 10

[Mark Striemer [:mstriemer]]

Romain, what are you hoping to answer here? In comment 8 Neil mentions that he hasn't added any new telemetry, but rather added support for adding telemetry to specific infobars.

Are there specific infobars you're interested in that we should add the telemetry for, or were you hoping for something more generic tracking all infobars?

Comment 11

[Tyler Duzan]

(In reply to Mark Striemer [:mstriemer] from comment #10)

Romain, what are you hoping to answer here? In comment 8 Neil mentions that he hasn't added any new telemetry, but rather added support for adding telemetry to specific infobars.

Are there specific infobars you're interested in that we should add the telemetry for, or were you hoping for something more generic tracking all infobars?

I believe the expectation here is that each infobar would be wired individually for telemetry so that we can get some handle on how often specific infobars occur and how users interact with them. Many of the infobars are triggered when a user encounters a problem and having an understanding of how often those sorts of problems occur in the wild can help us to improve the user experience to minimize or eliminate problem occurence.

This doesn't block shipping, but I think that addresses the question.

Comment 12

[Dan Mosedale (:dmosedale, :dmose)]

There's some amount of overlap here with the existing messaging system infobar infrastructure (which, as I understand it, uses the existing notification bar under the hood). :andreio, am I misunderstanding how things work? I'd love to hear your thoughts...

Comment 13

[Neil Deakin]

The current patch (just updated) will add telemetry for three situations related to an infobar:

• when a specific infobar is displayed ('shown')
• when it is dismissed by pressing the close button ('dismissed')
• when some other button or link is pressed, but by default this does not distinguish which button ('action'). A specific infobar can specify a telemetry key for each button to separate this out.

Each is a count of the number of times that action occurs. The telemetry is only triggered when
the infobar is shown to the user, so an infobar in a hidden tab does not invoke telemetry until
the user switches to that tab. If the user does not interact with the infobar at all, then the
'shown' telemetry will increase but the other values will not.

Comment 14

[Romain Testard [:RT]]

(In reply to Neil Deakin from comment #13)

The current patch (just updated) will add telemetry for three situations related to an infobar:

• when a specific infobar is displayed ('shown')
• when it is dismissed by pressing the close button ('dismissed')
• when some other button or link is pressed, but by default this does not distinguish which button ('action'). A specific infobar can specify a telemetry key for each button to separate this out.

Each is a count of the number of times that action occurs. The telemetry is only triggered when
the infobar is shown to the user, so an infobar in a hidden tab does not invoke telemetry until
the user switches to that tab. If the user does not interact with the infobar at all, then the
'shown' telemetry will increase but the other values will not.

This is great, is there a list of the specific infobars covered somewhere please?

Comment 15

[Neil Deakin]

Only the subframe crash infobar, added by bug 1644911, currently has telemetry assigned for it. The patches in this bug update this to follow the pattern as described in comment 13.

Telemetry can be added for other infobars can be added with a one-line change to where the notification is displayed, plus the telemetry info to Scalars.yaml.

Comment 16

[Neil Deakin]

Comment on attachment 9207948 [details]
Bug 1690390, add ability to perform telemetry when a notification bar is shown, a button is pressed, or is dismissed. Use this for the subframe crash notification bar which currently does this a different way, r=mstriemer

This patch adds infrastructure for allowing the collecting telemetry on notification bars (the bars that appear at the top of the browser page), but does not add any telemetry itself, except as noted for the subframe crash bar below.

I'm not clear if the added infrastructure itself requires a data review. If it was, Romain would be better able to fill out the necessary data.

For the subframe crash bar:

1. This patch expands the telemetry for the subframe crash notification bar, added by bug 1690390. The data review for this was done in bug 1687039.
2. The expiry is unchanged.
3. This expands the telemetry from indicating only the shown count, to also including the count of the times the notification is dismissed and the count of the buttons that are pressed (learn more and submit). This is added only because a request was made to collect this information in a standard way for all notification bars. I don't believe the fission team will specifically make use of the expanded data.

Comment 17

[Chris H-C :chutten]

Comment on attachment 9207948 [details]
Bug 1690390, add ability to perform telemetry when a notification bar is shown, a button is pressed, or is dismissed. Use this for the subframe crash notification bar which currently does this a different way, r=mstriemer

Hi! Whereabouts can I find the Data Review Request for the expanded collection? https://wiki.mozilla.org/Data_Collection

The infrastructure may not need review if each change or added user of the infra will have its own review. Telemetry Scalars as a system doesn't have a review since each new scalar (or set of scalars) is reviewed, as an example.

Comment 18

[Neil Deakin]

Hi! Whereabouts can I find the Data Review Request for the expanded collection? https://wiki.mozilla.org/Data_Collection

The comment above (16) has the three points for the three questions listed in the request form: https://github.com/mozilla/data-review/blob/main/renewal_request.md

Is there a different form or more details that are needed?

The infrastructure may not need review if each change or added user of the infra will have its own review.

That was the expectation yes.

Comment 19

[Chris H-C :chutten]

That's the form for a renewal. The full request form is here: https://github.com/mozilla/data-review/blob/main/request.md

If each changed or added use of the infrastructure will have its own review, then we're good here. Thanks for checking!

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:enndeakin, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 21

[Neil Deakin]

Attachment: Bug 1690390, add a flag to limit telemetry collection for notification bars to just selected changes as we only want to measure shown state for the subframe crash one, r=mstriemer

Depends on D107756

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cc7e02d3d418
change appendNotification to take a smaller number of arguments: the string type value, a dictionary containing the notification details, and the button list, r=mstriemer
https://hg.mozilla.org/integration/autoland/rev/a161177099af
add ability to perform telemetry when a notification bar is shown, a button is pressed, or is dismissed. Use this for the subframe crash notification bar which currently does this a different way, r=mstriemer,chutten
https://hg.mozilla.org/integration/autoland/rev/e6404b7424cd
add a flag to limit telemetry collection for notification bars to just selected changes as we only want to measure shown state for the subframe crash one, r=mstriemer

Comment 23

[Narcis Beleuzu [:NarcisB]]

Backed out for mochitest failures on test_notificationbox.xhtml

Backout link: https://hg.mozilla.org/integration/autoland/rev/22cbcaeea6fa79bd6530539e1df7babd3c023aa5
Log link: https://treeherder.mozilla.org/logviewer?job_id=351798174&repo=autoland&lineNumber=2007

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e42e88685e1e
change appendNotification to take a smaller number of arguments: the string type value, a dictionary containing the notification details, and the button list, r=mstriemer
https://hg.mozilla.org/integration/autoland/rev/f696d86a6372
add ability to perform telemetry when a notification bar is shown, a button is pressed, or is dismissed. Use this for the subframe crash notification bar which currently does this a different way, r=mstriemer,chutten
https://hg.mozilla.org/integration/autoland/rev/ed7aeef1b71a
add a flag to limit telemetry collection for notification bars to just selected changes as we only want to measure shown state for the subframe crash one, r=mstriemer

Comment 25

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/e42e88685e1e
https://hg.mozilla.org/mozilla-central/rev/f696d86a6372
https://hg.mozilla.org/mozilla-central/rev/ed7aeef1b71a