1412411 - Add scalars to track screenshots usage

Status: RESOLVED FIXED

(Firefox :: Screenshots, enhancement)

Description

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Goal: determine if Screenshots usage correlates with higher browser engagement.

Note that Screenshots currently tracks usage in GA, but in order to compare MAU/DAU engagement between Screenshots users and non-users, we'd like to add three scalars to the main ping:

screenshots.download - incremented each time a shot is created & downloaded during a session

screenshots.upload - incremented each time a shot is created & uploaded to the screenshots server during a session

screenshots.copy - incremented each time a shot is created & copied straight to the clipboard without downloading or uploading [this feature is planned, but not yet implemented]

We'd like to gather this data in all channels, including release, and gather it by default (that is, make it opt-out). I'll attach a first draft patch that disables gathering this data if the 'datareporting.healthreport.uploadEnabled' pref is set to false. Questions:
- Is this the right pref to check?
- Does the Telemetry service scalarAdd method do the Telemetry enabled check, so that a check inside Screenshots code isn't necessary?

We would like to keep these scalars around permanently, to track whether changes to the Screenshots feature set change the number of Screenshots sessions or the overall engagement of Screenshots users.

Comment 1

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Attachment: Bug 1412411 - Add Screenshots Telemetry scalar definitions to Scalars.yaml;

Review commit: https://reviewboard.mozilla.org/r/194114/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/194114/

Comment 2

[Chris H-C :chutten]

You need not fret about collecting/sending. If uploadEnabled is false, we will not send the information. So unless collecting this information is expensive (perf-wise), you can unconditionally collect information using any Telemetry mechanism and we'll take it from there (making sure upload's enabled, the user's been given an opportunity to consult the data privacy notification, and so forth)

"permanently" will mean increased scrutiny in the Data Collection review[1]. Generally speaking, setting it to expire after 5 versions and revisiting then whether you still want the data is usually sufficient. I would recommend the "expire after 5" route, because it isn't onerous and it will remind you to run some analysis on your data in six months :)

Feel free to r? or f? me on the patch (doesn't count as Data Collection review, that has to be from a Data Peer) so I can review your use of the mechanisms we have available. It seems as though you have a confident and correct idea of what and how to measure and analyse things, but it's possible on review that there might be something even better we can recommend at the time.

Along the way you might find the in-tree docs[2] and #telemetry on IRC to be useful.

[1]: https://wiki.mozilla.org/Firefox/Data_Collection
[2]: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/scalars.html

Comment 3

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Related: this bug would be a good use case to expose Telemetry to special WebExtensions (bug 1280234). Probably wouldn't do the extra WebExtension API work for this first pass, as we'd like to get this landed before 58 goes to beta, but maybe for 59.

Comment 4

[Chris H-C :chutten]

The plumbing for scalars being recorded by addons was done in bug 1395206 so that if you want a non-shipping addon to be able to record telemetry, and it can access nsITelemetry in JS, then it can register some dynamically and record to them, and we'll deliver it to the endpoint.

Screenshots seems like a special case since it's an addon that rides the trains.

Comment 5

[François Marier [:francois]]

Comment on attachment 8922951 [details]
Bug 1412411 - Add Screenshots Telemetry scalar definitions to Scalars.yaml;

https://reviewboard.mozilla.org/r/194114/#review199168

This is Category 2 data.

datareview+

::: toolkit/components/telemetry/Scalars.yaml:1050
(Diff revision 1)
> +      via the Firefox Screenshots feature. Excludes failed attempts due
> +      to any error.
> +    expires: never
> +    kind: uint
> +    notification_emails:
> +      - [e-mail]

Can you add your email address too?

We now ask that all new telemetry includes the email of a person. Could be a manager too if you prefer.

Comment 6

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 8922951 [details]
Bug 1412411 - Add Screenshots Telemetry scalar definitions to Scalars.yaml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/194114/diff/1-2/

Comment 7

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

It's a bit awkward, but we generally land code in Github first, then export to Firefox.

In this case, I've updated the commit attached here to just include the Scalars.yaml changes, since those can land now.

I'll close out this bug when the code changes (that were already reviewed) are landed as part of the next export.

For reference, the Github bug is https://github.com/mozilla-services/screenshots/pull/3711.

Comment 8

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Comment on attachment 8922951 [details]
Bug 1412411 - Add Screenshots Telemetry scalar definitions to Scalars.yaml;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/194114/diff/2-3/

Comment 9

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Updated once more to include actual individuals' email addresses, rather than the team's internal mailing list.

Sorry for the spam.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4df9bab22d51
Add Screenshots Telemetry scalar definitions to Scalars.yaml; r=francois

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/4df9bab22d51