1509890 - Add Telemetry for Track Changes M1

Status: RESOLVED FIXED

(DevTools :: Inspector, enhancement, P2)

Description

[Razvan Caliman [:rcaliman]]

For the Changes panel introduced with Firefox 65 in the Inspector, we want to have the same level of basic telemetry to compare it with its sibling sidebar panels and some custom telemetry to help identify how effective it is at solving user needs.

Basic:
- Number of times the Changes has been opened.
- How long has the Changes panel been active (seconds)

Custom:
- How long has the Changes panel been active (seconds) **and** the number of changes made in a DevTools session.

We want to test the hypothesis that users are more confident to make a larger number of changes in a single DevTools session when they know their changes are collected and summarized in one place - the Changes panel.

We expect to see a correlation between meaningful time spent in the Changes panel and an increased in the number of CSS changes.

For M2 of Track Changes we will have better mechanisms to test engagement with the Changes panel, actions like copy, export, persist, etc.

Comment 1

[Razvan Caliman [:rcaliman]]

Attachment: bug 1509890 - (Part 1) Add basic telemetry for Changes panel (time spent and open count). r=pbro

Adds entries to Histograms.json, Scalars.yaml and Events.yaml to
account for the Changes panel when measuring time spent with the panel
in view and opening count. This leverages pre-existing instrumentation
used for other Inspector sidebar panels.

Comment 2

[Razvan Caliman [:rcaliman]]

There are a few technical complexities for implementing the custom telemetry case explained in the description. 

I'll split the commits and data request forms and into two distinct parts (part 1 for basic, part 2 for custom), to ensure the basic telemetry gets clearance to launch with Firefox 65 in case the more complex one gets delayed.

Comment 3

[Razvan Caliman [:rcaliman]]

Attachment: data-review-part1.md

Hi Chris,

May I please have a review from you on this proposal to add basic telemetry to the new Changes panel in the DevTools Inspector sidebar? This request is for tracking opening count and time spent in the panel, similar to instrumentation already in place for the other Inspector sidebar panels.

I intend to follow up with a second data review request form (or an updated one, if you prefer) for the custom case explained in the description once I figure out how to do it.

The reason for splitting the request and implementation into two parts is to ensure we can release this feature instrumented with at least the basic telemetry when it goes to Firefox 65 Beta. Some technical complexities and upcoming travel for All Hands may delay the second part.

Comment 4

[Chris H-C :chutten]

Comment on attachment 9027936 [details]
data-review-part1.md

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate? 

Yes. These collections are Telemetry which means they are documented in their definitions files (Histograms.json, Scalars.yaml), in the Probe Dictionary, and on telemetry.mozilla.org's Measurement dashboard.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. These collections are Telemetry which means they can be controlled in Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Martin Balfanz.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on, all channels.

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? 

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. These collections are permanent.

---
Result: datareview+

Comment 5

[Chris H-C :chutten]

(In reply to Razvan Caliman [:rcaliman] from comment #3)
> Created attachment 9027936 [details]
> data-review-part1.md
> 
> Hi Chris,
> 
> May I please have a review from you on this proposal to add basic telemetry
> to the new Changes panel in the DevTools Inspector sidebar? This request is
> for tracking opening count and time spent in the panel, similar to
> instrumentation already in place for the other Inspector sidebar panels.
> 
> I intend to follow up with a second data review request form (or an updated
> one, if you prefer) for the custom case explained in the description once I
> figure out how to do it.
> 
> The reason for splitting the request and implementation into two parts is to
> ensure we can release this feature instrumented with at least the basic
> telemetry when it goes to Firefox 65 Beta. Some technical complexities and
> upcoming travel for All Hands may delay the second part.

I think a second review request would make the most sense.

Comment 6

[Razvan Caliman [:rcaliman]]

(In reply to Chris H-C :chutten from comment #5)
> (In reply to Razvan Caliman [:rcaliman] from comment #3)
> > Created attachment 9027936 [details]
> > data-review-part1.md
> > 
> > Hi Chris,
> > 
> > May I please have a review from you on this proposal to add basic telemetry
> > to the new Changes panel in the DevTools Inspector sidebar? This request is
> > for tracking opening count and time spent in the panel, similar to
> > instrumentation already in place for the other Inspector sidebar panels.
> > 
> > I intend to follow up with a second data review request form (or an updated
> > one, if you prefer) for the custom case explained in the description once I
> > figure out how to do it.
> > 
> > The reason for splitting the request and implementation into two parts is to
> > ensure we can release this feature instrumented with at least the basic
> > telemetry when it goes to Firefox 65 Beta. Some technical complexities and
> > upcoming travel for All Hands may delay the second part.
> 
> I think a second review request would make the most sense.

I'm going to land part 1 with the basic telemetry as-is. For the custom case, I will follow up in another bug. Thank you for the review!

Try run for part 1: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4bf2df3b650770d8615ee773b736c21c8e7ff926

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dc482031a974
(Part 1) Add basic telemetry for Changes panel (time spent and open count). r=pbro

Comment 8

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/dc482031a974