Closed Bug 1550065 Opened 9 months ago Closed 8 months ago

Add telemetry for Return to AMO experience

Categories

(Firefox :: Messaging System, enhancement, P1)

enhancement

Tracking

()

VERIFIED FIXED
Firefox 68
Iteration:
69.1 - May 13 - 26
Tracking Status
firefox68 --- verified

People

(Reporter: nanj, Assigned: andreio)

References

(Blocks 1 open bug)

Details

(Keywords: github-merged)

Attachments

(2 files)

No description provided.
Blocks: 1468680
See Also: → 1475354
Assignee: nobody → andrei.br92
Blocks: 1549770

+Corey to make sure that this will address the gaps he found in his report already.

If I am understanding the code correctly, there will be three pings:

  • That a Return-to-amo page was presented to user: message.template === "return_to_amo_overlay"
  • A user clicked to install extension on RTA page (this.onClickAddExtension): event: "INSTALL"
  • A user clicked to bypass installation on RTA page (this.onBlockButton): event: "BLOCK"

Is this correct? If so, this gives us exactly what we are missing for usage stats.

Attached file data_review.txt

The feature was already shipped but we never added proper telemetry to record user interaction. Thanks.

Attachment #9063815 - Flags: data-review?(chutten)
See Also: → 1468680
Blocks: 1543770
Comment on attachment 9063815 [details]
data_review.txt

Clearing data review for the moment while :andreio looks into reporting policy.
Attachment #9063815 - Flags: data-review?(chutten)

(In reply to Chris H-C :chutten from comment #5)

Comment on attachment 9063815 [details]
data_review.txt

Clearing data review for the moment while :andreio looks into reporting
policy.

Hi :chutten,

Trust reviewed data collection request with team. Ok for the additional collection as Tier 2 data. I don't see additional risk to users or anything concerning in the timing of the pings.

Comment on attachment 9063815 [details]
data_review.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is documented here: https://github.com/mozilla/activity-stream/blob/0a79759bc8b865586c28f44bd2c7f4997faa1947/docs/v2-system-addon/data_events.md#onboarding-interaction-pings

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection adheres to the Data Collections preference in Firefox's Preferences. Activity Stream also has its own opt-out if a user would like to only opt out of Activity Stream data collection.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Nan Jiang is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+
Attachment #9063815 - Flags: data-review+
Keywords: github-merged
Blocks: 1551751
Type: defect → enhancement
Priority: -- → P1
Status: NEW → RESOLVED
Iteration: --- → 69.1 - May 13 - 26
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 68

To verify the telemetry pings

  1. Set browser.newtabpage.activity-stream.telemetry, browser.ping-centre.log and browser.newtabpage.activity-stream.asrouter.devtoolsEnabled to true
  2. Open the Browser Console (https://developer.mozilla.org/en-US/docs/Tools/Browser_Console)
  3. Trigger the RTAMO message
    b. Go to about:newtab#devtools-targeting scroll to the bottom and press the Force Attribution button
    c. Open about:welcome
    d. Ensure the RTAMO message is displayed (might require refreshing the page if the Trailhead onboarding message is show instead).
  4. Confirm a telemetry event is logged to the console that contains
"message_id":"RETURN_TO_AMO_1","source":"NEWTAB_OVERLAY","action":"onboarding_user_event","event":"IMPRESSION"
  1. Clicking the Get Started with Nightly should log a similar message with the BLOCK event, and clicking the Add the Extensions button should log a message with the INSTALL event.

Verified fixed on Nightly 68.0a1 (20190516215643) with Win10x64 and macOS High Sierra 10.13.6
*Notes: to get the RTAMO overlay working on Windows, I had to restart the profile and refresh the page (this removed the Trailhead onboarding message)

The following telemetry pings are sent based on the specified user actions:

  1. Return to AMO overlay is shown:
    "message_id":"RETURN_TO_AMO_1","source":"NEWTAB_OVERLAY","action":"onboarding_user_event","event":"IMPRESSION"

  2. 'Add the Extension' button is clicked:
    "message_id":"RETURN_TO_AMO_1","source":"NEWTAB_OVERLAY","action":"onboarding_user_event","event":"INSTALL"

  3. 'Get Started with Nightly' button is clicked (skip extension installation):
    "message_id":"RETURN_TO_AMO_1","source":"NEWTAB_OVERLAY","action":"onboarding_user_event","event":"BLOCK"

Status: RESOLVED → VERIFIED
Component: Activity Streams: Newtab → Messaging System
You need to log in before you can comment on or make changes to this bug.