1620313 - CFR Experiment for Link to Existing Heartbeat Survey

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Description

[Tim Spurway [:tspurway]]

This request is for a new URL Bar CFR that will link directly to a web page with a Heartbeat survey when clicked (ie. there is no door hanger). The purpose of this is to gather data on engagement rates of the proposed Heartbeat V2 user interface to judge it's "interruptive power" compared to the existing Heartbeat interface.

The UX for this feature is laid out in the design as "Tactic 2 - CFR Messaging with Existing Surveys"

:rscholl will supply the link URL to the "example" survey for this experiment.

Note that this experiment will require code changes to land in mozilla-central, and so will likely be run in Firefox 75 (mid April 2020), at the earliest. :andreio could you give the level of effort required to create the Template for this feature?

• When (Firefox release or Date) do you expect this to show up to users?: Firefox 75
• What is the Business Goal or Context of the request (brief summary or link)?: linked to Desktop Growth initiatives and the Heartbeat V2 improvement plan
• How will success be measured?: this is a fact finding experiment to calibrate the suitability for using CFR-like user interruptions for the Heartbeat V2 initiative. The plan is to collect engagement data for this new survey interruption.
• Who is the directly accountable/responsible individual making this Messaging Request? :tspurway
• Please describe the targeting (i.e. which users should see this message) in plain English (e.g. all users who are not signed-in to a Firefox Account in the en-US locale, US region, in Firefox 72 and higher) : en-US locale, Firefox 75 and higher
• Enterprise clients included?: no
• Link to content or specification of the desired message: Firefox Message described in (https://mozilla.invisionapp.com/share/ZNW9NOUHMGV#/screens/407895486_Heartbeat_Micro_Survey), the actual survey is TBD by :rscholl
• User impact if declined: n/a
• User risk associated with releasing this Message (High/Med/Low/None): Low
• Why is the Message risky/not risky? (and alternatives if risky): we do not need a large sample to get decent data (< 1%). The message is being delivered with reliable/know mechanisms (Messaging System)

Comment 1

[Rosanne Scholl]

We need only a small percent of users who fit the target to see the message; however those who are given the message should be a random sample of users (ie, NOT just the first XX who are active.) We do not want to offer the survey to ALL clients in "en-US locale, Firefox 75 and higher".

Comment 2

[Andrei Oprea [:andreio]]

Some more details:
Because the CFR will not go out to everyone targeting will most likely be on Normandy en-US locale, version >= 75 && a sampled selection.
We want to show this CFR after the user started the browsing session. This will be configurable through the message (we do the same for WNPanel notifications).

Comment 3

[Tim Spurway [:tspurway]]

(In reply to Rosanne Scholl from comment #1)

We need only a small percent of users who fit the target to see the message; however those who are given the message should be a random sample of users (ie, NOT just the first XX who are active.) We do not want to offer the survey to ALL clients in "en-US locale, Firefox 75 and higher".

Thanks for the clarification, Rosanne. My intention is that this bug is only for tracking the creation and landing of this message UI. We will need to open a new bug to actually track the launching of it with Normandy to a sampled percentage of users.

Comment 4

[Andrei Oprea [:andreio]]

Attachment: Bug 1620313 - CFR Experiment for Link to Existing Heartbeat Survey

Comment 5

[Tim Spurway [:tspurway]]

[Tracking Requested - why for this release]: this is a low risk patch on existing CFR infrastructure required for experimentation on releasing Heartbeat 2.0

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ec8a428759da
CFR Experiment for Link to Existing Heartbeat Survey r=k88hudson

Comment 7

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/ec8a428759da

Comment 8

[Julien Cristau [:jcristau]]

Andrei, does this (and bug 1622211?) need uplift to beta?

Comment 9

[Andrei Oprea [:andreio]]

Yes for both. I was waiting for QA to finish testing before requesting uplift.

Comment 10

[Marius Coman [:mcoman], Ecosystem QA]

I have tested the "HEARTBEAT_TACTIC_2" message and I can confirm the following:

• Verified that the "HEARTBEAT_TACTIC_2" recommendation button is successfully triggered in the right part of the "Address Bar".
• Verified that the button UI respects the specifications from the documentation.
• Verified that the button UI respects the browser’s "Dark" theme colors.
• Verified that the correct page ("https://www.mozilla.org/" page was used for testing purposes) is successfully opened if the message’s "Improve Firefox" button is clicked.
• Verified that the button is no longer displayed and the message appears as blocked in the “about:newtab#devtools” page, if the "Improve Firefox" button is clicked.
• Verified that the message is displayed only once per day across all CFR recommendations and for a maximum amount of 3 times.
• Verified that the "HEARTBEAT_TACTIC_2" related telemetry pings are successfully displayed in the Browser Console.
• Using a local telemetry server, verified that the related telemetry pings are successfully sent and received by the server.

Verified using the latest Firefox Nightly (76.0a1 Build ID - 20200318213346) on Windows 10 x64, Mac 10.15.3 and Ubuntu 18.04 x64.

Comment 11

[Andrei Oprea [:andreio]]

Comment on attachment 9131523 [details]
Bug 1620313 - CFR Experiment for Link to Existing Heartbeat Survey

Beta/Release Uplift Approval Request

• User impact if declined: Unable to run feature experiment in 75
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1622211
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Covered by automated tests and QA. Feature can be turned off or triggered by Remote Settings without additional changes.
• String changes made/needed:

Comment 12

[Julien Cristau [:jcristau]]

Comment on attachment 9131523 [details]
Bug 1620313 - CFR Experiment for Link to Existing Heartbeat Survey

cfr update, verified by QA, approved for 75.0b7

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/fe062fc6b818

Comment 14

[Marius Coman [:mcoman], Ecosystem QA]

I have tested the "HEARTBEAT_TACTIC_2" message using the "Stage" server environment and I can confirm the following:

• Verified that the "HEARTBEAT_TACTIC_2" recommendation button is successfully triggered in the right part of the "Address Bar".
• Verified that the button UI respects the specifications from the documentation.
• Verified that the button UI respects the browser’s "Dark" theme colors.
• Verified that the correct page ("https://www.mozilla.org/" page was used for testing purposes) is successfully opened if the message’s "Improve Firefox" button is clicked.
• Verified that the button is no longer displayed and the message appears as blocked in the “about:newtab#devtools” page, if the "Improve Firefox" button is clicked.
• Verified that the message is displayed only once per day across all CFR recommendations and for a maximum amount of 3 times.
• Verified that the "HEARTBEAT_TACTIC_2" related telemetry pings are successfully displayed in the Browser Console.
• Using a local telemetry server, verified that the related telemetry pings are successfully sent and received by the server.

Verified using the latest Firefox Beta (75.0b7 Build ID - 20200322132212) on Windows 10 x64, Mac 10.15.3 and Ubuntu 18.04 x64.