1547062 - Telemetry and branch generation for trailhead first run

Status: VERIFIED FIXED

(Firefox :: Messaging System, enhancement, P1)

Comment 1

[Kate Hudson :k88hudson]

See https://docs.google.com/document/d/1t3dDeaAKwLpH39a2LZ7xYzX4n17w-IGiTUpk0xU9LdY/edit#heading=h.eeup78jxxkio for notes on a possible strategy for this.

Comment 2

[Kate Hudson :k88hudson]

Is it ok for us to include all users for 67.5 in a branch for this experiment, or do we need to exclude some users?

Comment 3

[Ed Lee :Mardak]

This is in the context of activity stream needing to do its own experiment user selection at startup where we do have a "control" group and if a user wouldn't be in the experiment would have the same experience as the "control" group anyway. So if the user would experience the same thing anyway, would it be okay to just part of a much larger "control" group?

Comment 4

[Michael Cooper [:mythmon]]

I checked with Sunah Suh, and she said that as long as the type parameter used with calling setExperimentActive is unique, there won't be any problems with tagging every user, as if they were in a really big control group.

Comment 5

[Ed Lee :Mardak]

https://github.com/mozilla/activity-stream/pull/4980

We'll have a primary experiment (interrupts) and secondary (triplets) with most people getting a default experience (no experiment data).

Default experience would show the new "Join" modal followed by 1 of the 4 triplets:

• triplet "supercharge" = mobile firefox, sync, firefox monitor
• triplet "payoff" = firefox monitor, facebook container, firefox send
• triplet "multidevice" = mobile firefox, pocket, send tabs
• triplet "privacy" = private browsing, tracking protection, firefox lockwise

The primary experiment has 5 possible "interrupts":

• control / existing blue fxa messaging
• join / new purple "Join Firefox"
• sync / new purple but with existing fxa messaging
• nofirstrun / nothing - looks like about:newtab
• cards / no modal straight to triplet

The secondary experiment has 4 possible "triplets" as described above and shown after "join"

We will land with a default trailhead.firstrun.branches pref value so nightly users will always see join-privacy. For testing, this value can be changed requiring restarting firefox and visiting about:welcome. For uplift to 67, the pref value will be set to "control" so everyone gets the existing experience there. Then for later uplift don't set the pref and experimentation will pick the appropriate branches.

For debugging, after visiting about:welcome and getting an experiment picked, one can go to about:newtab#devtools and confirm the picked branches.

Comment 8

[Ed Lee :Mardak]

The code is landing in nightly with equal probability of a user picked for 1) no experiment, 2) interrupts experiment or 3) triplet experiment. This picking happens when trailhead.firstrun.branches is set to blank/nothing/deleted. While we have the equal probability sampling, for testing, one can set app.normandy.user_id to these values to end up with a desired experiment variant (instead of creating a new profile and randomly getting some variant):

0  {"experiment":"","interrupt":"join","triplet":"multidevice"}
20 {"experiment":"","interrupt":"join","triplet":"payoff"}
2  {"experiment":"","interrupt":"join","triplet":"privacy"}
10 {"experiment":"","interrupt":"join","triplet":"supercharge"}
12 {"experiment":"interrupts","interrupt":"cards","triplet":"supercharge"}
11 {"experiment":"interrupts","interrupt":"control"}
7  {"experiment":"interrupts","interrupt":"join","triplet":"supercharge"}
1  {"experiment":"interrupts","interrupt":"nofirstrun"}
5  {"experiment":"interrupts","interrupt":"sync","triplet":"supercharge"}
15 {"experiment":"triplets","interrupt":"join","triplet":"multidevice"}
3  {"experiment":"triplets","interrupt":"join","triplet":"payoff"}
46 {"experiment":"triplets","interrupt":"join","triplet":"privacy"}
9  {"experiment":"triplets","interrupt":"join","triplet":"supercharge"}

(These user_id values will be different when we have different ratios in release)

For example, set app.normandy.user_id pref to "0" (string pref) and trailhead.firstrun.branches to empty string and restarting then going to about:welcome should show "join" interrupt with "multidevice" triplet.

Similarly, setting to "12" should show no modal and "supercharge" triplets. Also, going to about:telemetry -> Environment Data -> Experiments should show:

activity-stream-firstrun-trailhead-interrupts.branch 	cards
activity-stream-firstrun-trailhead-interrupts.type 	as-firstrun 

For just testing the various combinations of interrupts and triplets, one can set trailhead.firstrun.branches to <interrupt>-<triplet>, e.g., "cards-multidevice" or "control" (as there's no new triplet). This also requires a restart after setting the pref.

Comment 9

[Ed Lee :Mardak]

Also, when doing the branches = blank/nothing/deleted and user_id (or just randomly generated) to verify if about:telemetry shows the active experiment…

This should persist after restart and show the same active experiment.

And this should only happen for en/de/fr locales while other locales should always see the existing/control. (Again emphasizing a clean profile will by default show about:welcome specified in trailhead.firstrun.branches for any locale so make sure to set that pref to blank/nothing/deleted to allow experimentation testing.)

Comment 10

[Ed Lee :Mardak]

https://hg.mozilla.org/mozilla-central/rev/5c4931404035

Comment 11

[Marius Coman [:mcoman], Ecosystem QA]

I have verified this issue with the latest Firefox Nightly (68.0a1 Build ID - 20190507214514) installed, on Windows 10 x64, Arch Linux and Mac 10.14.4. I can confirm that all the elements form the "First Run" page are displayed according to the provided documentation and the telemetry pings are successfully sent.

Comment 12

[Ed Lee :Mardak]

Marking verified as per bug 1550098 Comment 13 for bugs that were status-firefox68=verified