1619981 - Investigate low enrollment rate for Chrome Switchers experiment

Status: RESOLVED WONTFIX

(Firefox :: Messaging System, defect, P1)

Description

[Andrei Oprea [:andreio]]

Experiment bug: https://experimenter.services.mozilla.com/experiments/chrome-switchers-card-for-onboarding-triplets-in-firefox-73/
Enrollments have been significantly lower than expected pointing to a problem with the attribution data not being available early enough at startup for all users.

Comment 1

[Ron Manning]

Hey Andrei, has this work been completed? If so, can we close this ticket?

Comment 2

[Andrei Oprea [:andreio]]

Telemetry shows attribution data in around ~19% of new profiles Windows Firefox 73 https://sql.telemetry.mozilla.org/queries/68231/source#172443
And the success rate of Normady recipes at first startup > 80% https://sql.telemetry.mozilla.org/queries/68887#173944
Given this the number of enrolled users in the experiment should be higher than it currently is.

Comment 3

[Ed Lee :Mardak]

pdahiya, we did make use of FirstStartup for some about:welcome experiment, and did we notice any issue with enrollment?

If not, I think we might then conclude the low enrollment is related to normandy recipe targeting on attribution data. So instead should we go with Su's other approach of enrolling more new profiles not targeting on attribution but analyze the results with ua attribution?

Comment 4

[Punam Dahiya [:pdahiya]]

(In reply to Ed Lee :Mardak from comment #3)

pdahiya, we did make use of FirstStartup for some about:welcome experiment, and did we notice any issue with enrollment?

Our last dynamic triplet experiment shows enrollment as expected with no issues

https://experimenter.services.mozilla.com/experiments/activity-stream-contextual-onboarding-experiment-for-dynamic-triplets-on-newtab-in-firefox-72-v2/

https://grafana.telemetry.mozilla.org/d/XspgvdxZz/experiment-enrollment?orgId=1&var-experiment_id=activity-stream-contextual-onboarding-triplets-firefox-72-bug-1601762-v2&from=1579564800000&to=1583366400000

If not, I think we might then conclude the low enrollment is related to normandy recipe targeting on attribution data. So instead should we go with Su's other approach of enrolling more new profiles not targeting on attribution but analyze the results with ua attribution?

I agree, It seems Normandy recipe is not able to filter based of attribution.ua on time. If we remove attribution.ua filter from Normnady recipe that means all the user enrolled in dynamic_chrome branch will be seeing 'Import data from Chrome ' card. Is that optimal? One solution I can think of is adding attribution.ua filter in targeting here https://searchfox.org/mozilla-central/source/browser/components/newtab/lib/OnboardingMessageProvider.jsm#425, This is possible if we can get that patch uplifted to 74 today

Comment 5

[Su-Young Hong]

Right. I think if you wanted to relaunch this, one option is:

• target widely for all new profiles (same as we did with the Dynamic Triplets experiment) - so no attribution.ua="chrome" filter
• have some sort of targeting in the pref/triplet card showing logic, so that only attribution.ua="chrome" users are getting the import from chrome triplet.

I think originally, the logic we had for moving the targeting into the Normandy recipe was there was no ability for the pref to target based on attribution, so targeting in Normandy was the only way to ensure only Chrome users got the Chrome card.

It sounds like that can be fixed. That's probably the best way going forward. The population breakdown would look something like this:

There is a gap in our data, however, if we do this. So for some users in treatment, they'll get the chrome card, for others, they'll get the "regular". Before, we could safely assume all users in treatment would see the Chrome card, so that wasn't an issue, but now, that's no longer assured.

It would be important in any experiment analysis to explicitly confirm who got shown the chrome card. Currently, I don't think our triplet impression telemetry reports that (it would report the same message_id I think), so we would need to fill that gap in our telemetry.

Comment 6

[Su-Young Hong]

Attachment: Screen Shot 2020-03-07 at 11.59.53 AM.png

Comment 7

[Punam Dahiya [:pdahiya]]

There is a gap in our data, however, if we do this. So for some users in treatment, they'll get the chrome card, for others, they'll get the "regular". Before, we could safely assume all users in treatment would see the Chrome card, so that wasn't an issue, but now, that's no longer assured.

It would be important in any experiment analysis to explicitly confirm who got shown the chrome card. Currently, I don't think our triplet impression telemetry reports that (it would report the same message_id I think), so we would need to fill that gap in our telemetry.

In 74, all users enrolled in treatment branch 'dynamic_chrome' will see Chrome card

Comment 8

[Su-Young Hong]

In that case, the issue changes to

• what will happen to users who don't have chrome when the Chrome card gets clicked? Is the experience exactly the same as a user who does have chrome (the import manager gets initiated, user gets option to import from whatever other browsers exist?)
• is product okay with showing a card that says "import from Chrome" to users who don't have chrome on their computer? (for jim)

Design, hypothesis, and interpretation of experiment will change depending on answer to first question.

Comment 9

[Jim T]

(In reply to Su-Young Hong from comment #8)

• what will happen to users who don't have chrome when the Chrome card gets clicked? Is the experience exactly the same as a user who does have chrome (the import manager gets initiated, user gets option to import from whatever other browsers exist?)

Yes, that's correct. The normal import manager gets initiated and the browser selection is based on what's detected/supported.

• is product okay with showing a card that says "import from Chrome" to users who don't have chrome on their computer? (for jim)

Yes, I'm ok with that for a couple of reasons:

• Most users will probably have chrome installed anyway so I don't see a big risk with having that language be a little off
• This broadens the experiment to cover more people who do have chrome, but didn't download firefox with it, so it will provide more interesting results to compare

Comment 10

[Andrei Oprea [:andreio]]

Closing this as wontfix. Because:

• users did get enrolled but the number was lower than expected
• the targeting used was similar as a previous experiment with successful enrollment (with the difference in the attribution targeting)
  It seems likely that the attribution data is not ready fast enough. As a guideline for future experiments we should not targeting on properties that require IO and only rely on pref attributes.

Comment 11

[Andrei Oprea [:andreio]]

Additionally bug 1621402 will add telemetry to capture read and decode errors for attribution data. This should help answer the question of data availability.