Closed Bug 1405217 Opened 7 years ago Closed 5 years ago

[Shield] Pref Flip Study: Gradual roll out form autofill in 56

Categories

(Shield :: Shield Study, defect)

Product:
Shield
Component:
Shield Study
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox56+ wontfix)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox56 + wontfix

People

(Reporter: chsiang, Assigned: chsiang)

References

(
URL
)

Details

(Whiteboard: [form autofill:MVP]) 

Basic description of experiment: 
We are ready to roll out Form autofill (address) support to Firefox 56 users in US using EN-US but we want to do a staged rollout to collect feedback and understand satisfaction before we roll this out 100% 

What is the preference we will be changing? 
extensions.formautofill.available

What are the branches of the study and what values should each branch be set to? The above pref should be set to "detect" on release branch.

What percentage of users do you want in each branch? 
Need help figuring this out, but we are targeting US users using EN-US with 37M MAU (https://sql.telemetry.mozilla.org/queries/39445#106233), what number of users would make sense to roll this feature out to initially? Since form filling isn’t something users do everyday, having a higher % should allow us to receive more feedback initially

What Channels and locales do you intend to ship to? 
Firefox 56 release. EN-US only

What is your intended go live date and how long will the study run? 
For the duration of release 56. We should turn the feature on by default from 57 and on

Are there specific criteria for participants? 
We only want to target users who have used the form autofill feature 

What is the main effect you are looking for and what data will you use to make these decisions? 
Avg time to complete forms are shorter and we retain users better with higher satisfaction.  

Who is the owner of the data analysis for this study? Analysis owner. 
To be assigned 

Will this experiment require uplift? 
No

QA Status of your code: 
Green and shipped on 56

Do you plan on surveying users at the end of the study? 
Yes 
https://docs.google.com/document/d/1zUcFtnn715qhqCNN6A71LtHGcty5smdy2Q7iPJ-jA5Y/edit
Flags: needinfo?(mgrimes)
Flags: needinfo?(kardekani)
Flags: needinfo?(glind)
Whiteboard: [form autofill:MVP]
Hi Gregg,
Would be great if you can provide feedback on % of users and assign an analysis owner. 

What percentage of users do you want in each branch? 
Need help figuring this out, but we are targeting US users using EN-US with 37M MAU (https://sql.telemetry.mozilla.org/queries/39445#106233), what number of users would make sense to roll this feature out to initially? Since form filling isn’t something users do everyday, having a higher % should allow us to receive more feedback initially
Depends on: 1404531
Do we need any QA on the shield rollout itself?  I believe we normally have some testing and signoff. Assuming that goes well, we should be fine to go ahead for 56 release.
status-firefox56: --- → affected
tracking-firefox56: --- → +
(In reply to chsiang from comment #0)
> Basic description of experiment: 
> We are ready to roll out Form autofill (address) support to Firefox 56 users
> in US using EN-US but we want to do a staged rollout to collect feedback and
> understand satisfaction before we roll this out 100% 
> 
> What is the preference we will be changing? 
> extensions.formautofill.available
> 
> What are the branches of the study and what values should each branch be set
> to? The above pref should be set to "detect" on release branch.
> 
> What percentage of users do you want in each branch? 
> Need help figuring this out, but we are targeting US users using EN-US with
> 37M MAU (https://sql.telemetry.mozilla.org/queries/39445#106233), what
> number of users would make sense to roll this feature out to initially?
> Since form filling isn’t something users do everyday, having a higher %
> should allow us to receive more feedback initially
> 
> What Channels and locales do you intend to ship to? 
> Firefox 56 release. EN-US only
> 
> What is your intended go live date and how long will the study run? 
> For the duration of release 56. We should turn the feature on by default
> from 57 and on
> 
> Are there specific criteria for participants? 
> We only want to target users who have used the form autofill feature 
> 
> What is the main effect you are looking for and what data will you use to
> make these decisions? 
> Avg time to complete forms are shorter and we retain users better with
> higher satisfaction.  
> 
> Who is the owner of the data analysis for this study? Analysis owner. 
> To be assigned 
> 
> Will this experiment require uplift? 
> No
> 
> QA Status of your code: 
> Green and shipped on 56
> 
> Do you plan on surveying users at the end of the study? 
> Yes 
> https://docs.google.com/document/d/1zUcFtnn715qhqCNN6A71LtHGcty5smdy2Q7iPJ-
> jA5Y/edit

Update the description of the experiment:

Basic description of experiment: 
We are ready to roll out form autofill (address) in Firefox 56 for users in US using en-US build. The study is to support a staged rollout to collect feedback and understand satisfaction before we roll this out 100%.
What is the preference we will be changing? 
extensions.formautofill.available

What are the branches of the study and what values should each branch be set to? 
The above pref should be set to "detect" on release branch.
Treatment - detect
Control - off

What percentage of users do you want in each branch? 
1%

What Channels and locales do you intend to ship to? 
Firefox 56 release. EN-US only ( 37M MAU)

What is your intended go live date and how long will the study run? 
10/18 Wednesday Pacific time. This is a staged rollout so the pref will not expire for the treatment group. 

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 
Since this is a feature parity to other browsers, we want to make sure it works as users expect- 1% of users are as or more satisfied as the ones who don’t have the feature enabled.
 Below data should be collected and analyzed for enhancement, not to make go or no-go decisions
Number of forms filled (auto, manual, auto+update)
Autofill- indicate happy users/ no issues with the feature
Auto+ update- fix things
Manual- don’t like to use it/ feature doesn’t work/ personal preference
Enable vs. disable (Address enable: 7522 / disable: 37 )
Time took to fill the form (auto, manual, auto+update)
Number of fields corrected from the autofilled value to something different
Survey results

Who is the owner of the data analysis for this study? 
Josh and Robert

Will this experiment require uplift? 
No

QA Status of your code 
Green and shipped on 56

Do you plan on surveying users at the end of the study? 
Yes https://docs.google.com/document/d/1zUcFtnn715qhqCNN6A71LtHGcty5smdy2Q7iPJ-jA5Y/edit
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #2)
> Do we need any QA on the shield rollout itself?  I believe we normally have
> some testing and signoff. Assuming that goes well, we should be fine to go
> ahead for 56 release.

Rob, do you know the answer to this question?
Flags: needinfo?(rrayborn)
Still waiting to hear about QA signoff.
Sorry for the delay. I did the QA this morning on the Shield side. We are ready to roll!
Flags: needinfo?(mgrimes)
Flags: needinfo?(kardekani)
Hi Liz, per Comment#6, we should be good to roll now ?
Flags: needinfo?(lhenry)
Yes, please go ahead.
Flags: needinfo?(lhenry)
Josh and I just launched the pref rollout.  Will start serving surveys next week.
Flags: needinfo?(rrayborn)
Flags: needinfo?(glind)
It would be good if a link to the Shield recipe was included in the bug so interested parties could check the rules.

Sorry, I havne't been following the pref flip recently…

I noticed that pref is only being set on the default branch, not the user branch like we discussed before. Since we're doing this as a rollout, not as much of an experiment, we don't want users to lose the feature once they get it (unless something horrible happens).
Flags: needinfo?(rrayborn)
@Matt - can you please clarify?  We don't like to use User prefs since they can override existing customization, nor are they persistent at the end of a study as it seems was implied above.  We were planning to leave the recipe open until the change hit release as we have for other recipes in the past.

The only reason I can think to use a user pref for this recipe is if Firefox is setting the pref as "User" automatically (some prefs do this).  In this case it may be appropriate to override.  Please inform me if this is the case.

Link for interested parties (VPN required): https://normandy-admin.prod.mozaws.net/recipe/307/
Flags: needinfo?(rrayborn) → needinfo?(MattN+bmo)
(In reply to Rob from comment #11)
> @Matt - can you please clarify?  We don't like to use User prefs since they
> can override existing customization, nor are they persistent at the end of a
> study as it seems was implied above.

Oh, now I remember why default prefs won't work… the formautofill SAO only does a pref check once at startup (doesn't listen for changes) so if Shield only sets the pref after startup from the recipe then the add-on will never be enabled.

I guess what I remembered was that when the pref is set on the user branch that it will persist during a restart and thus will actually work to pref on the SAO.

I thought that the user set pref would also persist after the study ends. Is there a way to do that?

>  We were planning to leave the recipe
> open until the change hit release as we have for other recipes in the past.

When you say "until the change hit release", do you mean until the code that ships with Firefox defaults to "detect" for en-US in the US?

> The only reason I can think to use a user pref for this recipe is if Firefox
> is setting the pref as "User" automatically (some prefs do this).  In this
> case it may be appropriate to override.  Please inform me if this is the
> case.

See above about why I think a user pref is necessary.

> Link for interested parties (VPN required):
> https://normandy-admin.prod.mozaws.net/recipe/307/

Public URL: https://normandy.cdn.mozilla.net/api/v2/recipe/307/
Flags: needinfo?(MattN+bmo)
Ah didn't realize that this was a startup-reliant feature.  That's a fair use-case for User pref that I should have thought of mentioning in the reply.  I'll create an identical recipe to launch immediately (our front-end does not support changing the existing recipe for this field).
Ready to close the bug now that we've rolled out the update?
Flags: needinfo?(chsiang)
There are two phases:
1) Pref flip to roll out the feature
2) Surveying some of the users with Shield next week or so

I'll mark this as fixed for 56 but we still need to track #2 somewhere. If it causes problems we can deal with survey in a new bug.
status-firefox56: affected → fixed
Flags: needinfo?(chsiang)
Oh, sure, we can leave this open then or file a new bug. Up to you.
Matt, anything new happening here? Is the survey out or should I still expect it? 
I'm just going through my last few open issues for 56.
Flags: needinfo?(MattN+bmo)
The surveys are still collecting data. I plan on killing this study Friday just to be safe. Whatever responses we have at that point is what we'll get.
Flags: needinfo?(MattN+bmo)
(In reply to Matt Grimes [:Matt_G] from comment #19)
> The surveys are still collecting data. I plan on killing this study Friday
> just to be safe. Whatever responses we have at that point is what we'll get.

Hi Matt, so how's the data we collected so far?
Flags: needinfo?(mgrimes)
Per Comment#16, closed this bug as we already rollout Address autofill on 56.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Reopening this as Cindy has now requested we got to 20% of the Release population. Surveys will go out on 11/30.
Status: RESOLVED → REOPENED
Flags: needinfo?(mgrimes)
Resolution: FIXED → ---
OK. I'm marking this affected and tracking for 57.
From email discussion, sounds like Ritu has signed off on this going to release 57 at 20%. 
 
It would be best to have the signoff in the bugs as well rather than in private email threads. Thanks.
status-firefox57: --- → affected
tracking-firefox57: --- → +
This is now live for 20% of Release 57 in en_US.
status-firefox57: affected → ---
tracking-firefox57: + → ---
Hi Matt,
Could you share the link to the results?

Thanks.
Cindy
Flags: needinfo?(mgrimes)
I think tyler worked on this one. Tyler?
Flags: needinfo?(mgrimes) → needinfo?(tdowner)
Survey Results for the credit card autofill survey, https://qsurvey.mozilla.com/r/28049_5a25b5624643a5.90854601
Flags: needinfo?(tdowner)
Hi Tyler,
I am looking for the survey results of this one- https://qsurvey.mozilla.com/collab/Forms-AF?branch=true
It's address fill, not credit card. 
It's a gradual rollout and we have the results from the 1% rollout in 56 here. https://data.surveygizmo.com/r/28049_5a0241796dd762.91129749
Just looking for the results of the 20% rollout in 57 which happened on Nov 21, 2017. 
Thanks.
Cindy
Flags: needinfo?(tdowner)
Flags: needinfo?(tdowner)
Depends on: 1474839
Status: REOPENED → RESOLVED
Closed: 7 years ago → 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.