Funnelcake test plan with details: https://docs.google.com/document/d/1Kle7LlVQ_gCJ2IFpQOSg7XVDtrXGNYGPacKIaYO1i9A/edit
A bit of context, for people about to see this bug... ;)
We've been asked to run a funnelcake ASAP to test some changes to the first-run onboarding experience, with the ultimate aim to improve new user retention over the first few weeks of usage. The next funnelcake opportunity will be based on the Firefox 51 release on 1/24, so we're keeping the scope of this work as tight as possible due to the short schedule and need for late beta uplifts.
The general work we're testing involves streamlining the first-run experience, so that users can more quickly see and use their data. In a nutshell: (see deps for details)
* Enable the existing automigration work, to avoid the "import data from another browser" wizard.
* Remove/defer the distracting default browser prompt and data choices prompt
* Show users about:newtab instead of about:home, so they will see either
A) New default tiles that are more useful
B) Their top sites that were automigrated from another browser
* Start shifting the visual design of about:newtab towards what Activity Stream will look like
Ryan, I know the SV team is already on this bug, but wasn't sure to what degree you were looped in, so I wanted to bring it to your attention that we'll be needing a QA round on the funnelcake(s) starting at the end of January.
(In reply to Peter Dolanjski [:pdol] from comment #2)
> Ryan, I know the SV team is already on this bug, but wasn't sure to what
> degree you were looped in, so I wanted to bring it to your attention that
> we'll be needing a QA round on the funnelcake(s) starting at the end of
Peter, our QA Engineers from Las Vegas will help with this request. Thanks for the heads-up.
We are already working on a round of testing for this iteration of Funnelcake and should have a preliminary sign-off ready by the end of the week.
Adding details of the Funnelcake test plan in the user story field. Please note we now consider 2 test builds with 1 control. All details are in the Google doc: https://docs.google.com/document/d/1Kle7LlVQ_gCJ2IFpQOSg7XVDtrXGNYGPacKIaYO1i9A/edit
(In reply to Grover Wimberly IV [:Grover-QA] from comment #4)
> Hi Peter,
> We are already working on a round of testing for this iteration of
> Funnelcake and should have a preliminary sign-off ready by the end of the
Hi Grover, can you update us on that? Per the doc I linked to in my comment above we will have 2 test builds with 1 control - both test builds include the streamlined stub installer.
Do you have test plans you can share? Let us know if we need to meet to discuss the experience. I'd particularly like to check that the stub ping data is right and that the attribution field allows segmenting cohorts for reports.
This might be a bit different than what we were running tests on last week. Our tests currently cover primarily automigration. (See below)
This is the degree we've been involved in so far. We can meet to discuss the new features of Funnelcake mentioned in Comment 5 and Comment 6. Let me know what time is best for you.
OK, let's meet, I'll follow by e-mail to schedule something.
Hey Grover, how are we looking on QA?
Hi Nicole, an email was sent yesterday and I believe Romain has you CC'd on the preliminary sign-off. Once I get the go-ahead, I will send the sign-off report.
Experiment started on 2/7/2017 @ 6:05pm PST.