Closed Bug 1310163 Opened 8 years ago Closed 7 years ago

Win64 Funnelcake comparative user sentiment Heartbeat experiment

Categories

(Shield :: Heartbeat, defect)

Product:
Shield
Component:
Heartbeat
Platform:
All
Windows
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: RT, Assigned: rrayborn)

References

Details

User Story

Context: A wider roll-out of Firefox 64 bit on Windows 7/8/8.1/10 is being planned through the stub installer that will make 64 bit Firefox the default build for eligible systems. Ahead of that roll-out hypothesis will be tested using Funnelcake (See test plan document as a reference for the test) and we need to collect comparative user sentiment to validate that user feedback reflects the metrics collected via Telemetry.

This bug is for the creation of a heartbeat campaign along with the Win64 funnelcake experiment (see bug 1309847 for Optimizely set-up and bug 1309844 for funnelcake build creation).

Start date: June 20th
Channel: Release 54
Test plan: https://docs.google.com/document/d/1cmvJUFNr4zg_1_bEXAsbSbEt2RqmN-gKYLrLy2WNnHY/edit#    


Some ideas to be validated through the interview process:
We could ask users to simply star rate Firefox after a few days of use and then have follow-up questions based on their rating:
Why did you score Firefox so low? 
- Firefox crashes too much
- Firefox is too slow
- Other (open text field)

Why did you score Firefox so high? 
- Firefox so stable I cannot believe it
- Firefox so fast
- Other (open text field)

Ideally we should collect details about the amount of memory on the machine (part of the funnelcake exercise is about confirming if a minimum memory requirement is needed or Win64).
      No description provided.
User Story: (updated)
Depends on: 1309847
Blocks: support-win64
OS: Unspecified → Windows
Hardware: Unspecified → All
Summary: Win64 funnelcake comparative user sentiment heartbeat experiment → Win64 Funnelcake comparative user sentiment Heartbeat experiment
I added a comment to the doc about beta vs release given that /firefox/new/ is for the release channel and /firefox/channel/ has all of the non release channels.
User story now updated to reflect the fact we target Release 52 to run the tests.
User Story: (updated)
Assigning this one to Rob. The system add-on lands in Firefox 52. That should give us everything we need to make this happen. Rob, can you set up some time with Romain to get understand their specific survey needs?
Assignee: nobody → rrayborn
Hi Rob, we're still on track for this to run during release 52. When is a good time to start this process so we don't delay the funnelcake experiments?
Flags: needinfo?(rrayborn)
We're having to delay this by 2 releases because of a flash bug (bug 1336517). A work-around is being implemented with FF54 (bug 1340276) so FF54 is the sooner we can run the funnelcake.
User story field now updated.
User Story: (updated)
Updating launch date to June 20th per https://bugzilla.mozilla.org/show_bug.cgi?id=1309844#c33
User Story: (updated)
We have completed our user sentiment survey comparing 32- and 64-bit users in our Firefox 54 Funnelcake experiment. Kamyar prepared the following report of the survey reports. As expected, user sentiment is slightly higher on 64-bit than 32-bit.

https://docs.google.com/document/d/1dlDqZ775FvksbFGd8dS4jnKwJJUSA0LFIprt3pPKTu0/

* 32 bit users prompted early - 4.59/5 (873 respondents)
* 64 bit users prompted early - 4.61/5 (922 respondents)
* 32 bit users prompted 1 week after install - 4.42/5 (397 respondents)
* 64 bit users prompted 1 week after install - 4.62/5 (480 respondents)
Status: NEW → RESOLVED
Closed: 7 years ago
Flags: needinfo?(rrayborn)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.