Closed Bug 1621834 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Activity Stream Do No Harm Experiment for simplified about:welcome in Firefox 75 Fx 75.0 to 76.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: shell)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment treatment - simplified dynamic 50%:

Value: None

Simplified welcome page that shows Sync, Monitor and Privacy cards on about:welcome.
        
- Control control - join dynamic 50%:

Value: None

About:welcome experience join-dynamic - join modal that appears on top of new tab with dynamic triplet cards - Sync, Monitor and Privacy
        

    What version and channel do you intend to ship to?

25% of Release Firefox 75.0 to 76.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:

Only enroll new users - we get this by excluding users with old profiles (profile creations date is less than launch date.) This is because we need telemetry date for this normally - but telemetry isn't guaranteed available that early. We expect this will cover the pave-over scenario - install with old profile with new firefox - this filter exclude the pave-overs.

Sticky: Has not seen about:welcome page. The recipe is excluding users who have trailhead.firstrun.didSeeAboutWelcome is true. new users on first startup may have this not set at normandy execution, or have it set to false, both of which will make them eligible for the experiment. non-new users will have this set to true. when they start the browser and normandy executes, the value for this pref should exist for normandy to filter them out of the experiment.
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Apr 14, 2020 - May 26, 2020 (42 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

https://docs.google.com/document/d/1KIOEEOjZFe_EzMjcyE0boXmmadhwb12z0c5gAAE3fs8/edit


Do you plan on surveying users at the end of the delivery? Yes/No.
Strategy and Insights can help create surveys if needed
 No

    Who is the owner of the data analysis for this experiment?

shong@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

PI request: https://jira.mozilla.com/browse/PI-507
Test Trail: https://testrail.stage.mozaws.net/index.php?/suites/view/7095&group_by=cases:section_id&group_order=asc&group_id=72595

This experiment was signed off as YELLOW. For further information see: https://experimenter.services.mozilla.com/experiments/do-no-harm-experiment-for-simplified-aboutwelcome-in-firefox-75/#comment768

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/do-no-harm-experiment-for-simplified-aboutwelcome-in-firefox-75/

Do No Harm Experiment for simplified about:welcome in Firefox 75

We want to ship a simplified, more performant, remote-capable experience on about:welcome as a new baseline for rapid learning/experimentation around activation this year. Although we are fairly confident it will not regress any general usage metrics, we want to confirm this is the case before rolling it out to new users as the new “control” experience.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1621347
More information: https://experimenter.services.mozilla.com/experiments/do-no-harm-experiment-for-simplified-aboutwelcome-in-firefox-75/

Do No Harm Experiment for Simplified about:welcome Page
Firefox Release 75
Firefox Release 76

We have finished testing the [PI-507] - Do No Harm Experiment for Simplified about:welcome Page experiment.

Quality status: RED - DON’T SHIP

Why is this experiment red?

  • During testing, we have found and logged the following blocker issue: Bug 1626904. This issue prevents us from being enrolled in the experiment in some cases and in the others we are enrolled but the old "about:welcome" page is displayed.
  • In order to test the enrollment part we have tried the following scenarios:
    • Using a Normandy stage recipe:
      The user is enrolled in the experiment, however the old “about:welcome” page is displayed until the page is refreshed.
    • Using a Normandy production recipe:
      In this case, we got mixed results, in one case the user was not enrolled at all and in the other case, the user was enrolled but the old “about:welcome” page was displayed.
    • Using a “pref.js” file containing the preference that we need to change:
      When we used the “pref.js” file, the preference was successfully set and the new simplified version of the “about:welcome” page was displayed. However, we cannot be sure that this will work also in a real scenario using a Normandy recipe given the results we have seen in the aforementioned scenarios and since this is an unnatural way of testing the enrollment

What needs to be done?

  • The following actions need to be taken in order for this experiment to be GREEN:
    • [Engineering] Bug 1626904 should be fixed or a more reliable way of enrollment has to be found.
    • [QA] Verify that end users will see the correct page once enrolled in the experiment after the fix lands or a different method is proposed.

Testing Summary:

Tested Platforms:

  • Windows 10 x64

Tested browser versions:

  • Firefox RC 75-build1 (en-US, es-ES, de)

Do No Harm Experiment for Simplified about:welcome Page
Firefox Release 75
Firefox Release 76

We have finished testing the [PI-507] - Do No Harm Experiment for Simplified about:welcome Page experiment.

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

  • After further investigation and help from the dev team, we still haven't found a reliable “natural” way to be enrolled in the experiment. The Engineering team has proposed that we move on with the launch of the experiment even with this issue, with the condition that they will monitor enrollments, and we are inclined to agree with this course of action.
  • This, coupled with the fact that the feature part of the experiment looks good and we are planning to find the root cause of the enrollment problem we will change the status from RED to YELLOW.

What needs to be done?

  • The following actions need to be taken:
    • [Engineering] The team should monitor the enrollment phase of the experiment in order to verify that the users that are enrolled have seen the intended page and if not, we should stop the experiment until we find the root cause of the problem.
    • [Engineering] Continue working with QA to try and find out the cause of this issue.

Testing Summary:

Tested Platforms:

  • Windows 10 x64

Tested browser versions:

  • Firefox RC 75-build1 (en-US, es-ES, de)

Regards,
Ciprian Muresan
Ecosystem QA

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Do No Harm Experiment for simplified about:welcome in Firefox 75 → [Experiment] Pref-Flip Experiment: Activty Stream Do No Harm Experiment for simplified about:welcome in Firefox 75 Fx 75.0 to 76.0 Release
Summary: [Experiment] Pref-Flip Experiment: Activty Stream Do No Harm Experiment for simplified about:welcome in Firefox 75 Fx 75.0 to 76.0 Release → [Experiment] Pref-Flip Experiment: Activity Stream Do No Harm Experiment for simplified about:welcome in Firefox 75 Fx 75.0 to 76.0 Release
Start Date: 2020-04-14 End Date: 2020-05-26
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.