Closed Bug 1620410 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Release Channel Test of QuantumBar v1 Fx 74.0 to 74.0 Release

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox74+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox74 + fixed

People

(Reporter: experimenter, Assigned: mconnor)

References

(Depends on 1 open bug,
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

None

    What are the branches of the experiment and what values should
    each branch be set to?

- Control control branch 50%:

Value: None

control branch
        
- Treatment treatmeant 50%:

Value: None

treatment branch
        

    What version and channel do you intend to ship to?

0.5% of Release Firefox 74.0 to 74.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
avoid the 2 running search experiments:
https://experimenter.services.mozilla.com/experiments/search-tips-aka-nudges/ and https://experimenter.services.mozilla.com/experiments/search-interventions-v2/
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Mar 12, 2020 - Mar 31, 2020 (19 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

> What is the main effect you are looking for and what data will you use to make these decisions? What metrics are you using to measure success?


We plan to use SABER to validate against unexpected regressions not observed in per-feature testing.  We may dig in more depth based on available time and/or unexpected results.

> Do you plan on surveying users at the end of the delivery?


No, but we plan to survey them sooner by targeting users in the study.

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

https://jira.mozilla.com/browse/PI-515

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/release-channel-test-of-quantumbar-v1/

Release Channel Test of QuantumBar v1

As a final checkpoint for a long term process, we plan launch a study with the full QuantumBar experience to 74 release users

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/release-channel-test-of-quantumbar-v1/

[Tracking Requested - why for this release]: This experiment is running in 74 as just a final go/no go confirmation before turning the pref on for everyone in 75.

status-firefox74: --- → affected
tracking-firefox74: --- → ?

We have finished testing the [PI-515] Release Channel Test of QuantumBar v1 experiment for Firefox 74.
Quality status: RED - DON’T SHIP

Why is this feature red?

  • During testing we have encountered the following issues:

  • Bug 1619547 (Blocker) - Redirect Search tips (google.com) are flicker displayed

    • Even though the issue is fixed on Firefox 75, it is still reproducible on the Firefox 74 version for users that have Google search engine set by default. These users will see the Redirect search tip only for a moment, after which it will immediately auto-dismiss. This behavior is confusing and will make the users question what the tip was about since it's auto-dismissed so quickly. Please note that this issue is reproducible for all the Google domains (see this list).
    • From the QA point of view this issue is a Blocker for the experiment.

  • Bug 1621608 (Normal) - The Onboarding tip is quickly dismissed if about:newtab page is on focus and the browser is restarted

    • We didn’t manage to reproduce this issue on Firefox 75 and above, so probably it was fixed along the way. However, this bug still affects the Quantumbar experiment for 74.

  • Bug 1621944 (Normal) - [Intermittent] The onboarding tip is displayed in about:newtab only after clicking in the Address Bar at browser normal restart

  • Bug 1621288 (Normal) - An "unenrollFailed" event is sent besides the regular "unenroll" one when unenrolling from the QuantumBar v1 experiment by manually modifying a pref

    • We managed to reproduce this issue only while testing this experiment. However, we are not sure how this issue might affect the experiment. Probably @mythmon can help with this and might have more details

  • However, we had a sync-up meeting with Mike de Boer and talked about how we can mitigate the Search Tips issues which are blocker for this experiment:

    • Our proposal is to release this experiment without the Search Tips functionality.
    • By doing this the users enrolled in this experiment will no longer encounter the Search Tips issues.

  • Besides these, we have encountered the following known issues that are reproducible on older Firefox versions (without Quantumbar enabled): Bug 1621671, Bug 1616872, Bug 1577924. From the QA point of view we don’t consider them as blockers for this experiment.

What needs to be done?

  • Discuss Bug 1621288 to see how this “unenroll failed” event affects the experiment.
  • Update the Experimenter page in order to have the treatment branch of the QuantumBar v1 experiment without the Search Tips functionality. In this case the users will no longer receive the Search Tips functionality if they are enrolled in the treatment branch.
  • If we cannot remove the Search Tips functionality of this experiment, the Sign Off for it would remain RED because of the mentioned issues regarding the Search Tips.
  • If we can exclude Search Tips from this experiment and Mythmon confirms that Bug 1621288 will not have a high impact on the experiment, we will resend the sign off as GREEN.

Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • Mac 10.15;
  • Debian 9;

Tested Firefox versions:

  • Firefox 74.0 en-US;
  • Firefox 74.0 de;
  • Firefox 74.0 es-ES;
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Release Channel Test of QuantumBar v1 → [Experiment] Pref-Flip Experiment: Release Channel Test of QuantumBar v1 Fx 74.0 to 74.0 Release
Start Date: 2020-03-12 End Date: 2020-03-31

The engineering team updated the Experimenter page to exclude the Search Tips functionality for this experiment in order to avoid the blocker issue mentioned in the previous sign-off.
Also, the Bug 1621288 was triaged and it’s not considered a blocker for this experiment.

From the QA side, we have updated the Normandy stage recipe used for testing the experiment to also exclude the Search Tips functionality. We have verified that the enrollment and unenrollment work as expected and also that the Tips are no longer triggered if enrolled in the treatment branch.

Considering this we will change the status of the sign-off from RED to GREEN.

User Story: (updated)
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.