Closed Bug 1580353 Opened 5 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Add-on Blocklisting using Remote Settings Fx 71.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox71+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox71 + fixed

People

(Reporter: experimenter, Assigned: esmyth)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

extensions.blocklist.useXML

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment XML blocklist cohort 50%:

Value: true

Members of this cohort will use the current blocklist implementation based on the daily download of a large XML file.
        
- Treatment RemoteSettings blocklist cohort 50%:

Value: false

Members of this cohort will use the proposed blocklist implementation based on RemoteSettings which provides differential JSON-formatted updates.
        

    What version and channel do you intend to ship to?

20% of Beta Firefox 71.0

    Are there specific criteria for participants?

Prefs:
extensions.blocklist.enabled == True
Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 28, 2019 - Nov 11, 2019 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success

No significant difference (< 5%) of Remote Settings blocklist efficacy when compared to the XML-based implementation.

No significant increase (≤ 5%) in crash rate in:
  - overall crash reports
  - OOM crash reports
  - shutdown crashes

No significant regressions (≤ 5%) on the following telemetry probes:
  - timestamps.first_paint
  - timestamps.about_home_topsites_first_paint
  - FX_PAGE_LOAD_MS_2,
  - FX_TAB_SWITCH_COMPOSITE_E10S_MS

At least 2.5% improvement to browser startup time measured by:
  - timestamps.first_paint

Do you plan on surveying users at the end of the experiment? No.

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Signed-off as GREEN. For more information see https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/#comment495.
PI-160 (https://moz-pi-test.atlassian.net/servicedesk/customer/portal/9/PI-160)

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/

Add-on Blocklisting using Remote Settings

We believe that migrating the add-on blocklist from a static XML file to use Remote Settings will result in a 2.5-7.5% improvement to browser startup time. We also need to validate that the change will not break the addon blocklist functionality particularly via crashes/failure to load the blocklist or false negatives for blocklist checks.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/add-on-blocklisting-using-remote-settings/

[Tracking Requested - why for this release]: Given risks associated with this change, our risk mitigation plan requires experimental validation. FF70b is the first beta period available following the approval of the risk assessment and mitigation plan.

tracking-firefox71: --- → ?

(In reply to esmyth from comment #1)

F70b is the first beta period available following the approval of the risk assessment and mitigation plan.

Do you mean to have this experiment in 70beta or 71beta?

Flags: needinfo?(esmyth)

71 beta. I must've been thinking about something else when I wrote that comment.

Flags: needinfo?(esmyth)

Add-on Blocklisting using Remote Settings
Beta 71

We have finished testing the Add-on Blocklisting using Remote Settings experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any new issues and the only known issue is a Normandy related one that is unlikely to be encountered by users.

Testing Summary:

  • TestRail: Test suite

Tested Platforms:

  • Windows 10 x64
  • Ubuntu 16 x64
  • MacOS 10.15

Tested Firefox versions:

  • Firefox Nightly 71
  • Firefox Beta 71

Regards,
Robert Martin

User Story: (updated)
Summary: [Experiment]: Pref-Flip: Add-on Blocklisting using Remote Settings → [Experiment] Pref-Flip Experiment: Add-on Blocklisting using Remote Settings Fx 71.0 Beta
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
status-firefox71: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.