Closed Bug 1623382 Opened 6 years ago Closed 5 years ago

[Experiment] Pref-Flip Experiment: Enable Service Worker Debugging on DevEdition Fx 77.0 to 78.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox74 unaffected, firefox75 unaffected, firefox76 unaffected, firefox77 fixed, firefox78 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox74 --- unaffected
firefox75 --- unaffected
firefox76 --- unaffected
firefox77 --- fixed
firefox78 --- fixed

People

(Reporter: experimenter, Assigned: Harald)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

devtools.debugger.features.windowless-service-workers

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Debugging Enabled 50%:

Value: true

Control with Service Worker Debugging enabled
        
- Control Debugging Disabled 50%:

Value: false

Control with Service Worker Debugging disabled
        

    What version and channel do you intend to ship to?

72% of Beta Firefox 77.0 to 78.0

    Are there specific criteria for participants?


Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

May 12, 2020 - Jun 16, 2020 (35 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

The following performance metrics should not increase (>5%):

- `MEMORY_TOTAL`
- `GC_MAX_PAUSE_MS_2`
- `DEVTOOLS_COLD_TOOLBOX_OPEN_DELAY_MS`
- `DEVTOOLS_WARM_TOOLBOX_OPEN_DELAY_MS`
- `DEVTOOLS_TOOLBOX_PAGE_RELOAD_DELAY_MS`

Engagement in the following should remain the same (5%):

- `DEVTOOLS_[TOOLBOX|INSPECTOR|WEBCONSOLE|JSDEBUGGER|NETMONITOR]_OPENED_COUNT`

    Who is the owner of the data analysis for this experiment?

cdowhygelund@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

Experiment testing: https://jira.mozilla.com/browse/PI-612
TBD

Feature testing: https://jira.mozilla.com/browse/PI-362
76 Test report GREEN
77 Test report GREEN

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/service-worker-debugging-performance-experiment/

Service Worker Debugging Performance Experiment

DevTools new feature, Service Worker Debugging, has had performance leak regressions during development that got addressed. We might miss other performance/memory issues that are not covered by automation. To answer that question, we want to a/b test the feature on DevEdition and validate that performance/memory does not regress.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-167
More information: https://experimenter.services.mozilla.com/experiments/service-worker-debugging-performance-experiment/

Blocks: 1603203
No longer blocks: 1603190

We need to make sure we enable this first in Nightly (1603190), so this slips to 77 as we depend on DOM team in bug 1618546.

status-firefox74: --- → unaffected
status-firefox75: --- → unaffected
status-firefox76: --- → unaffected
status-firefox77: --- → ?
Depends on: 1603190

We have finished testing the DevTools Experiment: Service Worker Debugging experiment for Firefox Developer Edition 77.0bx.
The experiment targets Firefox Developer Edition 77 and 78. Considering the fact that there is no Firefox Dev Edition 78 build available yet, we will perform another round of testing during the Nightly code freeze on what will be the latest Nightly 78.0a1 available since there will be only a few changes for this version during that time and it will be more stable.

Quality status: GREEN - SHIP IT

Why is this experiment green?

  • No issues have been found during testing.

Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • Mac 10.15;
  • Ubuntu 18.04 x64;

Tested Firefox versions:

  • Firefox Developer Edition 77.0b5 en-US;
  • Firefox Developer Edition 77.0b5 de;
  • Firefox Developer Edition 77.0b5 es-ES;
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Service Worker Debugging Performance Experiment → [Experiment] Pref-Flip Experiment: Enable Service Worker Debugging on DevEdition Fx 77.0 to 78.0 Beta
Start Date: 2020-05-20 End Date: 2020-06-24

DevTools Experiment: Service Worker Debugging
Firefox Developer Edition 78.0bx

We have finished testing the DevTools Experiment: Service Worker Debugging experiment for Firefox Developer Edition 78.0bx.

The experiment is already live and since will be also live during Dev Edition 78 we have performed another round of testing during the Nightly 78.0a1 code freeze to ensure that there are no new issues with the experiment between the Firefox versions.

Quality status: GREEN

Why is this experiment green?

During testing, we have encountered a new issue (Bug 1641561). However, this is a feature bug which it’s not considered of high impact for the experiment and for the end user experience.
Testing Summary:

Tested Platforms:

  • Windows 10 x64;
  • macOS 10.15;
  • Ubuntu 18.04 x64;

Tested Firefox versions:

  • Nightly 78.0a1 en-US;
  • Nightly 78.0a1 de;
  • Nightly 78.0a1 es-ES;
User Story: (updated)
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.