Closed Bug 1583396 Opened 3 months ago Closed 2 months ago

[Experiment] Pref-Flip Experiment: Unimplemented-CSS use counters in beta Fx 70.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(firefox70+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox70 + fixed

People

(Reporter: experimenter, Unassigned)

References

(
URL
)

Details

User Story

Experiment Type: Pref Flip Experiment

    What is the preference we will be changing

layout.css.use-counters-unimplemented.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment collection 100%:

Value: true

We will collect data from these users.
        

    What version and channel do you intend to ship to?

5% of Beta Firefox 70.0

    Are there specific criteria for participants?

app_build_id > '20190926' (or display version ≥ 70b10)
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 01, 2019 - Oct 08, 2019 (7 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

This is not an experiment; we want to use this for measurement.

    Who is the owner of the data analysis for this experiment?

tdsmith@mozilla.com

    Will this experiment require uplift?

False

    QA Status of your code:

In progress. Filed https://jira.mozilla.com/browse/PI-306

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/unimplemented-css-use-counters-in-beta/

Unimplemented-CSS use counters in beta

Enable unimplemented-CSS use counters for a small random sample of the beta channel in order to measure Firefox's compatibility with the Web.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

More information: https://experimenter.services.mozilla.com/experiments/unimplemented-css-use-counters-in-beta/

[Tracking Requested - why for this release]: We intend to ship this experiment to beta 70.

tracking-firefox70: --- → ?

Unimplemented CSS use counter collection in beta
Firefox Beta 70.0bx

We have finished testing the Unimplemented CSS use counter collection in beta experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • We haven’t found any issues during testing. Still, we would like to mention that the PI request was assigned and tested only today due to the limited timeline (the launch date is set on October 1st).

Testing Summary:

  • Verified that the “layout.css.use-counters-unimplemented.enabled” pref can be set to “true” value.
  • Verified that the “layout.css.use-counters-unimplemented.enabled” pref doesn’t reset to its default value after restarting/reopening the browser.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram is generated after visiting and closing a website with CSS properties that Firefox doesn’t support yet.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram changes its value after visiting and closing several websites with CSS properties that Firefox doesn’t support yet using the following actions:
    • “Close tab” button
    • “Close Other Tabs” option from context menu
    • “Close Tabs to the Right” option from context menu
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram changes its value after visiting and closing several websites with CSS properties that Firefox doesn’t support yet on a New Window.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram changes its value after visiting and closing several websites with CSS properties that Firefox doesn’t support yet on a New Private Window.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram changes its value after reloading several websites with CSS properties that Firefox doesn’t support yet.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram changes its value after closing 2 duplicated websites with CSS properties that Firefox doesn’t support yet.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram doesn’t change its value after visiting and closing websites with CSS properties that Firefox supports.
  • Verified that the “USE_COUNTER2_CSS_PROPERTY_Zoom_DOCUMENT” histogram starts counting from 0 after restarting/reopening the browser.

Tested Platforms:

  • Windows 10 x64
  • Ubuntu 16.04 x64
  • macOS 10.14.5

Tested Firefox versions:

  • Firefox Beta 70.0b10 (en-US, fr)

Thanks very much for the fast testing!

User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: Unimplemented-CSS use counters in beta → [Experiment] Pref-Flip Experiment: Unimplemented-CSS use counters in beta Fx 70.0 Beta
Start Date: 2019-10-01 End Date: 2019-10-08
Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
status-firefox70: affected → fixed
User Story: (updated)
You need to log in before you can comment on or make changes to this bug.