Closed Bug 1668001 Opened 4 years ago Closed 4 years ago

[Experiment] Pref-Flip Experiment: (Beta only) Validate that the new print UI does not regress Firefox print user experience Fx 82.0 to 82.0 Beta

Categories

(Shield :: Shield Study, task, P3)

Product:
Shield
Component:
Shield Study
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: experimenter, Assigned: RT)

References

(
URL
)

Details

User Story

Delivery Type: Pref Flip Experiment

    What is the preference we will be changing

print.tab_modal.enabled

    What are the branches of the experiment and what values should
    each branch be set to?

- Treatment Treatment 99%:

Value: true

Modal UI print dialogue
        
- Control Control 1%:

Value: false

Old Print dialogue
        

    What version and channel do you intend to ship to?

50% of Beta Firefox 82.0 to 82.0

    Are there specific criteria for participants?

Prefs:

Experiments:

Any additional filters:
Countries: all

Locales: all

    What is your intended go live date and how long will the experiment run?

Oct 05, 2020 - Oct 19, 2020 (14 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

We will measure experiment success as follows:
- Around 50k DAU print with the new UI during the test (https://sql.telemetry.mozilla.org/queries/72640#182114)
- Ability to monitor print error/print success rate and identify segments of users with higher failure rates
- Monitor average prints per existing and new user and identify segments of users with deviations when compared to control
- Monitor incoming bugs

    Who is the owner of the data analysis for this experiment?

None

    Will this experiment require uplift?

False

    QA Status of your code:

Pending validation that no QA is necessary

Signed-off as YELLOW as per comment https://experimenter.services.mozilla.com/experiments/beta-only-validate-that-the-new-print-ui-does-not-regress-firefox-print-user-experience/#comment1170

    Link to more information about this experiment:

https://experimenter.services.mozilla.com/experiments/beta-only-validate-that-the-new-print-ui-does-not-regress-firefox-print-user-experience/

(Beta only) Validate that the new print UI does not regress Firefox print user experience

Validate that the new print UI does not regress Firefox print user experience on Beta.

Experimenter is the source of truth for details and delivery. Changes to Bugzilla are not reflected in Experimenter and will not change delivery configuration.

Data Science Issue: https://jira.mozilla.com/browse/DO-328
More information: https://experimenter.services.mozilla.com/experiments/beta-only-validate-that-the-new-print-ui-does-not-regress-firefox-print-user-experience/

QA support for Print UI Beta 82 roll-out
Firefox Beta 82

We have finished testing the [PI-819] QA support for Print UI Beta 82 roll-out experiment.

Quality status: YELLOW - SHIP IT CONDITIONALLY

Why is this experiment yellow?

  • We encountered several issue that have a negative impact on the user experience:
    Bug 1669187 - Can't set page orientation to landscape without printing in portrait first when print.tab_modal.enabled is set to false - major
    Bug 1668621 - Setting custom print margins no longer works for users who haven't printed before, as of v81 - major
    Bug 1669188 - The print shortcut is wrongly displayed when the print.tab_modal.enabled pref is set to false until the Firefox menu is displayed a second time - normal
    Bug 1668208 - Save to PDF's print preview cuts off content in vertical frames test page - normal
    Bug 1668813 - Color mode dropdown is not locked for Black&White printers - normal

What needs to be done?

  • The following actions need to be taken in order for this experiment to be GREEN:
    [Engineering] The issues mentioned in the above section should be addressed by the Engineering team.
    [Engineering] Despite the issues raised by both Ecosystem and Release Desktop QA teams, the experiment can be launched to users from the Beta channel. However we would not recommend doing the same in Release unless the population size was much smaller.
    [QA] Should verify the bug reports once they are fixed or addressed.

Testing Summary:

  • Verified printing to PDF in both Portrait and Landscape mode using the old and new UI.
  • Verified printing with a printer in both Portrait and Landscape mode using the old and new UI.
  • Verified all “Scale” options using the old and new UI.
  • Verified that images are correctly printed using the old and new UI.
  • Verified that documents with custom Margins and headers/footers are correctly printed.
  • Verified that the “Simplify Page” page option works as expected and that simplified documents are printed correctly.
  • Verified that all the “Page Range” options work as expected and documents with multiple copies are printed correctly.
  • Verified that documents are correctly printed with different paper sizes.
  • Verified that different pages and documents are correctly displayed.
  • Verified that the user is correctly enrolled in the experiment using Normandy Stage server.
  • Verified that the targeted prefs are correctly set for the treatment and control branches after enrolling in the experiment.
  • Verified that the experiment remains active after browser update.
  • Verified that the user is unenrolled after disabling the recipe from Normandy stage server.
  • Verified that the user is unenrolled after removing the experiment from the “about:studies” page.
  • Verified that the user is unenrolled if one of the targeted prefs is manually changed.

Tested Platforms:

  • Windows 10 x64
  • macOS 10.15
  • Linux MX 4.19
  • Linux Mint 20

Tested Firefox versions:

  • Firefox Beta 82.0b5 - en-US (BuildID 20200929175845)
  • Firefox Beta 82.0b7 - en-US (BuildID 20201002191150)
  • Firefox Nightly 83.0a1 - en-US (Build ID 20201004212809)
User Story: (updated)
Summary: [Experiment]: Pref-Flip Experiment: (Beta only) Validate that the new print UI does not regress Firefox print user experience → [Experiment] Pref-Flip Experiment: (Beta only) Validate that the new print UI does not regress Firefox print user experience Fx 82.0 to 82.0 Beta
Start Date: 2020-10-06 End Date: 2020-10-20
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.