1465195 - [Shield] Pref Flip Study: Accessibility Inspector Tool

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Yura Zenevich [:yzen] (please use "needinfo")]

Basic description of experiment: 

The experimental universe is users who have ever previously engaged with devtools (the devtools.telemetry.tools.opened.version.pref is not undefined).
Users will be split equally into experiment and control cohorts. The accessibility panel will be visible by default in the devtools toolbox to the experimental cohort. It will be hidden behind a devtools preference checkbox in the control cohort.

What is the preference we will be changing?

'devtools.accessibility.enabled'

What are the branches of the study and what values should each branch be set to?

Test cohort: true
Control cohort: false

What percentage of users do you want in each branch?

Users with 'devtools.accessibility.enabled' set to false - 50%
Users with ‘devtools.accessibility.enabled’ set to true - 50%

What Channels and locales do you intend to ship to?

Dev Edition channel, Firefox 62, All locales

What is your intended go live date and how long will the study run?

Week of or the week after the 62 branch goes into Dev Edition (2018-06-25)
The study will run for two weeks after a one-week enrollment period.

Are there specific criteria for participants?

Participants must have ever opened devtools (preference `devtools.telemetry.tools.opened.version.pref` is not undefined).

What is the main effect you are looking for and what data will you use to make these decisions?

Demonstrate that adding the accessibility panel is a safe product change
Does not decrease (more than 5%): total devtools toolbox active time 
Does not decrease (more than 5%): devtools retention at 2 weeks (fraction of users who opened the toolbox in the second week of the experiment)
Assess the popularity of the accessibility panel compared to other tools
Number of total accessibility panel opens
Number of unique profiles activating the accessibility panel
Fraction of users using the accessibility panel more than n times

Who is the owner of the data analysis for this study? 

Tim Smith <[e-mail]>

Will this experiment require uplift?

No

QA Status of your code

Yellow (on April 26, 2018): PI team has been QA’ing the feature continuously for several months now. Most bugs are being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=1151468 (Note: not all of them are blockers)

Do you plan on surveying users at the end of the study?

Yes. We would like to get an overall satisfaction score and have a short questionnaire that could help with some insights about discoverability of the tool, the goods and the bads.

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists:

Original meta bug - https://bugzilla.mozilla.org/show_bug.cgi?id=1151468
Original UX spec - https://docs.google.com/presentation/d/1QIo8IkGD-s5x_qG_Qf_YK72VAkLjr4bYFddDn6KkEC0/edit#slide=id.p
Introduction article to the Accessibility Inspector tool - https://www.marcozehe.de/2018/04/11/introducing-the-accessibility-inspector-in-the-firefox-developer-tools/

Comment 1

[Yura Zenevich [:yzen] (please use "needinfo")]

Link to PHD document https://docs.google.com/document/d/1ZSCdmzVzyz_RpBGaSoPpliPLADRkv1NyeOZq64UEHvo/edit?ts=5af33741

Comment 2

[Yura Zenevich [:yzen] (please use "needinfo")]

ni? Carmin for visibility

Comment 3

[Yura Zenevich [:yzen] (please use "needinfo")]

ni? Ilana and Tim to confirm the Shield Study owner.

Comment 4

[Yura Zenevich [:yzen] (please use "needinfo")]

^

Comment 5

[Ilana]

Science review: R+

Comment 6

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

We should target 10% of the dev channel in order to power the retention endpoint.

Notes on power analysis: https://docs.google.com/document/d/1NoBsVds6I4zjnQpFTUNxdcIF2nOha2rwsX1yRSPvShw/edit?usp=sharing

Comment 7

[MarniePW [:marnie]]

Per Krupa, Cosmin is testing this.

Comment 8

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Hi team,

As Krupa mentioned I will be helping you with manual QA for Pref Flip Study: Accessibility Inspector Tool shield study. We already started testing the Accessibility features functionality on latest Firefox Dev Edition 62.0b1 build and also verifying the telemetry probes and will send a sign off mail after we finish testing.

Considering the fact that Firefox Dev Edition 62.0b1 is released, do we have a specific date when the shield study will be shipped?

Comment 9

[Yura Zenevich [:yzen] (please use "needinfo")]

Hi Robert,
I believe I have to ni? you for a sign off before sending an intent to ship email. FYI, this feature is already in 61 (under DevTools setting) and implementation is not study specific.

Comment 10

[Yura Zenevich [:yzen] (please use "needinfo")]

Cosmin is handling the QA sign off

Comment 11

[Robert Helmer [:rhelmer]]

(In reply to Yura Zenevich [:yzen] from comment #9)
> Hi Robert,
> I believe I have to ni? you for a sign off before sending an intent to ship
> email. FYI, this feature is already in 61 (under DevTools setting) and
> implementation is not study specific.

If the code+pref to be flipped by shield already received peer review and landed, then we're good here. Thanks!

Comment 12

[Cosmin Muntean [:cmuntean], Ecosystem QA]

Pref Flip Study: Accessibility Inspector Tool
Targeted version: Developer Edition 62.x

We have finished testing the Pref Flip Study: Accessibility Inspector Tool.

QA’s recommendation: YELLOW - NEEDS DISCUSSION

Reasoning:
According to the PHD document a survey should be triggered at the end of the study. Since the survey is triggered by Normandy, our testing efforts are limited, therefore we are not able to verify it. Our concern in this case is that the survey may not be correctly implemented by Normandy recipe, therefore the end users won't have access to this part of the study.

We also tested the Accessibility Inspector Tool functionality and we have found two new issues:

Bug 1470198 - "Accessible Information Unavailable" text is displayed when selecting certain iframes as the currently targeted document
Bug 1469526 - After inspecting an element from an iframe, the tree is wrongly shrank when the scrollbar of the iframe reappears/disappears

We don't consider these issues as blockers for the study, but we strongly recommend that they are fixed as soon as possible.

Testing Summary:
- Full Functional test suite: https://goo.gl/L2d2SH
- Verified that the Telemetry probes are correctly sent;
- Performed regression testing after reported issues were fixed;

Tested Platforms:
- Windows 10 x64;
- Windows 7 x64;
- macOS 10.13;
- Ubuntu 16.04 x64;

Tested Firefox versions:
- Nightly 62.0a1;
- Dev Edition 62.0b1;

Comment 13

[Yura Zenevich [:yzen] (please use "needinfo")]

Hi Rob, could you comment regarding the survey and what'd need to be QA'ed there?

Comment 14

[MarniePW [:marnie]]

I have today as the launch date for this pref flip Shield study. From Matt's email, sounds like the survey isn't a holdup for launch (see Comment 12, above). That said, I believe we need signoff from relman and the Intent to Ship email needs to be sent before the study can launch.

Comment 15

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #14)
> I have today as the launch date for this pref flip Shield study. From Matt's
> email, sounds like the survey isn't a holdup for launch (see Comment 12,
> above). That said, I believe we need signoff from relman and the Intent to
> Ship email needs to be sent before the study can launch.

Sent the intent to ship email. Marnie, would you know who from relman needs to be ni?ed here?

Comment 16

[MarniePW [:marnie]]

Yep, it's Liz Henry, she's NI'd above.

Comment 17

[Liz Henry (:lizzard) (relman/hg->git project)]

I think the purpose of an "intent to ship" email is to give a broad opportunity for cross-team feedback. So, ideally, that would come well before we ship a study.

Comment 18

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #17)
> I think the purpose of an "intent to ship" email is to give a broad
> opportunity for cross-team feedback. So, ideally, that would come well
> before we ship a study.

Hi Liz, would shipping the study some time next week be reasonable? I was basing the timing on the "How to ship a pref flip study" document that mentions a 5 day relman notice for studies on the release channel (this one is Dev Edition).

Comment 19

[Yura Zenevich [:yzen] (please use "needinfo")]

Removing ni? based on the Matt_G email regarding the survey QA.

Comment 20

[MarniePW [:marnie]]

(In reply to Yura Zenevich [:yzen] from comment #18)
> (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #17)
> > I think the purpose of an "intent to ship" email is to give a broad
> > opportunity for cross-team feedback. So, ideally, that would come well
> > before we ship a study.
> 
> Hi Liz, would shipping the study some time next week be reasonable? I was
> basing the timing on the "How to ship a pref flip study" document that
> mentions a 5 day relman notice for studies on the release channel (this one
> is Dev Edition).

Hi Yura,

Shipping next week would take some pressure off Liz and allow folks that might have concerns to comment on your Intent to Ship email. If you're good with that, we'll plan on July 2 as your ship date, pending Liz's feedback.

Comment 21

[Yura Zenevich [:yzen] (please use "needinfo")]

July 2nd would work great, thanks!

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes, that's fine with me. Thanks!

Comment 23

[MarniePW [:marnie]]

Liz, is that your official signoff for the study?

Comment 24

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes, is there a better way I should sign off? I can also answer the Intent to Ship email. I'll do that now.

Comment 25

[MarniePW [:marnie]]

Nope, this is fine. Some folks do an explicit "R+" reply in Bugzilla so I don't miss it.

Comment 26

[Matt Grimes [:Matt_G]]

This is now live for 10% of Dev Edition. According to my calendar we should end the study on July 23rd. Let me know if that date is incorrect.

Comment 27

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Matt Grimes [:Matt_G] from comment #26)
> This is now live for 10% of Dev Edition. According to my calendar we should
> end the study on July 23rd. Let me know if that date is incorrect.

Hi Matt, this might've been brought up already Tim, but the it seems like the study did not actually launch this week; as he can’t find any trace of it in telemetry, the normandy client hasn’t reported any enrollment events and main pings aren’t being decorated with the experiment stub. ni? for visibility here.

Comment 28

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Two factors emerged in discussion this morning that explained why no clients were enrolled:

* The `devtools.telemetry.tools.opened.version` pref we intended to target on was removed in Bug #1296723, which landed in 62 (the current dev edition). Some clients probably still should have had this set, but

* the Normandy recipe was written to target `devtools.telemetry.tools.opened` instead, which I'm not sure devtools had ever set on dev edition.

Happily, the second factor made the first more visible, since no clients were enrolled.

As the assigned analyst, I propose that we should remove the targeting criterion and proceed; the net effect of targeting on the preference was to exclude first-time devtools users from the experiment, which we were on the fence about anyway. The analysis did not depend on the targeting, since we will only consider the outcomes of users who were actually exposed to the experiment (i.e. opened devtools). The experiment will not affect the user experience of users who do not open devtools. The fraction of users to deploy to should not change.

Comment 29

[Matt Grimes [:Matt_G]]

We've removed the preference targeting line from the study. It is now live again. Let us know if you don't see data flowing in soon.

Comment 30

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

> According to my calendar we should end the study on July 23rd. Let me know if that date is incorrect.

Since we got a late start, can we extend this to the 26th?

Comment 31

[Matt Grimes [:Matt_G]]

No problem. I've updated our calendar accordingly.

Comment 32

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

My write-up is here: https://dbc-caf9527b-e073.cloud.databricks.com/#notebook/20292/resultsOnly

Thanks, all!