1312950 - [SHIELD] Study Validation Review for Unified URL Bar

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Matt Grimes [:Matt_G]]

Please supply the following:
* Define study branches in the user story of the bug
* What is the main effect you are looking for?  If not looking for an effect, what is the total number of participants needed (and be sure to state why you need that many)?
* How will this effect be measured? (What is the instrumentation?  E.g. X% more windows opened, etc).
* What is the required effect size to draw a conclusion?

Prior art:
* If there is prior art (e.g. testpilot, usertesting.com, field research, etc.), review it in the bug (and link as necessary)

* If there are previous results (particularly if they allow for the removal of experimental branches), review them here.

* State whether or not you intend to test your add-on and how you'll go about doing this

Comment 1

[Javaun Moradi [:javaun]]

Attachment: fig1.png

Comment 2

[Javaun Moradi [:javaun]]

Thanks so much Matt. Given our past conversations I'd like to have someone review the experiment design. We had a rough guess on enrollment needs, but that will depend on effect size, confidence, error rate, etc. Shouldn't I wait on analyst review before I attempt to answer those? 

 We have more detail in an internal doc. 

(In reply to Matt Grimes [:Matt_G] from comment #0)
> Please supply the following:
> * Define study branches in the user story of the bug

Done. There is more detail in an internal google doc. I can share. 

> * What is the main effect you are looking for?  If not looking for an
> effect, what is the total number of participants needed (and be sure to
> state why you need that many)?

User satisfaction, overall search volume. More details in doc. The original conversation suggested 3000 (3x1000 for our branches) but we somehow landed on 25% CRTL 1/25% CTRL 2/50% test group and I don't recall why


> * How will this effect be measured? (What is the instrumentation?  E.g. X%
> more windows opened, etc).
Telemetry probes for searches conducted. 
Heartbeat for satisfaction

> * What is the required effect size to draw a conclusion?
> 
> Prior art:
> * If there is prior art (e.g. testpilot, usertesting.com, field research,
> etc.), review it in the bug (and link as necessary)

Bill's latest usertesting.com trial, using our Shield Add-on: 
https://docs.google.com/a/mozilla.com/document/d/1wEdPbDpOqgsDGxhrBRyh5A8NYu_XeYR4CTijDjWFJTo/edit?usp=sharing

There was a prior Telemetry experiment but the branches weren't designed in a way that we had confidence in the results, I'm told. 

> 
> * If there are previous results (particularly if they allow for the removal
> of experimental branches), review them here.
> 
> * State whether or not you intend to test your add-on and how you'll go
> about doing this

Comment 3

[Javaun Moradi [:javaun]]

Gregg initially looked at the requirements doc we had. Would he do the statistical review? Or is that something Spenrose, Dzeber, or Amit could do? We have a lot of analysts familiar w/ search.

Comment 4

[Matt Grimes [:Matt_G]]

Gregg can approve experimental design. I believe he already has via email, but we'd need it here as well. I think he may have already provided guidance on sample sizes as well, but would need to confirm that. Once the experiment is live, you'll need to identify an analyst that is going to pull data for you and do the final analysis. I think spenrose, dzeber, or amit would be great suggestions for that. Let's make sure that is clearly stated in the data review bug.

Comment 5

[[e-mail]]

Seconding Matt_G above.  I suspect that it will ultimately fall upon the shoulders of dzeber or Amit to perform the analysis.

Comment 6

[Panos Astithas (he/him) [:past] (please ni?)]

I believe this work is now completed.