1292679 - Custom form for Shield Studies

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Custom Bug Entry Forms, defect)

Description

[Matt Grimes [:Matt_G]]

Hey folks! We'd like to create a custom bugzilla form for deploying shield studies. 

The meta bug should live under Shield: Shield Studies and be titled as follows “[SHIELD] Title of experiment” where the user inputs the title.
CC List:
[e-mail]
[e-mail]
[e-mail]



Creating this bug should spawn several blocker bugs as follows:

[Blocker 1]
Component - Shield: Shield Study 
Title - “[SHIELD] Study Validation Review for [Title of experiment]”
CC List:
[e-mail]
[e-mail]
[e-mail]


Description:
* Define study branches in the user story of the bug

Power test (for experiments):
* What is the main effect you are looking for?  If not looking for an effect, what is the total number of participants needed (and be sure to state why you need that many)?

* How will this effect be measured? (What is the instrumentation?  E.g. X% more windows opened, etc).

* What is the required effect size to draw a conclusion?

* Given that effect size, what is the sample size needed?

Roadmap:
* If there is prior art (e.g. testpilot, usertesting.com, field research, etc.), review it in the bug (and link as necessary)

* If there are previous results (particularly if they allow for the removal of experimental branches), review them here.

* State whether or not you intend to test your add-on internally (moco-wide) prior to launch

**The bug must be CLOSED as “resolved fixed” to indicate sign-off achieved**


[Blocker 2]
Component - Shield: Shield Study
Title - “[SHIELD] Data review for [Title of Experiment]”
CC List:
[e-mail]
[e-mail]
[e-mail]

Description:
You must identify and document all basic and custom instrumentation to address your hypothesis (including extended Telemetry data collection)

Your bug must include a sample payload that your study will submit via Unified Telemetry.  This must reside in the /docs/metrics.md of your repository or some similar location (core to the add-on code).

Your bug must answer the following questions:
* If you have extended Telemetry or custom instrumentation, why are you taking these additional measurements?  (E.g. these measurements are outcome measures of the study)
* How long will the study run?
* Who is the owner of the data analysis for this study?
* Does your data analyst have an analysis plan?  (An outline of a strategy for analysis is sufficient)
* Will your study require third-party data collection?  E.g. Google analytics, loggly, any Apache server logs, etc.


**The bug must be CLOSED as “resolved fixed” to indicate sign-off achieved**


[Blocker 3]
Component - Legal: Firefox
Title - “[SHIELD] Legal review for [Title of Experiment]”
Bug should be marked confidential
CC List:
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]

Description:

* Include links to consent form and debrief survey for legal review
* Get consent form template from Firefox Strategy and Insights
* Draft privacy bullets based on data documentation. Be prepared to link to the documentation (the /docs/metrics.md file).


**Legal must confirm within the bug and mark it as CLOSED and Resolved Fixed to indicate sign-off achieved.**




I think that's everything. Let me know if you need clarification. Thanks!

Comment 1

[David Lawrence [:dkl]]

Normally we have one form that has all of the questions combined and then we create the blocker bugs on the backend with the appropriate answers split out. Can you create a template that has the questions as they would appear on a single page for us? We can separate it into sections if there are no common questions.

Also we would know what type of field each question is, such as simple text, large text, checkbox, drop down, etc. Also which ones are required to answer or not.

Thanks
dkl

Comment 2

[Matt Grimes [:Matt_G]]

Ok. Here's a link to a doc I put together: https://docs.google.com/a/mozilla.com/document/d/1xUHpUIyfDSyqDsfpty3cWvfOPyDxQarbo5xbq8K0zhk/edit?usp=sharing

1. It has all the questions and what sort of field they require.
2. I put in the names of the bugs they should spawn that coincide with each section. You can take those out if you like or leave them in. Your call.
3. Anything in red is a duplicate question. We don't want the form to require answers to duplicate questions, but we do want the answers to appear in the appropriate bugs.
4. I forget that there is another bug called [SHIELD] Shipping status for [Title of Experiment]. I have added that to the document.
5. All bugs (including the meta bug) should initially be assigned to the bug submitter.

Let me know if you have questions. Thanks!

Comment 3

[David Lawrence [:dkl]]

(In reply to Matt Grimes [:Matt_G] from comment #2)
> Ok. Here's a link to a doc I put together:
> https://docs.google.com/a/mozilla.com/document/d/
> 1xUHpUIyfDSyqDsfpty3cWvfOPyDxQarbo5xbq8K0zhk/edit?usp=sharing
> 
> 1. It has all the questions and what sort of field they require.
> 2. I put in the names of the bugs they should spawn that coincide with each
> section. You can take those out if you like or leave them in. Your call.
> 3. Anything in red is a duplicate question. We don't want the form to
> require answers to duplicate questions, but we do want the answers to appear
> in the appropriate bugs.
> 4. I forget that there is another bug called [SHIELD] Shipping status for
> [Title of Experiment]. I have added that to the document.
> 5. All bugs (including the meta bug) should initially be assigned to the bug
> submitter.
> 
> Let me know if you have questions. Thanks!

Thanks for the updated specs. I forgot to ask earlier, but what is the timeframe for this that you need it by? This no longer sounds like a quick form and will take a bit of time to implement, debug, feedback, etc. 

Thanks
dkl

Comment 4

[Matt Grimes [:Matt_G]]

Sooner is better, since we spend a lot of time putting this all together manually. It is not a fire though. We can keep doing it manually for a few weeks if needed.

Comment 5

[David Lawrence [:dkl]]

Going to start working on this this week. Should have something up for your feedback/testing soon.

dkl

Comment 6

[David Lawrence [:dkl]]

After some delay I now have a prototype for you to test out and give feedback. It is installed on our testing instance so you can try it out and let me know of any issues.

The CC list members we can add by putting them on the default cc list for the Shield Study component which is preferable over hardcoding them in the Bugzilla code. That way they can be altered easily if needed.

https://bugzilla-dev.allizom.org/form.shield.studies

To https://github.com/mozilla-bteam/bmo.git
   a41aa41..e74e2e2  development -> development

dkl

Comment 7

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

+1 approved.  This is an improvement over current process and will relieve some of :mgrimes' work.

(there will be revisions sometime In The Future :) )

Comment 8

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

Attachment: screen shot of the form flow

Comment 9

[David Lawrence [:dkl]]

(In reply to Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat ) from comment #7)
> +1 approved.  This is an improvement over current process and will relieve
> some of :mgrimes' work.
> 
> (there will be revisions sometime In The Future :) )

So does this mean it is cool to be pushed live? Did you go through the steps of creating a test bug or two and are fine with the end result? Just want to be clear :)

Comment 10

[Josh Gaunt [:jgaunt]]

This seems like it could be a good place to enforce standardization of crucial fields in the telemetry ping payloads, per https://github.com/mozilla/shield-studies-addon-utils/issues/44

Comment 11

[David Lawrence [:dkl]]

(In reply to David Lawrence [:dkl] from comment #9)
> (In reply to Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )
> from comment #7)
> > +1 approved.  This is an improvement over current process and will relieve
> > some of :mgrimes' work.
> > 
> > (there will be revisions sometime In The Future :) )
> 
> So does this mean it is cool to be pushed live? Did you go through the steps
> of creating a test bug or two and are fine with the end result? Just want to
> be clear :)

Ping. Are we okay to push this live or do more changes need to be made?

dkl

Comment 12

[Matt Grimes [:Matt_G]]

Looks great. Please push it live!

Comment 13

[David Lawrence [:dkl]]

To https://github.com/mozilla-bteam/bmo.git
   ee119fd..f2fb476  master -> master

Will be in next weeks push to production.

dkl

Comment 14

[David Lawrence [:dkl]]

This form should be live now.

dkl