Closed Bug 1604872 Opened 6 years ago Closed 6 years ago

CoE project survey creation to learn about experiment success factors, impact, and cost.

Categories

(Product Innovation :: Project Request, task)

Product:
Product Innovation
Component:
Project Request
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: shell, Assigned: rrayborn)

Details

Request Description
Survey goal: determine how successful the experiment was, factors we think impact success/risk, factors to measure the effort, an impact estimate, and what areas the experiment related to?

This will be a non-anonymous survey offered after completing every experiment. A link will live permanently in the Results section of an experiment and we also want the ability to email a link to ask people to fill it out.

This will be a long running survey - no end-date in site. We expect to go back retrospectively for some teams to gather info - but primary focus is going forward on-going information gathering

Deadline
We are hoping to launch early-ish in the new year.

Priority Level
(1 = High 2 = Medium, 3 = Low): 2? Matt Grimes was my CoE mentor/partner for this effort. This is important as a key aspect to trying to determine where it is smartest/most successful to invest in experimentation.

Priority Level Description
Experiments support the KPI driving projects - and are currently torn between different teams trying to meet their KPIs and other experiment requests. We are trying to have ways to judge where we get the biggest bang for our buck in supporting experiments (and where we are actually spending our time) - and have data to back that up.

Supporting Information
https://docs.google.com/document/d/1FHIFo9elzRbDi_qIzXRezVUaY3WJk1E8x9Zw5V5Yqzo/edit Matt has helped us get to this stage of survey questions. Emily Thompson reviewed and we incorporated information/phrasing from questions she asked in H1 of data scientists. Now I am hoping for help writing these questions as well-written survey questions and helping make sure we are requesting the well-formatted feedback.

Assignee: nobody → rrayborn
Product: User Research → Product Innovation

I met with Shell late Dec and we decided I was ideally suited to assist with this when taking into consideration my proximity to the experiments team.

Assignee: rrayborn → jgaunt
Status: NEW → ASSIGNED

For the question "If this experiment related to KPIs, which ones? " please add the following as multi-select check boxes.
Desktop reach (desktop MAU)
Mobile reach (mobile MAU)
Firefox account relationships (FxA MAU)
Other - Write In (Required) [____________________]

we want 3 URLs for the survey to send out in 3 ways:

  1. retrospectively on older experiments in the last 6 months
  2. one to put in Experimenter results section permanently so anyone can answer after an experiment completes.
  3. one that I can mail out to people after an experiment ends - to solicit their feedback.

We also will want to ability to group multiple responses from the same experiment , based on matching text responses to this question "Please add the Experimenter URL from this study". This is to avoid experiments with only one response being totally discounted based on experiments with many responses.

Changes:
Okay, I fixed that question, added a question on page one to gate on experiments that are in-fact complete (I think this is a good addition), and added validation to force a complete experimenter URL.

URLs:
So the easy way to do this is to use SG's URL parameters, so we can do:

  1. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=old
  2. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=experimenter
  3. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=mail

The advantage of this approach is that you can add whatever new origins you want and SG will handle them appropriately without set up. The disadvantage is that it's transparent to the survey taker, and that respondents could copy/paste without the url params.

The alternative would be hardcoded URLs, let me know if you prefer that.

Analysis:

We also will want to ability to group multiple responses from the same experiment , based on matching text responses to this question "Please add the Experimenter URL from this study". This is to avoid experiments with only one response being totally discounted based on experiments with many responses.

I'm not confident that SG supports grouping on freetext fields in a usable way; let me experiment here. Worst case we load the live-updating CSV into Google Sheets for additional analysis.

Assignee: jgaunt → rrayborn

@Shell - thoughts here? Do you feel good about where we're at? Are we okay with the constraint that SG doesn't make grouping by a text string easy/possible(?) for their built-in analysis?

Flags: needinfo?(sescalante)

(In reply to Robert (rrayborn, he/him) from comment #3)

Changes:
Okay, I fixed that question, added a question on page one to gate on experiments that are in-fact complete (I think this is a good addition), and added validation to force a complete experimenter URL.

It doesn't hurt anything - but I don't think it adds anything except another question. The survey can only be displayed in Experimenter when it moves to "Complete" - as it will be in the Results section (hidden until complete). The other links I will only send out after an experiment ends.

URLs:
So the easy way to do this is to use SG's URL parameters, so we can do:

  1. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=old
  2. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=experimenter
  3. https://qsurvey.mozilla.com/collab/Experimenter-Study-Feedback?origin=mail
    The advantage of this approach is that you can add whatever new origins you want and SG will handle them appropriately without set up. The disadvantage is that it's transparent to the survey taker, and that respondents could copy/paste without the url params.

Perfect - thank you! The way you set this up looks like it will work great.

The alternative would be hardcoded URLs, let me know if you prefer that.

Analysis:

We also will want to ability to group multiple responses from the same experiment , based on matching text responses to this question "Please add the Experimenter URL from this study". This is to avoid experiments with only one response being totally discounted based on experiments with many responses.

I'm not confident that SG supports grouping on freetext fields in a usable way; let me experiment here. Worst case we load the live-updating CSV into Google Sheets for additional analysis.

OK - figured we'd ask and we can cover that issue if we hit it. I highly suspect that we will =\ - but sounds like we can analyze around it.

Flags: needinfo?(sescalante)

(In reply to Robert (rrayborn, he/him) from comment #4)

@Shell - thoughts here? Do you feel good about where we're at? Are we okay with the constraint that SG doesn't make grouping by a text string easy/possible(?) for their built-in analysis?

Yup - I'm good with everything except that extra question that was added for if it is complete. It doesn't hurt - but doesn't add anything - so I'd prefer it was removed if possible.

Flags: needinfo?(rrayborn)

Thanks for clarifying the use-case. Fixed :)

Let us know if you have other questions/comments/requests.

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Flags: needinfo?(rrayborn)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.