1459678 - (blipz1) Blipz: Crowdsourced Webcompat reports

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Rosana Ardila]

Basic description of experiment: 
The experiment aims to test if crowdsourcing user feedback about web compatibility can help us detect more issues. We will prompt users to provide feedback about the page they are currently looking at and then follow up with a simplified webcompat issue report.
  
What is the preference we will be changing? 
Irrelevant for this experiment

What are the branches of the study and what values should each branch be set to? 
- Branch 1: experience without context, copy that provides little context. No link to "why am I seeing this", nor to an explanatory thank you page
- Branch 2: experience with some context, copy with more context: link from the prompts to a "why am I seeing this" page
- Branch 3: experience with context: link to "why am I seeing this" page and "thank you" page with brief explanation of webcompat and why feedback is important, as well as quick explanation to launch the experience without prompt 

What percentage of users do you want in each branch? 
Splitting them equally

What Channels and locales do you intend to ship to? 
EN
Release

What is your intended go live date and how long will the study run? 
Mid may, it should run for 2 weeks.

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions?
We will continue testing this approach if we can can get 100 reports
We will continue testing this approach if we can get valid actionable platform/product issues
From the size of the testing population, and the relative percentage of valid to invalid reports for the top reported sites that result in actionable platform or product issues, we will obtain a baseline to understand how big the crowdsourcing funnel needs to be for it to generate the same amount of valid issues as webcompat.com
With branches getting more context about web compatibility and the importance of their feedback, we are expecting users on it to react more to our prompts, or even report without a prompt, and be less frustrated with the experience
We expect to gather data on how users perceive top sites: are these performing well for users or not. This is exploratory data gathering and might serve as a baseline for monitoring top sites in the future
We will develop a standalone web extension for mobile and desktop webcompat that uses this prompting mechanism and user experience if at least 1% of the test population engages, providing at least the answer to the first question
We will continue testing with the current user experience if under 20% of the users are frustrated and choose the "never show me this again" option or reported being annoyed in the survey.

Who is the owner of the data analysis for this study?
Tim Smith

Will this experiment require uplift? 
No

QA Status of your code: 
We will update this with the request filed


Do you plan on surveying users at the end of the study? 
Yes, link to survey will follow


Link to any relevant google docs:
PHD: https://docs.google.com/document/d/1St1NtChp1yE25IeJQFVf19gCBiu0RR66UmKD9CYGSlw/edit

Experiment Design:https://docs.google.com/document/d/15NvkpnhOddsbOwGYmR83M40pipyEKX3FfsD0e043_98/edit#heading=h.fh4lrbuzzzfl

UX mockups:
https://projects.invisionapp.com/share/PSH8K1XB5QE#/screens/293769201

Comment 1

[Rosana Ardila]

Mika, Marshall, 
I wanted to know what would be the best way for you to review this experiment. The main documents are linked in the initial comment, but I'm happy to jump to a call if that is easier.

Many thanks for your help!

Comment 2

[Mika]

Hi Rosana, 

Who at Mozilla will have access to the feedback/screenshots provided by users, and where is that data stored? There is a chance that a screenshot could contain sensitive data, so it would be good to have a plan on how to handle that.

Thanks,
mika

Comment 3

[Mike Taylor [:miketaylr]]

(In reply to Mika from comment #2)
> Who at Mozilla will have access to the feedback/screenshots provided by
> users, and where is that data stored? There is a chance that a screenshot
> could contain sensitive data, so it would be good to have a plan on how to
> handle that.

Right now the plan is to file all feedback reports (with or without screenshots) as GitHub issues in a private repository at:

https://github.com/mozilla/webcompat-blipz-experiment-issues

That repo is locked down to mozilla org members, who are part of the webcompat team:

https://github.com/orgs/mozilla/teams/webcompat/members

(currently 7 people, all Mozilla employees)

We will probably also add Tim Smith, who is doing data analysis. We can further restrict to only the immediate team members working on the project, if that's helpful (~4 people).

Comment 4

[Rosana Ardila]

Mika, 

I hope that Mike could answer your question. Let us know if you need us to open a legal bug.

Thanks again so much for your help!

-Rosana

Comment 5

[Mika]

Hi Mike, 

The plan you proposed in comment 3 sounds good, especially if you can restrict to only those working on the project and Tim (if he needs the access).  If you do come across a screenshot containing sensitive info - after you handle the issue, can you  delete it from the private repo?
Thanks,
mika

Comment 6

[Mike Taylor [:miketaylr]]

(In reply to Mika from comment #5)

> The plan you proposed in comment 3 sounds good, especially if you can
> restrict to only those working on the project and Tim (if he needs the
> access).  If you do come across a screenshot containing sensitive info -
> after you handle the issue, can you  delete it from the private repo?

Yep, absolutely. We can remove the image link from the private issue, and delete the image resource from the s3 bucket.

Comment 7

[Mike Taylor [:miketaylr]]

(In reply to Mika from comment #5)
> The plan you proposed in comment 3 sounds good, especially if you can
> restrict to only those working on the project and Tim (if he needs the
> access).

Sounds good. I'll create a new Github team and restrict membership to myself and Tom during development. And probably add Rosana and Tim once we're in analysis phase.

Comment 8

[Mike Taylor [:miketaylr]]

ni? Lonnen for data steward review signoff.

Comment 9

[Lonnen :lonnen]

Some of the data collected is category 3, and this installs opt-out but prompts before sending any data. This pattern is established in crash reporting, and some of the test pilot experiments (bloc, etc) but is novel for a study to my knowledge.

I'd like a bit of UI that allows the user to inspect the payload before transmission, through a 'details' or 'more info' section in the wizard.

Bug 1460430 was filed to discuss whether we can attach the telemetry ID. As I understand it, for this first study, we will go ahead without it but may want to add it in a replication if this study is promising and the additional context would prove valuable.

conditional r+ when the above is met

Comment 10

[Mike Taylor [:miketaylr]]

(In reply to Lonnen :lonnen from comment #9)
> I'd like a bit of UI that allows the user to inspect the payload before
> transmission, through a 'details' or 'more info' section in the wizard.

I've filed https://github.com/mozilla/webcompat-blipz-experiment/issues/44 to track that work.

Comment 11

[Rosana Ardila]

Attachment: Blipz doorhanger

Lonnen, Mika,

this is our current UI iteration. The full experience is here [1]. I wanted to clarify some points that have come up during the reviews. 

- The user chooses the type of error, a textbox to provide more information and then sees the screenshot they are about to submit. Does this cover the user seeing everything they are about to submit? Or do we need to include extra information (like OS and FX version) and include a "details" section analog to the crash reporter?

- since we are only asking for feedback on 25 very popular pages (so not really a way to identify a single user), do we have to still include "include the address of this website with your feedback"?

- Do we have to include an extra sentence here around who will have access to the data, e.g. "This data will only be available for a restricted group of Mozilla employees". Would this be also something we could put in an expandable "details" section?

Many thanks for your help!

Rosana

[1] https://projects.invisionapp.com/share/SGIIXJD9FKE#/screens/295725605_V1_1-1

Comment 12

[Mike Taylor [:miketaylr]]

(In reply to Rosana Ardila from comment #11)
> - since we are only asking for feedback on 25 very popular pages (so not
> really a way to identify a single user), do we have to still include
> "include the address of this website with your feedback"?

(my understanding was yes, because we're asking for the entire URL, not just the domain)

Comment 13

[Ilana]

I can R+ for science when I see the final decision criteria.

Comment 14

[Lonnen :lonnen]

I met with Mika yesterday and we looked over the proposed UI changes in https://github.com/mozilla/webcompat-blipz-experiment/issues/44 Specifically:

- removing the checkbox and always including the URL
- adding a more details section that leads to a new pane showing exactly what info will be sent (https://user-images.githubusercontent.com/67283/40019825-fcd754c4-5785-11e8-9610-d9ec1bfdcf06.png)

With these changes, data review: R+

Comment 15

[Ilana]

Science: R+

Comment 16

[Liz Henry (:lizzard) (relman/hg->git project)]

Ryan, want to track this for 61 release?

Comment 17

[Mike Taylor [:miketaylr]]

Attachment: webcompat_blipz_experiment-1.0.0.xpi

Mythmon, can we please get this signed for QA for their testing?

Comment 18

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 19

[Mike Taylor [:miketaylr]]

Thank you!

Comment 20

[Mike Taylor [:miketaylr]]

Attachment: webcompat_blipz_experiment-1.0.0rc1.zip

Hi mythmon, could we get this RC signed for QA to verify the issues we've fixed?

thanks!

(assuming we fixed all the bugs and don't need additional builds, we can just rename back to -1.0.0.xpi)

Comment 21

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 22

[Mike Taylor [:miketaylr]]

Attachment: webcompat_blipz_experiment-1.0.0rc2.zip

hiya mythmon, QA has asked for one final build for verification. can we get this rc2 signed please? thanks!

Comment 23

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 24

[Ryan VanderMeulen [:RyanVM]]

Signing off for RelMan to go live on July 2nd at 0.1% rollout for en-US Beta62 & Release61 users.

Comment 25

[Emil Pasca [:emilpasca]]

Webcompat Blipz Experiment Shield Study
Targeted: (EN locale) Firefox Release 61, Firefox Beta 62

Hi guys,

We have finished testing the "Webcompat Blipz Experiment" Shield Study.
QA’s recommendation: GREEN - Ship it

Reasoning: 
 - All blocker and major issues that were found have been fixed and verified. 

Testing Summary:
 - Full Functional test suite: TestRail(https://testrail.stage.mozaws.net/index.php?/plans/view/10488);
 - Verified that the Telemetry pings are correctly sent;
 - Performed regression testing after reported issues were fixed;

Tested Firefox versions:
 - Firefox Release build v60.0.2 (initially targeted
 - Firefox Release build v61.0;
 - Firefox Beta build v62.0b3;	

Tested Platforms:
 - Windows 10 x64;
 - Mac 10.13.3;
 - Ubuntu 16.04 x64;

Regards,
Emil

Comment 26

[Robert (rrayborn, he/him)]

We've launched.  Excited for the results!

Comment 27

[Matt Grimes [:Matt_G]]

I've ended this study per our conversation with the team. Flagging the analyst, Tim Smith, to provide details on the outcome of the study before closing this bug.

Comment 28

[Ryan VanderMeulen [:RyanVM]]

Tim, any updates on this analysis?

Comment 29

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

The Blipz team has been discussing the findings and we're working on preparing our report. A look at the telemetry is available at https://metrics.mozilla.com/protected/tdsmith/20180719-blipz_report_draft.html.

Comment 30

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

This final report is here: https://metrics.mozilla.com/protected/tdsmith/2018-08-blipz_v1_report.html

Thanks!

Comment 31

[Jens Hausdorf :jens1o [ni? me]]

Why does this study needs to be protected?

Comment 32

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Thanks for the nudge; the report is now world-visible at https://metrics.mozilla.com/tdsmith/2018-08-blipz_v1_report.html.