1301775 - Heartbeat study on Insecure Password Warning

Status: RESOLVED FIXED

(Shield :: Heartbeat, defect)

Description

[Tanvi Vyas[:tanvi]]

Potential questions we are looking to answer:

1) Does the warning scare users way from Firefox?
2) Does the user notices the warning?
3) Does the user understand what the warning means?
4) Does the warning cause the user to try the https version of the site?
5) Does the warning cause the user to use a different password than they do for their highly sensitive accounts?
6) Does the warning keep the user from logging in or creating an account?
7) Does the warning cause developers to fix their sites?

Comment 1

[Matt Grimes [:Matt_G]]

Giving this one to Tyler. He may have some additional questions for you. What date are you looking to land this in Beta?

Comment 2

[Tyler Downer [He/Him]]

Hey Tanvi,
Let's setup some time to discuss this.

Comment 3

[Tanvi Vyas[:tanvi]]

(In reply to Tyler Downer [:Tyler] from comment #2)
> Hey Tanvi,
> Let's setup some time to discuss this.

Sure.  We should include Peter.  Peter, do we still want a heartbeat study?

As for timeline, a url bar warning for insecure passwords is in Beta 50, but not in release 50.  The url bar warning will likely go into release 51.

We are implementing a contextual warning that appears in the username and password fields itself; that will likely go into release 52.

Are heartbeat studies only run on release?

Comment 4

[Tyler Downer [He/Him]]

We can run a HB Study on any channel, typically we run them on release and Beta only, but we have access to all channels.

Comment 5

[Tyler Downer [He/Him]]

We will launch two Heartbeat Studies to support this feature:

Survey 1:
en-*, Release, 3-4k responses. Launch Nov 29th
We will present users with 3 websites (A popular social networking site, a popular link aggregator and a generic website) that have login fields. We will display the websites without the insecure password notification and with (users will be randomly selected for each branch) and then users will be asked how much they trust the website and how likely they are to use it. We are looking for differences in the websites with the prompt and without.

Survey 2:
en-*, Beta, 3k responses, Launch around Feb 1st.
We present a screenshot of the dialog, and then ask what user's impressions of it were, if it changed the likelihood that they would use that website, and the category of website they saw the prompt on it.

Comment 6

[Matthew N. [:MattN]]

(In reply to Tyler Downer [:Tyler] from comment #5)
> Survey 2:
> en-*, Beta, 3k responses, Launch around Feb 1st.
> We present a screenshot of the dialog, and then ask what user's impressions
> of it were, if it changed the likelihood that they would use that website,
> and the category of website they saw the prompt on it.

There is no new dialog so what do you mean? Do you mean the identity panel (aka. control center), the URL bar indicator, or the contextual warning in the autocomplete popup?

Comment 7

[Tyler Downer [He/Him]]

In the meeting with tanvi last week she demo'd the contextual warning in the autocomplete popup, which we will use in both surveys. The URL bar indicator is too small and most likely will not be noticed by most users, making it difficult to survey.

Comment 8

[Tanvi Vyas[:tanvi]]

Needinfo'ing myself since this is waiting on em getting screenshots.  I've been waiting for the remaining bugs to land, but maybe we should just go with what we have.

Would npr be a good website to use for the test: http://www.npr.org/oauth2/login, or should we create a fake website and host it on one of our domains?

How many websites do we need?

Comment 9

[Tanvi Vyas[:tanvi]]

I've emailed Tyler some screenshots and we will figure out next steps from there.  If anyone knows a popular site that still uses HTTP for login, please share!  Thanks!

Comment 10

[Luke Crouch [:groovecoder]]

http://www.nytimes.com/ uses insecure login.

Comment 11

[Tanvi Vyas[:tanvi]]

Tyler, nytimes is actually a good one and the warning shows up nicely in there UI.

http://www.nytimes.com/ - click login - click on username or password field - take screenshot.

Comment 12

[Tanvi Vyas[:tanvi]]

The heartbeat study is in progress.

Comment 13

[Tyler Downer [He/Him]]

Results from the first heartbeat Study, https://docs.google.com/a/mozilla.com/document/d/1sOjGnP5-X_6qdgT7J0GCx4-Mnu-8bkb6t2UGWc1x4ZY/edit?usp=sharing

Please review before wider distribution to the desktop-insights group next week.

Comment 14

[Peter Dolanjski [:pdol]]

The results look good to me and are quite straight forward.  Thanks Tyler.

Comment 15

[Tanvi Vyas[:tanvi]]

Clearing needinfo as we have gone through results with Tyler.  Thanks Tyler!