1002003 - Run user testing on log-in

Status: RESOLVED FIXED

(Webmaker Graveyard :: User Testing, defect)

Description

[Matt Thompson (@OpenMatt :OpenMatt)]

* Ask users to sign up for an account. Are the reasons for doing so clear? Would you know where to click? What about opting into email updates? Where are people getting stuck? 

* In many ways, this is the most crucial part of our Engagement Ladder. Let's test, document insights, and ship agile improvements based on what we learn.

Comment 1

[Matt Thompson (@OpenMatt :OpenMatt)]

Larger context for this work: https://wiki.mozilla.org/Webmaker/Engagement_Ladder

Comment 2

[Matt Thompson (@OpenMatt :OpenMatt)]

* Adding Cori and Tony, in case they want to join in the fun at MoFo All Hands  :) 
https://wiki.mozilla.org/AllHands/Funnel_Inside#Learning_from_Users:_getting_good_at_usability_testing

Comment 3

[David Ascher (:davida)]

Would be good to add quant to the qual - what is our conversion ratio, and can we user-test alternative signup flows? I agree this is quite likely the highest impact knob when it comes to non-event-driven signups.

Comment 4

[Karen Smith]

Etherpad created to hold a creation of the script or steps https://etherpad.mozilla.org/WebmakerTestingAccount

Comment 5

[Karen Smith]

Review of v1 of a script is requested from Cassie and Matt.  Outstanding issues include:
- timeline (when do we want to test this)
- link (are we testing the live site for this?) 

Matt, can you help me to figure out appropriate whiteboard tags to get this added to the wiki here: https://wiki.mozilla.org/Webmaker/Engagement_Ladder

Comment 6

[Matt Thompson (@OpenMatt :OpenMatt)]

[engagement] [testing] 

R+ from me on script. Over to Cassie.

Comment 7

[Cassie McDaniel [:cassiemc]]

Left quite a few comments / suggestions in the etherpad. So R- until those are addressed, but this is an excellent start. It may also be worthwhile reviewing the data we have BEFORE we run this test, so that we can test further assumptions or gain more understanding. So I am switching the dependency with bug 1002312.

I've added the whiteboard tags [engagement] [testing] which should make it show up in the bugzilla list under usability testing.

Also, FYI Karen, instead of flagging 'need info' when you'd like a review, add the link as an attachment (paste as text), then flag that attachment for a review. I can show you how to do this in person, if you'd like. "Review" generally takes priority over 'need info' in my process, and it ensures that we are reviewing the correct link or piece of content.

Comment 8

[Cassie McDaniel [:cassiemc]]

From Karen via email:
"We have an etherpad of a script drafted here but honestly, in working through the practicalities of implementing on usertesting.com I don't think it's worth doing.  This is mainly going to test Persona and take too long to get to any Webmaker insights. "

I agree, and move we tackle this bug by using Matthew's new login prototype here: http://xmatthewx.github.io/webmaker-login/prototype/index.html

He has some final adjustments to make, but then we will be able to test a new UX quite nicely as opposed to Persona. Have made this bug depend on that prototype, bug 1014790

Comment 9

[Matt Thompson (@OpenMatt :OpenMatt)]

* next train, given log-in work

Comment 10

[Karen Smith]

Matthew started a script here https://webmaker.etherpad.mozilla.org/login-user-testing that can be used for user testing.

Comment 11

[Matt Thompson (@OpenMatt :OpenMatt)]

* Not sure if this will be ready to begin testing on [July11] train, or if we should push to next

Comment 12

[Brett Gaylor [:brett]]

Karen would like to see some of these new features tested - are you available this sprint?

Comment 13

[Karen Smith]

Attachment: Matthew's script for login testing

Comment 14

[Karen Smith]

Matthew's script is drafted as per the attachment.  I have very limited time before July 11th but can help get this set up if it is the right testing direction.  Please advise Brett or Cassie.

Comment 15

[Matt Thompson (@OpenMatt :OpenMatt)]

* Hey, Karen. The script looks good -- I made a small tweak to the opening. 
* Talked to Brett and Cassie -- feel free to move ahead.

Comment 16

[Cassie McDaniel [:cassiemc]]

Karen does this exist as a test script on usertesting.com or just in the etherpad? If the latter, do you have the bandwidth to get it into usertesting.com? (if not, no worries, just let me know)

Comment 17

[Karen Smith]

I will get it done today and update here when it's complete.

Comment 18

[Karen Smith]

Ready to deploy. See: 

"Webmaker Sign In - July 2014 - version A"
"Webmaker Sign In - July 2014 - version B"
"Webmaker Sign In - July 2014 - version C"

Comment 19

[Matt Thompson (@OpenMatt :OpenMatt)]

* Is this blocked on anything?

Comment 20

[Matthew Willse [:mattheww]]

Karen is away now. These are queued at the top of the drafts:
https://www.usertesting.com/dashboard/drafts

I spot checked things, but I'm not familiar with usertesting.com. If Cassie and/or Matt have a minute to make sure nothing is broken, that'd be great. 

I do believe that these are good to launch. We just need someone with the credentials and the power to click Go...

Comment 21

[Matthew Willse [:mattheww]]

Matt, do you have the power to launch a test on usertesting.com while Cassie is on PTO?

Comment 22

[Matt Thompson (@OpenMatt :OpenMatt)]

* Matthew: I believe this is ongoing / done. Can you update this ticket?

Comment 23

[Matthew Willse [:mattheww]]

Attachment: Summary results from user test: https://webmaker.etherpad.mozilla.org/login-usertesting-1

Comment 24

[Matthew Willse [:mattheww]]

Testing complete. Any further tests will be added to a new bug.