896460 - Prioritize scenarios for user testing of Webmaker.org

Status: RESOLVED FIXED

(Webmaker Graveyard :: webmaker.org, defect)

Description

[Cassie McDaniel [:cassiemc]]

Deliverable: a list of prioritized kits with questions we want answered that can be written into scenarios.

Based on documentation here: https://etherpad.mozilla.org/usertestingwmo

Comment 1

[Brett Gaylor [:brett]]

Attachment: Popcorn Maker structured tasks

Comment 2

[Brett Gaylor [:brett]]

Attachment: Webmaker.org first impressions

Comment 3

[Brett Gaylor [:brett]]

Attachment: Webmaker backgrounder

Comment 4

[Brett Gaylor [:brett]]

Added some works in progress

The popcorn maker structured tasks takes a different format - it is obviously quiet structured!

The difference is that with popcorn maker, we're actually looking for QA - whereas with webmaker.org, we're interested in impressions - we may want to create a popcorn maker / thimble one that is more about a survey.

Comment 5

[Cassie McDaniel [:cassiemc]]

Comment on attachment 780089 [details]
Webmaker backgrounder

Karen and I huddled heads this morning and came up with this feedback:

** This survey can be done on its own, but if other surveys are also completed, there’s no way to tie this data with their other completed surveys. This one will still have value, but perhaps less value, as we cannot provide a well-rounded picture of the user throughout the other surveys.

So, we can EITHER always include these questions at the top of other surveys as a way of getting this bkg info on all users. OR, we take them as separate pieces of data.

** Overall we like this survey. Think we'll get a good snapshot of makers.

Would Add:
* Age
* Gender
* Where in the world do you live
* What kind of computer do you use? Select all that apply. - (place near the bottom, by the internet question, so they aren't thinking digital making from the beginning)
Desktop - Apple
Desktop - Windows
Mobile - Apple
Mobile - Android
Mobile - other


This, and additional feedback, is noted here: https://docs.google.com/a/mozillafoundation.org/document/d/1Fx-u_WzifayzEGi97ku3aDPDFKOlh38EyU9M2GAMRUQ/edit

Comment 6

[Cassie McDaniel [:cassiemc]]

Comment on attachment 780088 [details]
Webmaker.org first impressions

Karen and I also huddled heads on this one and came up with a different, more scenario based structure. This is detailed here: https://docs.google.com/a/mozillafoundation.org/document/d/1Fx-u_WzifayzEGi97ku3aDPDFKOlh38EyU9M2GAMRUQ/edit

Comment 8

[Brett Gaylor [:brett]]

Attachment: User survey revision 2

Combined background, webmaker.org and post surveys into one

Comment 9

[Cassie McDaniel [:cassiemc]]

Comment on attachment 780677 [details]
User survey revision 2

Like it a lot! Didn't feel too long. Much more specific. 

If anything, may add a couple more impressions to the parting survey:

* How would you rate the design of the site
* How would you rate the functionality of the site
* How likely would you be to return to Webmaker.org
* What was your favourite feature
* What was your least favourite feature

(not necessarily in this order)

Comment 10

[Brett Gaylor [:brett]]

OK - updated.  Same document.

Comment 11

[Cassie McDaniel [:cassiemc]]

Looks good. Was Laura able to test this out yesterday?

Karen and I are going to have a look at the Popcorn survey next week, and hopefully incorporate any feedback from the testing session.

Comment 12

[Brett Gaylor [:brett]]

Yeah seems like they got 12 responses!

As spreadsheet:
https://docs.google.com/a/mozillafoundation.org/spreadsheet/ccc?key=0AkvvpcilLIOndE4xSG1oaEM2OEVDRHB5LXhHSjc2UHc#gid=0

W/ analytics:
https://docs.google.com/a/mozillafoundation.org/forms/d/1zmNsO4ifJjpGb4tH_NGLTjUzuR20s25iK-2zqTDZN2s/viewanalytics

Comment 13

[Cassie McDaniel [:cassiemc]]

Comment on attachment 780087 [details]
Popcorn Maker structured tasks

Popcorn Maker (remove Quality Testing from title)

We know this is an old survey, but it does come across as very functional – we should be testing overall impression as well. Here are some ideas (take what you want from the list) for some more in-depth feedback potential from users.

* Test future functionality ideas
    TowTruck
    Mobile app idea
    Media manager
    Integration with other tools?
* Test Flow
    Getting in and out of popcorn
    Getting back to the site
    Visiting My Makes
    Integrating My Makes
    What would you want to do with My Makes (share? like? arrange?)
* Test layout
    Was the publish button where you expected it to be
    Specific interface details - like the quote button; the make details bar (do people click details?), 
    What about screen real estate – did most important tasks have the most weight?
    Tabbing?
    Did/would users like the 1. 2. 3. process? 
    Language / labels
* Functionality
    Timeline
    Play with Ins and Outs
    Duration
    Can people add events (do they know what events are)
* Work-arounds
    Are people using any? What do they consider a workaround
    how can this inform feature requests
* Test our assumptions
    Do people really understand remix? 
    Are they learning anything? Or is this another way of consuming content? 
    Do they see a big difference between consuming, editing, or remixing? 
    Do people link the idea of open Internet with “remixing” - e.g. they understand that anything they find on the site is remixable? Do they put this project in the same category as Wikipedia?
* Have them walk through steps of creating a real project (something they might actually be invested in, as opposed to pretend or overly generic scenarios). Give a pretend theme, eg. if you were going to make a make for a family’s birthday, what would you want to include?
* Ask about barriers to their making, eg. where do they keep their digital assets (or keep track of things they like) to use in Popcorn projects
* What do people want to do with their make once it’s complete
    Do they share it? Where do they share it
    would they post it on their profile page
    Do they want others to remix their work?
    Do they want people to like it
    

Important questions, not popcorn related:

* How will swag be sent out? Should do it sooner rather than later. We should ask their t-shirt size in the survey.
* Can we get some of the real webmaker badges? (Karen saw them at DML, Leah had them) Maker Party Swag kit may also be an option.

Something to add to general survey:

* How do users identify with Mozilla? Check all that apply: current or past Firefox user / do they attend Mozilla events / Have they used Webmaker.org before / Do they host Mozilla-related events / Do they volunteer their time for Mozilla / do they use Firefox OS on mobile / none of the above

All of this feedback is collated in this doc again: https://docs.google.com/a/mozillafoundation.org/document/d/1Fx-u_WzifayzEGi97ku3aDPDFKOlh38EyU9M2GAMRUQ/edit

Comment 14

[Thomas Park [:tompark]]

Lots of interesting research questions to explore! Is this user testing going to be done in person at workshops and/or online at /feedback? Some comments:

I like the scenarios-based questions, which make things concrete. I'm curious how it goes when people complete it on their own -- these scenarios would definitely translate well face-to-face.

How about asking for Webmaker accounts in the survey? Could be interesting to see how they use the tools, but has privacy implications.

The survey seems on the lengthy side. Responses could be biased toward people who are passionate about webmaker, depending on how the survey is administered. I'd look to trim the fat or focus it down more if possible. Using pagination for the survey might make it feel shorter too (side benefit: prevents data loss, e.g. accidentally hit reload 90% through).

What about asking for hobbies and interests in general, as a transition into "what kinds of things do you enjoy making?"

Comment 15

[Cassie McDaniel [:cassiemc]]

(In reply to Thomas Park [:tompark] from comment #14)
> Lots of interesting research questions to explore! Is this user testing
> going to be done in person at workshops and/or online at /feedback? Some
> comments:

We have the survey online, so hopefully it will be done ad hoc on an individual basis, as well as in person.

> I like the scenarios-based questions, which make things concrete. I'm
> curious how it goes when people complete it on their own -- these scenarios
> would definitely translate well face-to-face.
> 
> How about asking for Webmaker accounts in the survey? Could be interesting
> to see how they use the tools, but has privacy implications.

Yeah, privacy is a concern here.
 
> The survey seems on the lengthy side. Responses could be biased toward
> people who are passionate about webmaker, depending on how the survey is
> administered. I'd look to trim the fat or focus it down more if possible.
> Using pagination for the survey might make it feel shorter too (side
> benefit: prevents data loss, e.g. accidentally hit reload 90% through).

Agreed it is long. Laura – about how long did it take you to get through the survey? Not sure pagination is possible in google forms (their UI capabilities suck!).

> What about asking for hobbies and interests in general, as a transition into
> "what kinds of things do you enjoy making?"

I like this. Brett, can you add it?

Comment 16

[Laura Hilliger [:epilepticrabbit]]

When we got the 12 responses it was because I gave the URL to some of our Super Mentors. We didn't run through the survey together, so I don't know how long that would take.

Comment 17

[Kat Braybrooke [:codekat]]

Note: Long resolved.