Closed Bug 660996 Opened 13 years ago Closed 13 years ago

Implementing ongoing brand awareness survey

Categories

(www.mozilla.org :: General, defect)

Product:
www.mozilla.org
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lforrest, Assigned: sgarrity)

References

Details

Attachments

(4 files)

What's New Mockup
54.11 KB, image/png
Details
Download Confirmation Mockup
69.65 KB, image/png
Details
What's New page survey mockup A.
223.80 KB, image/png
Details
What's New page survey mockup B.
223.10 KB, image/png
Details 
      No description provided.
Assignee: nobody → lforrest
As part of our goal to effectively communicate our brand positioning to the world, we'd like to establish a way to measure brand awareness on mozilla.com on a ongoing basis. 

We are currently meeting to decipher the best way to do this. Implementation details will be supplied here.
Some updates here: We're currently investigating the best technical solution to implement a ongoing survey like this. The two options:

1. SiteSpect

2. iFrame through SurveyGizmo 

Rebecca - I know you've done some investigation. Can you provide a recommendation? 

Also, please include any suggestions regarding metering (meaning the percentage of total exposures per day)

Thanks!
These aren't exclusive options, actually.  The SurveyGizmo iFrame can be embedded nearly anywhere.  The option of SiteSpect allow us to sample a proportion of users that land on certain web properties (e.g., SUMO, AMO, What's New, Downloads).  More specifically, this sampling is done server-side rather than client-side.

However, Blake has informed me that it's not really a good idea to use the SiteSpect framework for something that needs to be stable long-term.  What we should do is use the SiteSpect with the SG iFrame as a prototype for some amount of time, nailing down our survey design and exposure rate.  Then when we have figured out the details, we should go to the developers of the Mozilla properties that we're interested in using and have them work on implementing the survey directly into the properties. 

Also, we probably don't want to rely too much on SurveyGizmo with respect to an ongoing survey, since our contract is up on October and I'm not sure if we're going to extend past that.  

Does that make sense?  Basically what I'm saying is, we can build the prototype with SurveyGizmo, take the iFrame for that survey and run it through SiteSpect.  We can do this for a little bit, maybe a month or two, just to get an idea of the relationship between exposures per day and response rate.  That way, when we go to the web dev guys, we can just say "build this survey into this property with these parameters for metering."
Thanks. Yes - I think this makes sense, although building survey functionality into the site will be a fairly large project. Still - thanks for making the next steps clear. Those are essentially:

"...build the prototype with SurveyGizmo, take the iFrame for that survey and run it through SiteSpect.  We can do this for a little bit, maybe a month or two, just to get an idea of the relationship between exposures per day and response rate."

Next Steps: Figure out exact design of this survey. Meaning: does it take over the whole page? Just the header? etc. I'll update the details here. 

Implementation Details: To capture the thoughts of both new and existing users, 2 different pages will be used: What's New, and the "Thanks for Downloading" page.
Hey Steven - Attached are two mockups of what we'd like this to look like. Since this will be served out to a very small % of people it's okay if it takes over the whole page. 

Can you guys help finesse this UI? 

Next Steps:
1. I'll update this bug with the SurveyMonkey iFrame code
2. Rebecca can help us with the sampling info
Attached image What's New Mockup — 

    
    

    
  

      
      
      
      

        Attached image
          
        Download Confirmation Mockup
        — 
      

    


  

    
    

    
  
Hey Laura. This won't be too intrusive (in terms of blocking other content) will it?

    
    

    
  
(In reply to John Slater from comment #9)
> Hey Laura. This won't be too intrusive (in terms of blocking other content)
> will it?

Moderately - check out the mockups and let me know your thoughts. Note that this will only be shown to a very small % of users over time.

    
    

    
  
SurveyGizmo iFrame Code:

There are two different survey versions since we want to separate the new vs. existing user data. 

1. What's New Version

<iframe src="http://www.surveygizmo.com/s3/iframe/616308/cc98acb71d45" frameborder="0" width="700" height="500" style="overflow:hidden"></iframe>

2. Download Complete Version

<iframe src="http://www.surveygizmo.com/s3/iframe/616314/73136edf8696" frameborder="0" width="700" height="500" style="overflow:hidden"></iframe>

    Tip #1: Change the pixel measurements for width and height to fit inside the width of your web page and to match the height of your survey.
    Tip #2: If you're having issues with the iframe not scrolling properly, remove style="overflow:hidden" from the iframe code before adding it to your website

Steven - I can also provide a Javascript or HTML embed if that will be easier to implement. Since this survey will run on a continuing basis let's be sure to make this fit into our site well. There are a number of other templates I can choose from to make it blend in more with the site, if needed. Let me know your thoughts.

    
    

    
  
(In reply to Laura Forrest from comment #10)
> Moderately - check out the mockups and let me know your thoughts. Note that
> this will only be shown to a very small % of users over time.

Hey Rebecca, how small of a % do you think we'll be showing this to?

    
  
Assignee: lforrest → steven

    
    

    
  
My hunch is that we'll have to adjust the % once we see the actual response rates.

For example, Blake and I have ran surveys off of What's New and the en-us Download page and the response rates are very different. A one to two question survey that's a yes/no will probably have the highest response rate, but it's really hard to say.

So, we've shown surveys to 3000-5000 people via SIteSpect and get anywhere from a 10%-25% response rate.  This one will probably be at the upper end of that or maybe even slightly higher, but I wouldn't expect more than that.

The second thing to think about is the time frame; how often do you want to collect this information?  I had imagined that it would most similar to a multi-day tracking poll, which means that you'll want to gather data over a few days and then aggregate over those days (eg, M/T/W or M-F, there are many options).  Weekends are typically bad.  

The idea I had was to take that amount that we're looking for (e.g., say we're looking for 800-1000 respondents), determine where in the 3000-6000 exposure range returns that n of respondents most reliably, and then divide those exposures over the span of days we decide on using.  

Does this make sense?

So it follows: how often do you want your data points (e.g. weekly/bimonthly/monthly/etc)?  It must be at the week level or higher in order to do multi-day tracking polls.  Weekly will have the most exposures obviously; the survey will essentially always be running in that scenario.

    
    

    
  
That makes sense, thanks! 

In my opinion a monthly number will be sufficient. That way we won't always be pestering people (weekly) and have enough of a number to start identifying and tracking trends over time. 

John, Jane - what do you think about that?

    
    

    
  
Hey Steven - Rebecca and I are trying to figure out sampling methodology. If we want to only serve this out to 5% of visitors how exactly would you go about doing that (technically speaking)? Server side through a network load balancer, or otherwise?

    
    

    
  
(In reply to Laura Forrest from comment #14)
> That makes sense, thanks! 
> 
> In my opinion a monthly number will be sufficient. That way we won't always
> be pestering people (weekly) and have enough of a number to start
> identifying and tracking trends over time. 
> 
> John, Jane - what do you think about that?

Personally, I would like to see us do this a little more frequently than once a month - bi-weekly would be my preference. It would be helpful to see if there's any impact in additional brand campaigns we do etc, and track changes.

    
    

    
  
I agree with Jane that monthly isn't enough. I like the tracking poll model, so I'd like to do it every two weeks at the least, and possibly every week if the sample size is small enough that we don't pester *too* many people.

    
    

    
  
I don't see any problem with weekly/biweekly/monthly, depending on the sampling method.

I would still like to know the details regarding the probability of a respondent being part of the sample.  This is a technical detail, which is addressed in Comment #15.  It would strengthen the representativeness claim of this tracking poll if we could determine this.  It is also important for any statistical analysis.

Again, some of these technical issues are only as important as the rigor we expect from this metric.

    
    

    
  
(In reply to Rebecca Weiss from comment #18)
> I don't see any problem with weekly/biweekly/monthly, depending on the
> sampling method.

Let's start off with bimonthly (every-other week) and go from there. 

> I would still like to know the details regarding the probability of a
> respondent being part of the sample.  This is a technical detail, which is
> addressed in Comment #15.  

Steven - can you help answer this question?

It would strengthen the representativeness claim
> of this tracking poll if we could determine this.  It is also important for
> any statistical analysis.
> 
> Again, some of these technical issues are only as important as the rigor we
> expect from this metric.

    
  
Target Milestone: --- → 3.10

    
    

    
  
Some further information: basically, I want to know the technical details of how a visit is selected to be included in the survey condition.

So, for example, do we have two servers (one hosting the "control" webpage and another hosting the "treatment" webpage, where treatment = administered a questionnaire)?  If so, by what means is one visit sent to the treatment server as opposed to the control server?  

I'm not even sure this is the way you're planning to sample visits, so if it might be easier to talk about these details in-person/on the phone.

    
    

    
  
(In reply to Rebecca Weiss from comment #20)
> Some further information: basically, I want to know the technical details of
> how a visit is selected to be included in the survey condition.
> 
> So, for example, do we have two servers (one hosting the "control" webpage
> and another hosting the "treatment" webpage, where treatment = administered
> a questionnaire)?  If so, by what means is one visit sent to the treatment
> server as opposed to the control server?  
> 
> I'm not even sure this is the way you're planning to sample visits, so if it
> might be easier to talk about these details in-person/on the phone.

Steven - Can you chime in here?

    
    

    
  
(In reply to Rebecca Weiss from comment #20)
> So, for example, do we have two servers (one hosting the "control" webpage
> and another hosting the "treatment" webpage, where treatment = administered
> a questionnaire)?  If so, by what means is one visit sent to the treatment
> server as opposed to the control server?  

We've done this a few different ways in the past. For A/B testing, Blake has used SiteSpect, which I think does create two distinct versions of the page. For most survey samples we've done before, though, we do it all client side.

There is one copy of the page that every visitor downloads. The page contains a bit of javascript that does a simple random selection weighted to the desired sample size (5%, etc.).This javascript then adds the survey-related HTML to the page.

Does that answer the question?

    
  
OS: Mac OS X → All
Hardware: x86 → All
Target Milestone: 3.10 → 3.11

    
  
Target Milestone: 3.11 → 3.12

    
    

    
  
(In reply to Steven Garrity from comment #22)
> (In reply to Rebecca Weiss from comment #20)
> > So, for example, do we have two servers (one hosting the "control" webpage
> > and another hosting the "treatment" webpage, where treatment = administered
> > a questionnaire)?  If so, by what means is one visit sent to the treatment
> > server as opposed to the control server?  
> 
> We've done this a few different ways in the past. For A/B testing, Blake has
> used SiteSpect, which I think does create two distinct versions of the page.
> For most survey samples we've done before, though, we do it all client side.
> 
> There is one copy of the page that every visitor downloads. The page
> contains a bit of javascript that does a simple random selection weighted to
> the desired sample size (5%, etc.).This javascript then adds the
> survey-related HTML to the page.
> 
> Does that answer the question?

It does, thanks. 

Is there enough to start implementing the UI? Would be great to have this close to launching this quarter. (Meaning UI finalized, worked through most sampling questions)

    
    

    
  


  

    
    

    
  


  
Here are two mockups showing possible layout/style for the survey on the What's New page. One uses traditional radio buttons and a submit button, the other just uses two submit buttons.

Laura, thoughts on the style and button options?

    
    

    
  
Thanks Steven! I prefer the button options over the radio buttons since it's one less click. 

Question: What do you think should happen after a user submits their answer? Ideally it should say "Thanks!" and then tell them about us, with a "learn more" link. Let's use: "Firefox is made by Mozilla, a non-profit devoted to making the web a better place. Learn more." (link: http://www.mozilla.org/about/mission.html) (Cc-ing Matej here for approval). 

It would be great if you could help design that part of this UX as well.

On a separate note we'll also need a way to make sure we don't show this to the same person twice. Perhaps some sort of cookie will do the trick.

    
    

    
  
Looks good. Just a small tweak:

Firefox is made by Mozilla, a non-profit dedicated to making the Web a better place. Learn more.

Also, a similarly small change to the question: should say "Did you know that..." and Web should be capitalized.

Thanks!

    
    

    
  
(In reply to Laura Forrest from comment #26)
> Thanks Steven! I prefer the button options over the radio buttons since it's
> one less click.

We do too. What about the style/layout. Do you prefer one of the background colors, or the question on the left or right?

> Question: What do you think should happen after a user submits their answer?

As you suggested, the page will reload and the question will be replaced with a "Thanks" message - we'll use the text you and Matej suggested.

Matej, what about that "Have a second? HELP US OUT" text - I think it's the right idea, but I made that up on my end. Can you give it a thumbs up or suggest tweaks.

    
  
Target Milestone: 3.12 → 4.0

    
    

    
  
(In reply to Steven Garrity from comment #28)
> 
> Matej, what about that "Have a second? HELP US OUT" text - I think it's the
> right idea, but I made that up on my end. Can you give it a thumbs up or
> suggest tweaks.

I'd maybe throw an exclamation point on HELP US OUT! Other than that, I think it's great.

    
    

    
  
(In reply to Matej Novak [:matej] from comment #29)
> I'd maybe throw an exclamation point on HELP US OUT! Other than that, I
> think it's great.

Cool. Exclamation points are usually only $1 each, but this one is pretty big and bold, so I'll add that to our next invoice.

I understand that this survey should also run on the "thanks for downloading" page - I think we can do a (slightly) modified version of the same layout/style for that page. A key difference there is that if we submit back to that same page (with the thanks-for-answering content), it would trigger the download again, so we'll probably have to submit to a dedicated "thank-for-answering" page.

Laura, would you like *both* whatsnew and download pages to submit to this dedicated thanks-for-answering page (just for consistency) or keep the whatsnew page submitting to itself and showing the thanks-for-answering content right in the whatsnew page?

    
    

    
  
(In reply to Steven Garrity from comment #30)
> (In reply to Matej Novak [:matej] from comment #29)
> > I'd maybe throw an exclamation point on HELP US OUT! Other than that, I
> > think it's great.
> 
> Cool. Exclamation points are usually only $1 each, but this one is pretty
> big and bold, so I'll add that to our next invoice.

You got it!
> 
> I understand that this survey should also run on the "thanks for
> downloading" page - I think we can do a (slightly) modified version of the
> same layout/style for that page. A key difference there is that if we submit
> back to that same page (with the thanks-for-answering content), it would
> trigger the download again, so we'll probably have to submit to a dedicated
> "thank-for-answering" page.

Yikes - I'd like to avoid linking them to a dedicated page. Can we just trigger some sort of "intra-page" state change that triggers the "thank you" message upon button click? 

In terms of fitting the survey onto this page it would be good if you used about the same size module and had it fit on the top of the page, pushing all the download details to the bottom. Would also be good if we retained the "Thanks for downloading" messaging in some way.  

> Laura, would you like *both* whatsnew and download pages to submit to this
> dedicated thanks-for-answering page (just for consistency) or keep the
> whatsnew page submitting to itself and showing the thanks-for-answering
> content right in the whatsnew page?

I think it just depends on the experience - where possible, I'd like to keep the user on the same page even if that means being inconsistent.

    
    

    
  
In regards to the different layouts: I prefer the lighter yellow version.

    
  
Target Milestone: 4.0 → 4.1

    
  
Whiteboard: r=96045 b=trunk
Target Milestone: 4.1 → 4.0

    
  
Target Milestone: 4.0 → 4.1

    
    

    
  
Laura, we've got this set and ready in the Firefox 8.0 whatsnew page, since the 7.0 whatsnew page isn't being used (right?).

As the 8.0 whatsnew page is redirected to the 8.0beta/whatsnew page while 8.0 is in beta. To work around this, we've added a ?no-redirect tag on the URL that bypasses this beta redirect (on stage only, so far)

http://www-dev.allizom.org/en-US/firefox/8.0/whatsnew/?no-redirect

As you'll see if you submit the survey, we do have the "Thanks" page setup too.

We can do the 1% sampling on this with Javascript. All visitors to the whatsnew page will get a version of the page with both the default promo feature (whatever that ends up being for the 8.0 release - we've got the current email promo in there for now) and the survey feature. The survey will be hidden for most people. For a random 1 in 100 visitors, though, the default feature will be hidden and the survey will be displayed.

That is unless there's a server-side method of pulling out the 1% you'd like to use.

We'll use a cookie to make sure the same person doesn't get the survey twice.

We'll also port the survey over into the download-thanks page too.
Whiteboard: r=96045 b=trunk → r=96045,96053 b=trunk

    
    

    
  
Also, Im'm aiming to have this ready for the 4.1 push (though the whatsnew page isn't being used right now, so that part won't be in use until Fx8.0 next month, but the download page can start right away).

    
    

    
  
A bit of a hitch on the download page version. We can't show you the "thanks" message in the download page, because we have to submit to surveygizmo, and send you back to a URL - but we don't know the URL at that point because it has operating system, language, etc. encoded in it from the button link.

We could use the iFrame embed method for this page, but we'd have to move the styles into surveygizmo (a minor hassle, but doable). However, we would have to drop the "Yes/No" buttons and go back to yes/no radio buttons with a submit button - at least on this page. Laura, thoughts?

Also, do we have a support contact at SurveyGizmo?

    
    

    
  
(In reply to Steven Garrity from comment #33)
> Laura, we've got this set and ready in the Firefox 8.0 whatsnew page, since
> the 7.0 whatsnew page isn't being used (right?).

Correct. It will be used again with 8.0 release. 
 
> As the 8.0 whatsnew page is redirected to the 8.0beta/whatsnew page while
> 8.0 is in beta. To work around this, we've added a ?no-redirect tag on the
> URL that bypasses this beta redirect (on stage only, so far)

Clever, clever!
 
> http://www-dev.allizom.org/en-US/firefox/8.0/whatsnew/?no-redirect
> 
> As you'll see if you submit the survey, we do have the "Thanks" page setup
> too.

Thanks for adding that in, I think it really helps conclude the experience in a nice way. And it looks good, so no changes there. 
 
> We can do the 1% sampling on this with Javascript. All visitors to the
> whatsnew page will get a version of the page with both the default promo
> feature (whatever that ends up being for the 8.0 release - we've got the
> current email promo in there for now) and the survey feature. The survey
> will be hidden for most people. For a random 1 in 100 visitors, though, the
> default feature will be hidden and the survey will be displayed.

Let's stick with 1% for now - I'd like to loop in Rebekah from metrics to talk figures a bit more, but any changes we can throw in before Fx8 and be fine.  
 
> That is unless there's a server-side method of pulling out the 1% you'd like
> to use.

This methodology sounds good to me. 
 
> We'll use a cookie to make sure the same person doesn't get the survey twice.
> 
> We'll also port the survey over into the download-thanks page too.

Great.

    
    

    
  
(In reply to Steven Garrity from comment #35)
> A bit of a hitch on the download page version. We can't show you the
> "thanks" message in the download page, because we have to submit to
> surveygizmo, and send you back to a URL - but we don't know the URL at that
> point because it has operating system, language, etc. encoded in it from the
> button link.
> 
> We could use the iFrame embed method for this page, but we'd have to move
> the styles into surveygizmo (a minor hassle, but doable). However, we would
> have to drop the "Yes/No" buttons and go back to yes/no radio buttons with a
> submit button - at least on this page. Laura, thoughts?
> 
> Also, do we have a support contact at SurveyGizmo?

Hmmm...in this case I think the best thing to do is not show a "thanks" state of this survey to these people. Switching to Radio buttons is a viable idea, but I think it's important that the two surveys match one another exactly, in order for the data to be as accurate/comparable as possible. 

Also, it's a bit different from the What's New page in that since we want to simply push the "download instructions" module down, this page won't exactly be fully a blank dead-end after they submit their answer.

    
    

    
  
A clarification to the above: my intent is to show a "thank you" state on the What's New version of this survey, but not on the "Thanks for downloading" version of this survey due to the complexities you explained in Comment 35.

    
    

    
  
Trunk is now updated with the iframe embed method on both the whatsnew and download pages. This method allows the thank-you message to be displayed in-page without reloading the page.

The unfortunate side-effect of this method is that we lost the client-side javascript access to the S.G. code that allowed us to re-write the radio buttons into Yes/No buttons - so we're back to radio buttons and a Submit button on both pages.

To see the survey on the download page, hit the download button on http://www-dev.allizom.org/en-US/firefox/new/

and the whatsnew page: 
http://www-dev.allizom.org/en-US/firefox/8.0/whatsnew/?no-redirect
Whiteboard: r=96045,96053 b=trunk → r=96045,96053,96265,96267 b=trunk

    
    

    
  
This is great - both look good. 

Ready to push!

    
    

    
  
(In reply to Laura Forrest from comment #40)
> This is great - both look good. 
> Ready to push!

Great - just to clarify, we still have to set it up to only show for 1% of vistors - so don't push anything yet. We'll get that setup tomorrow. Thanks.

    
    

    
  
Steven - Can you ping James once this is ready to be pushed? It's still in the 4.0 milestone and I want to make sure it doesn't get left behind. Merci!

    
    

    
  
This is all set in trunk. The survey is set to run on the 8.0 whatsnew page (see the no-redirect url in comment #39 and on the download thanks page for 1% of visitors. A cookie is set to ensure that the survey does show up twice for the same person.
Status: NEW → RESOLVED
Closed: 13 years ago
Keywords: qawanted
Resolution: --- → FIXED
Whiteboard: r=96045,96053,96265,96267 b=trunk → r=96045,96053,96265,96267,96372 b=trunk

    
  
Target Milestone: 4.0 → 4.2

    
    

    
  
Raymond - we can temporarily switch the download page survey to load 100% of the time (rather than 1%). Want us to do that?

    
    

    
  
(In reply to Steven Garrity from comment #45)
> Raymond - we can temporarily switch the download page survey to load 100% of
> the time (rather than 1%). Want us to do that?

yeah. Please do

    
    

    
  
raymond, the survey sample size is now 100% in trunk. Try the download button on:
http://www-dev.allizom.org/en-US/firefox/
to see the survey. Note that there is a cookie set that will keep you from seeing it twice.

Let me know when you're done testing so I can revert to the 1% sample size
Whiteboard: r=96045,96053,96265,96267,96372 b=trunk → r=96045,96053,96265,96267,96372,96525 b=trunk
Whiteboard: r=96045,96053,96265,96267,96372,96525 b=trunk → =
Whiteboard: =

    
    

    
  
pushed to production r96541

    
    

    
  
Switched the survey back down to 1% in r96546 in trunk. James, can you push that to production?

    
    

    
  
(In reply to Steven Garrity from comment #50)
> Switched the survey back down to 1% in r96546 in trunk. James, can you push
> that to production?

Good call, pushed in r96547

    
    

    
  
Steven - Question for you moving forward on this - will anything need to be updated after product releases, meaning, to have the survey show during Fx9 does anything need to change, or are we good to go?

Thanks! I can hardly wait for the data to start flowing in.

    
    

    
  
The survey code is baked into the 8.0 whatsnew page, so we'll have to make sure we carry it over when we create the 9.0 whatsnew page. We usually use the last version to create the latest, so we should be fine, but it might make sense to double check when the 9.0 release approaches.

Also, I'm not sure I let you know this, but the download page and whatsnew page are separate surveys in SurveyGizmo, titled:
Brand Awareness Tracking - Download Confirmation (iframe variation)
Brand Awareness Tracking - What's New 8.0 Page (iframe variation)

When 9.0 comes around, we can make a copy and keep the 9.0 whatsnew page results separate, or we can use the same survey for both 8.0 and 9.0 whatsnew pages.
Component: www.mozilla.org/firefox → www.mozilla.org
Component: www.mozilla.org → General
Product: Websites → www.mozilla.org

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: