Closed Bug 672674 Opened 13 years ago Closed 13 years ago

[BrandAwarenessTracking] Copy for survey questions

Categories

(Marketing :: Copy, task)

Product:
Marketing
Component:
Copy
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: lforrest, Assigned: jslater)

References

Details

As part of our goal to effectively communicate our brand positioning to the world, we'd like to establish a way to measure brand awareness on mozilla.com on a ongoing basis. 

We specifically are interested to know what % of new and existing users know Firefox is made by a non-profit.

We will run some form of a survey to both new and existing users. Please create the copy for this survey. We're starting by asking just this one question.

Note: we don't need to explicitly ask if they know we're a "non-profit" - it can be more broad than tax-structure, for example, if they know that Firefox is created by a organization with "a greater mission" or that has "non-commercial interests" in mind. 

John - I know you have a good sense of this. Can you provide a few different versions of the copy so we can pick the best suited for this type of survey? 

Thanks!

Wiki: 
https://wiki.mozilla.org/Engagement/Analytics/Brand-Awareness
Timing: would be nice to have this by Monday next week.
Assignee: Mnovak → jslater
Blocks: 660996
Rebecca, do you have any best practices I should know about before working on this? For example, should we make it a yes/no question or some other format? Any word choices we should avoid or try to use?

Thanks-
Hi John,

I mentioned this to Laura before; I'm not entirely sure the best way to get around bias regarding the use of yes/no.  If we go with yes/no, particularly if we are asking questions that have a social norm towards positive sentiments (e.g., "non-profit" = good) or being knowledgeable (knowing something as opposed to not knowing), we're going to run the risk of people saying yes without meaning it (acquiescence bias).  This is something to think about...I don't have an easy solution to this problem.

The "mission-driven" question is something we tried with an NPS scale on a "what's new" page; that *seems* to get the point across, but we could always try variations and see if there are differences in the proportions of respondents (e.g., ask about "mission-driven/greater mission" in one iteration, "non-commercial interests" in another, see if there are different numbers in similar samples).  In which case, I defer to your judgment about what sort of language expresses the non-profit issue.

The other thing to point out is that we did a yes/no on that same NPS survey, looking to see how the proportions of NPS differed between the yes and no respondents to knowledge of the mission...I don't know if you've seen this question or not.  That is on the metrics blog if you wanted to take a look at it.
This may be a stupid question, but what's NPS?
NPS = Net Promoter Score.  So, it's a marketing metric (which is somewhat controversial, and I won't go into that here) that asks your customer to rate how willing they are to recommend your product or service to a friend or colleague on a scale of 0-10.  

Then, you take the proportion of people who report 9-10 ("promoters"), subtract the proportion of 0-6s ("detractors"), ignore the 7-8s ("neutrals") and have a single value that summarizes your customer base.  Companies will get really into that number...

Anyway, the general idea = the higher your NPS, the better.  It's really not that clear what the value ACTUALLY represents with respect to your customers, but there I go, diving right into the controversy...
Ah yes, I know net promoter, just didn't recognize the acronym in this context.

Anyway, recognizing the inherent flaws of the system we're working within, here are a few suggested starting points...am open to any and all feedback.

* Did you know Firefox is produced by a non-profit organization? (y/n)
* Did you know Firefox is produced by a community-based non-profit? (y/n)
* Did you know Firefox is made with a mission to make the web better? (y/n)
* Firefox is made by a non-profit organization. Does knowing that make you more likely to use it? (y/n)
* Firefox is made by a non-profit to make the web better. Does that make you feel more or less favorable about it? (more/less/no change)

I could come up with more variants, but the main directions seem to be the simple non-profit awareness, or assuming (or stating) the non-profit bit and asking how people feel about it. 

What do you guys think? Just thinking through the options, is it possible to run both directions at the same time (with 1/2 the viewers seeing one question and 1/2 seeing the other)?
Hey, I think these are looking good. I especially like the approach about being really explicit as to asking people if their knowledge of Firefox being made by a non-profit would change their opinion or usage.

I'll wait for feedback on comment 6, but let me know if I can help here now that I'm back.
I've been thinking about the "yes/no" and inducing bias issue, and I think that if we do make this survey a long-running metric (and if we aren't totally dedicated to the notion that the proportions we get are truly representative of the real world), it might not matter.  For example, if we get 50% that say they do know, that might not *actually* mean that 50% of users know, but it might not matter if we see that the proportion always stays at 50% or higher.  Thoughts?

This is an excellent start!  Comments in-line:

* Did you know Firefox is produced by a non-profit organization? (y/n)
* Did you know Firefox is produced by a community-based non-profit? (y/n)

I think the first one is a little better, since the fewer "good feeling" ideas you appeal to in a question, the more precise your measure.  Meaning, I bet the second question would lead to a higher proportion of "yes" than the first question.

* Did you know Firefox is made with a mission to make the web better? (y/n)

Question: are we interested specifically in the *non-profit* aspect of Mozilla, or just the general notion of Mozilla as a positive force?  

* Firefox is made by a non-profit organization. Does knowing that make you more likely to use it? (y/n)

I like this question because it's not getting at people's knowledge of Mozilla as a non-profit, but I'm not sure that yes/no is the way to go here.  First, a lot of people will want a "don't know."  Second, we should probably do this as "...more or less likely to use it?" more likely/less likely/not sure.  Third, I am not sure that anyone would say they are less likely to use a product made by a non-profit, unless it went against their economic ideology ;)

* Firefox is made by a non-profit to make the web better. Does that make you feel more or less favorable about it? (more/less/no change)

I like this one the best...but I have the same problem as with the previous.  I find it hard to believe that anyone will say that this piece of information will make them feel less favorable towards Firefox. But, unlike the other question, we can say "no change" and I think that addresses that issue nicely.  

I'll have to go over these a couple more times in my head, but there's my first pass.
(In reply to comment #8)

Thanks for your input here Rebecca, and feel free to add more thoughts as they come! 

In general the immediate priority of this survey is to gauge non-profit/greater-good awareness and track it over time, instead of answering the question of *if* this knowledge correlates to browser use. We sort of know that already, and I agree with Rebecca's thoughts above on how people probably would not report a "less favorable" reaction in a survey like this. 

> I've been thinking about the "yes/no" and inducing bias issue, and I think
> that if we do make this survey a long-running metric (and if we aren't
> totally dedicated to the notion that the proportions we get are truly
> representative of the real world), it might not matter.  For example, if we
> get 50% that say they do know, that might not *actually* mean that 50% of
> users know, but it might not matter if we see that the proportion always
> stays at 50% or higher.  Thoughts?

I agree here. The main value of this metric is to give us some form of a baseline so we can see when we fluctuate.  
 
> This is an excellent start!  Comments in-line:
> 
> * Did you know Firefox is produced by a non-profit organization? (y/n)
> * Did you know Firefox is produced by a community-based non-profit? (y/n)
> 
> I think the first one is a little better, since the fewer "good feeling"
> ideas you appeal to in a question, the more precise your measure.  Meaning,
> I bet the second question would lead to a higher proportion of "yes" than
> the first question.

Agree. This is my second favorite. 
 
> * Did you know Firefox is made with a mission to make the web better? (y/n)
> 
> Question: are we interested specifically in the *non-profit* aspect of
> Mozilla, or just the general notion of Mozilla as a positive force?  

Good question - we're interested in the general notion of Moz as a positive force, not just "non-profit" status, per se. 

This option gets my top vote.  
 
> * Firefox is made by a non-profit organization. Does knowing that make you
> more likely to use it? (y/n)

As stated above, this metric is about awareness, not awareness's affect on browser usage so let's go with one of the versions from above. 
 
> I like this question because it's not getting at people's knowledge of
> Mozilla as a non-profit, but I'm not sure that yes/no is the way to go here.
> First, a lot of people will want a "don't know."  Second, we should probably
> do this as "...more or less likely to use it?" more likely/less likely/not
> sure.  Third, I am not sure that anyone would say they are less likely to
> use a product made by a non-profit, unless it went against their economic
> ideology ;)
> 
> * Firefox is made by a non-profit to make the web better. Does that make you
> feel more or less favorable about it? (more/less/no change)
> 
> I like this one the best...but I have the same problem as with the previous.
> I find it hard to believe that anyone will say that this piece of
> information will make them feel less favorable towards Firefox. But, unlike
> the other question, we can say "no change" and I think that addresses that
> issue nicely.  
> 
> I'll have to go over these a couple more times in my head, but there's my
> first pass.

Let us know if you have additional thoughts and if not, I'll float the two favorites by Chris Beard to see if he has a preference then we can go from there!
Thanks all. I have some more thoughts on this but no time to add them at the moment (will do so later today).

Also cc'ing Jane, as she's working on similar info-gathering through her channels (email, twitter, etc) and it'd be good to make sure we're using consistent language.
Would love to get Chris' input on this, but he's going to be OOTO for the rest of the week and don't want to let that block us moving forward.

So, let's proceed with this for now:

* Did you know Firefox is made with a mission to make the web better? (y/n)

I'm still not sure the language is 100% right though. Matej/Jane...any additional thoughts?
How about one of these variations?


Did you know that Firefox is made to make the Web a better place?

Did you know that there's a mission behind Firefox to make the Web better?

Did you know that Firefox is built with a mission for a better Web?

Did you know that Firefox is on a mission to make the Web better?
(In reply to comment #12)
> How about one of these variations?
> 
> 
> Did you know that Firefox is made to make the Web a better place?
> 
> Did you know that there's a mission behind Firefox to make the Web better?
> 
> Did you know that Firefox is built with a mission for a better Web?
> 
> Did you know that Firefox is on a mission to make the Web better?

I prefer the original from batch #1: "Did you know Firefox is made with a mission to make the web better? (y/n)" since it's the most straight-forward. 

Jane/Rebecca - any additional thoughts or preferences? Would be nice to get this nailed down soon. Let me know and I'll help get the CBeard sign-off.
I also prefer the original.
I dont want to rat-hole us here - but when I read "Did you know Firefox is made with a mission to make the web better? (y/n)"  Mission doesn't necessarily mean non-profit to me, and doesn't get to why making the web better is needed and for who? (i.e. It could be Google saying this about Chrome). I therefore dont think the answer is helpful in measuring if folks understand who we are and what we're doing for the Web and ultimately for them. 

Realize this should not be a composite question, and this ends up being too wordy perhaps ....


Did you know Firefox is made by a non-profit organization with a mission to make the web better? (y/n)" 

Did you know Firefox is made by a non-profit organization with a mission to make the web better for you? (y/n)" 

Did you know Firefox is made by a non-profit organization with a mission to put you in control of your online life? (y/n)"
1) So it seems we really need to nail this down: are we primarily interested in knowledge of Mozilla as a non-profit or as a positive force?  Meaning, what exactly do we want with a metric that shows us proportion of users that have knowledge of Mozilla as a non-profit?  Are we just interested in gauging the degree of sentiment towards Mozilla, or are we specifically interested in the non-profit label (e.g., we're interested in developing marketing campaigns with a non-profit theme).

Additionally, I have a suspicion that if you ask about non-profit status, you will artificially increase the proportion of "yes" responses since people generally don't like to reveal that they don't know something, particularly if we are the ones asking them about our non-profit status.  We *could* test this language with an A/B sample on the What's New page...

2) The problem with the composite question is that it's not clear to what the respondent is saying yes: that Firefox is made by a non-profit, or that the mission is to make the web better.  So, if we are really interested in the non-profit measure specifically, we would have split these into two questions (e.g., 1 asking about non-profit, 2 asking about the mission of the non-profit).  This is not infeasible, particularly since it's a yes/no question; I doubt it would drop the response rate too much.  It's just not as light-weight as a single question.

3) I had a similar feeling with respect to "making the web better."  I think adding "for you" (option #2 is a good idea.
Hey didn't want to rat-hole as said - or mess with the methodology you are creating. Understand what you're saying "non-profit" vs. "positive force" - and possible results. I think John you already agreed on that above - so I'll defer to this. (Guess I had also seen this as an educating opportunity to those reading / answering the question).

"Did you know Firefox is made with a mission to make the web better for you? (y/n)"
More feedback is welcome!  It's a good idea to really nitpick this language stuff until we're sick to death of it, because we can't easily change it later...
This is a really good discussion - am glad we're having it. Fwiw, Jane was able to articulate some things that had been bothering me with the proposed language but I wasn't able to articulate them last night.

For me, it's a bit of both - simply asking whether or not people know we're a non-profit doesn't entirely get at the point, b/c it's also important for people to understand *why* we're a non-profit (making the web better for you). But on the flip side, like Jane said, simply saying "making the web better" isn't really unique to us...it feels too vague and squishy to be all that useful.

That's why I've been struggling with this. Rebecca, what do you think? Maybe we should look at this as a 2 question thing? How would that work, exactly? Would it produce relevant info, and could we combine the two questions into a single metric (I really like the idea of having one simple metric we could track...not two). Beyond that, I actually liked the composite wording, but will defer to your expertise about how that affects the results.
I agree; it really is worth spending extra time on the phrasing.  

Again, the only problem I have with asking about non-profit status as a yes/no is the lack of credibility in the results.  Consider if you received an email from the Red Cross with the question "Did you know that the Red Cross is a non-profit?"  It would be embarrassing to say no to that question, particularly since it's being asked by the Red Cross.  People will say yes to that question even if they didn't know, and the rate at which people will say yes will vary wildly.

One potential way to deal with this issue would be to ask a multiple choice question, e.g. "What kind of organization is behind Firefox?  A) A for-profit corporation B) A non-profit C) A university D) A government E) I don't know" (I have put very little thought into the question formation and the choices, so bear with me).  It's heavier than a yes/no, which makes it unappealing, but the results would be more reliable.

With respect to the composite question, the standard practice would be to break that into two questions.  Again, consider the scenario where you asked the composite question "Did you know Firefox is made by a non-profit organization with a mission to make the web better for you? (y/n)" Let's say we received proportions of 60% yes.  These include respondents who say yes because they knew that Firefox was made by a non-profit, but didn't know the mission; respondents who knew the mission, but didn't know that it was a non-profit behind the mission; respondents who knew that it was a non-profit behind the mission AND that the mission is to make the web a better place, respondents who said yes even though they didn't know that the mission was to make a better place OR that it was a non-profit behind Firefox...and so forth.  It would be a complicated metric to interpret.

So, let me wrap up this comment with one question that will nail down whether we need one or two questions: what do we want this metric to reflect *specifically*?  The level of knowledge of non-profit status or the proportion of people who know about the mission?  I do have an opinion on this, but we should leave it open for discussion first.

An aside: if we are really interested in both, but want a single metric, we could ask both questions and then subset on those who got the "non-profit" question correctly.  There still might be that acquiescence bias problem with respect to asking a second question "Did you know that our mission is to make the web better for you?" but we should answer the question I posed above first...
I've spoken to most of you offline and we've decided to narrow these down to a few final approaches to then circulate more internally. 

Based on the valid possible approaches above I think the final options should be:

1. Greater Good Focused: Don't mention "non-profit" but allude to working for a greater good with:

"Did you know Firefox is made with a mission to make the web better for you? (y/n)"

2. Explicitly state we are a Non-profit and ask:

"Did you know Firefox is made by a non-profit organization with a mission to make the web better for you? (y/n)" 

3. Provide a multiple choice option-like quiz that will hopefully eliminate some of the reporting bias:

"What kind of organization is behind Firefox?  A) A for-profit corporation B) A non-profit C) A university D) A government E) I don't know" [the options here may change a bit]

How does that sound? I don't think the perfect question exists so let's establish options we're happy enough with and go from there.
I'm leaning toward #2. Jane, what do you think?

What's our deadline here? Would love to get cbeard's input as well.
Talked to Chris...he's ok with #2 as well:

"Did you know Firefox is made by a non-profit organization with a mission to make the web better for you? (y/n)"
Are we okay with the possibility that it's not precisely clear what yes refers to?

Meaning, if a user says "yes," is that in response to "being made by a non-profit organization"  or that "the non-profit's mission is to make the web better for you?"  Those are two logical statements to evaluate, and it's not clear to me if we're intending to use one statement to represent the respondent's evaluation of both of them.

For example, I can easily see respondents saying "yes" to the non-profit portion, regardless of whether they knew the mission or not.  "Yes, I knew that Firefox was made by a non-profit, but no, I didn't know that the specific mission was to make the web better for me." The alternative is true, "Yes, I knew that Firefox's mission was to make the web better for me, but no, I didn't know that it was made by a non-profit." 

This means that we should be careful about treating the proportion of "yes" respondents as the conjunction of both knowledge of non-profit status AND knowledge of the mission.
Duly noted. My take on it is that it's not the end of the world if the raw metric isn't exactly perfect. What's more important is being able to track the relative rise and fall of that metric over time.

So, given that I'm thinking it's probably ok to go with #2 given that it most closely matches our overall brand message.
Guys -- I'm sorry for the late response. I am really OK with #2. thanks for this.
Okay - great convo guys, thanks. Let's continue with #2 while keeping the limitations of this sort of question in mind.

Closing this bug.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.