Text on about:feedback uses leading language/misdirection to game positive reviews

NEW
Unassigned

Status

()

defect
P5
normal
7 years ago
3 months ago

People

(Reporter: Unfocused, Unassigned)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

Bug 774479 added an about:feedback page to solicit app reviews on Google Play. To me, the language/flow used is leading and dishonest.

The three initial options presented are:
* I love it
* I ran into some problems
* I have an idea

Only "I love it" leads to adding a review on the Google Play store. 

It seems this would be effective in gaining many additional positive reviews, but it feels dishonest. Reviews should be transparent by representing a balanced view of users based on the merits *and* demerits of the application, not from gaming only positive reviews.
There seems to be some prior discussion about this in the comments in bug 752396.
The thought behind this feature is that people when they like something they don't feel compelled to say so about it as much as they they do when they don't like it. They rush more often to voice frustrations than to give praise when something works. This feature helps balance this human characteristic and it does so for Firefox in a similar way that a lot of the other mobile apps do.
I wouldn't mind seeing text/links to review at the end of feedback and idea flows too.
I understand where you're coming from, Blair, because I had a similar initial reaction (though more "this feels like gaming" than "this feels dishonest.") I don't feel that way now, though (otherwise I wouldn't have supported the feature), and I'll try to explain why.

It's easy for us to see it as gaming, because we can see that the downstream effect of this will be a higher average rating in the store (at least, to the extent that this feature gets user attention). I think it's good that we hold ourselves to a high standard and police for lapses, but I think this is actually a better outcome for users, too.

Google's feedback mechanism isn't very rich, but it's the tool people have and so they use it. I don't believe most people who give 1-star reviews with text that describes a bug are want to be mean or to punish us; I think they want to get their issue addressed. We comb those comments as well as we can with the tools provided, and focus on fixes for major complaints, but it's a lossy process with no mechanism for follow up conversations. People will also write "reviews" that just request a new feature, but the tool requires them to assign a star rating to that request. What star rating do you give "have you considered adding a custom readability font?"

The one-size-fits-all limitations of market reviews would be a nuisance for users regardless, but because they also hurt the rating we present in our primary distribution channel, we doubly can't afford to ignore it. Lower star ratings for things like bug reports or RFEs hurt our ability to get people to try Firefox for Android, which hurts our ability to deliver on our mission in mobile. I don't think it's weaselly to look for ways to make that better. We should succeed on our merits, for sure, but I don't want us to take black eyes because of implementation choices google made in their market. They could have built a "send devs a bug report" button and they didn't, that's their call to make and I don't fault them for it, but I don't want to take the hit for them not doing it, either.

I saw a sign in a restaurant a while back that said "If you enjoy your meal, please tell others. If there's a problem, please tell us!" That feels closer to what's happening here. Obviously you can go tell the world whatever you like about Firefox but the page that we provide is very deliberately segmenting based on what kind of feedback you have so that we can maximize the impact it has on the product.

By contrast: when the new Android builds were getting close to beta readiness, someone asked if we should introduce it as a fresh app ID instead of as an update. That would wipe the slate clean of all the reviews that the old one had garnered, and we were confident the new one would be rated much higher. We didn't do it, because it would strand our existing users for the benefit of a ratings bump, and we didn't want to make that trade. Months after the release, we're still paying the mathematical price there (ratings for release are 40 basis points higher if you only look at new ratings vs. the total pool) but I'm confident that it was the right decision because it put our users' needs first.

I know this is wordy, but I'm not sure I've written it up anywhere before. Does what I'm saying make sense to you (even if you maybe disagree on where the tradeoff line should be drawn)?
(In reply to Johnathan Nightingale [:johnath] from comment #4)
> I know this is wordy, but I'm not sure I've written it up anywhere before.
> Does what I'm saying make sense to you (even if you maybe disagree on where
> the tradeoff line should be drawn)?

Yes, and I agree. I think there is an honest need for optimizing the flow for all those reasons. However, to me the current flow crosses the line by only leading to the Play store through that one choice, whose wording doesn't suggest what the end-action will be (which feels like a bait-and-switch). I know various other apps do this too - I get concerned when we start doing what I normally see as shady behaviour by other apps, even if that's the status quo.

Just having a link to the Play store elsewhere would help relieve my strongly uncomfortable feeling with this - something like what mfinkle suggested in comment 3, or an unobtrusive link at the bottom of the main page (something to see when you're looking for it, not the primary focus).
I understand our intentions as explained by johnath in comment 4, but I think it's noteworthy that reactions like comment 0 above have been *very* common among our own community.  I think the techy, early-adopter users that form our likely core audience on Android may feel much the same.  If it takes five long paragraphs to explain why this *isn't* gaming the system, then perhaps the distinction is too fine and we risk creating widespread negative reactions among our more influential users, our pure intent notwithstanding.

I like the suggestion in comment 5 that we should offer a clearer choice up front of "rate us on Google Play" versus "report a problem to Mozilla."  I also think we could tone down language that might feel too manipulative, like where we specifically suggest "a 5 star rating."
Posted patch patchSplinter Review
In addition to all the reasons above, I think it's more important than ever to fix this because of the changing environment.  There's a growing backlash against manipulative rating solicitations that is currently spreading from the iOS world to Android.  And of course, as Mozilla we should hold ourselves to a high standard in honesty and user empowerment.

This patch keeps the same basic flow, but changes the language to let users know up front what actions they are really choosing between.  The new options are:

  "Share the love: Help spread Firefox by rating it on Google Play"
  "Report a problem: Tell Mozilla about your problems or suggestions"

This might not please the most hard-core critics (who could still argue that this is "unfairly" diverting negative feedback away from the store) or the users like myself who would rather not see any rating solicitation at all.  But I think it's a positive step toward asking for ratings and feedback in as honest a way as possible.
Assignee: nobody → mbrubeck
Status: NEW → ASSIGNED
Attachment #8375879 - Flags: review?(mark.finkle)
This shows the current UI compared to the new UI after the patch is applied.

* The copy on the initial page is changed.
* The intermediate page between "I love it" and the Play Store is gone.
* The "report a problem" page is unchanged.
Attachment #8375897 - Flags: feedback?(ibarlow)
Attachment #8375879 - Attachment description: about-feedback → patch
Hmm... reviewing my own work, some changes I'd like to make:
* Add a "No thanks" link to the first page (like in the 2nd page of the old UI).
* Change "It only takes a minute" to something else.  It sounds redundant with "Have a minute?"
(In reply to Matt Brubeck (:mbrubeck) from comment #7)
> Created attachment 8375879 [details] [diff] [review]
> patch
> 
> In addition to all the reasons above, I think it's more important than ever
> to fix this because of the changing environment.  There's a growing backlash
> against manipulative rating solicitations that is currently spreading from
> the iOS world to Android.  And of course, as Mozilla we should hold
> ourselves to a high standard in honesty and user empowerment.
> 
> This patch keeps the same basic flow, but changes the language to let users
> know up front what actions they are really choosing between.  The new
> options are:
> 
>   "Share the love: Help spread Firefox by rating it on Google Play"
>   "Report a problem: Tell Mozilla about your problems or suggestions"
> 
> This might not please the most hard-core critics (who could still argue that
> this is "unfairly" diverting negative feedback away from the store) or the
> users like myself who would rather not see any rating solicitation at all. 
> But I think it's a positive step toward asking for ratings and feedback in
> as honest a way as possible.

Matt, I'm curious where you are seeing these concerns being raised. Outside of this bug, I don't think I have seen a single user make any such negative comment about our feedback request interface. 

And that's in the year-and-a-half this has been live.

I am extremely hesitant to make any change like this that has the potential to diminish our Google Play rating at the moment -- in particular, removing the specific ask for a 5 star rating. It's a sad but true reality that those little stars matter, and are often the deciding factor on whether users will install us or not.  And frankly, until we as Mozilla start making a bigger investment in selling Firefox for Android to the world (and we're working on getting that support), I am loathe to make any other changes like this before such plans are in place. Especially since we keep a close eye on feedback that comes in from our users, and the topic of us asking for a 5 star rating simply doesn't come up.

We have a great product with a weak marketing story around it. When we have a great product with a great story, I'd be open to revisiting this conversation.
Attachment #8375897 - Flags: feedback?(ibarlow) → feedback-
(In reply to Ian Barlow (:ibarlow) from comment #10)
> Matt, I'm curious where you are seeing these concerns being raised.

I found at least one Play Store review from the past three days that gave the rating solicitation screen as a reason for a low rating.  I am certain you will find more if you have a way to search our reviews over a longer period.

Five minutes of searching on input.mo.o finds a bunch of items like these:
https://input.mozilla.org/en-US/dashboard/response/4136871
https://input.mozilla.org/en-US/dashboard/response/4139652
https://input.mozilla.org/en-US/dashboard/response/4142092
https://input.mozilla.org/en-US/dashboard/response/4146003
https://input.mozilla.org/en-US/dashboard/response/4153405
https://input.mozilla.org/en-US/dashboard/response/4155408
https://input.mozilla.org/en-US/dashboard/response/4167189
https://input.mozilla.org/en-US/dashboard/response/4185205
https://input.mozilla.org/en-US/dashboard/response/4186398
https://input.mozilla.org/en-US/dashboard/response/4187418
https://input.mozilla.org/en-US/dashboard/response/4188421
https://input.mozilla.org/en-US/dashboard/response/4188526
https://input.mozilla.org/en-US/dashboard/response/4193714
https://input.mozilla.org/en-US/dashboard/response/4196306
https://input.mozilla.org/en-US/dashboard/response/4197280
https://input.mozilla.org/en-US/dashboard/response/4202018

And here are just a few of the links from the "growing backlash" that I mentioned:
http://www.polygon.com/2014/2/6/5386406/dungeon-keeper-stacks-deck-in-eas-favor-when-it-comes-to-android
http://effyr.tumblr.com/
http://www.marco.org/2013/12/14/rate-this-app

As I said, this started in the iOS world and I've just seen it spreading to more Android reviews and sites in the past week or so, but I expect it to have even more impact on Android apps if it continues.

Could we at least redesign the page to still incorporate the "5-star" call to action while also informing the user about the difference between the two choices (before they make the choice)?
Most of the feedback I'm reading in your Input links is more about "I already rated this, stop showing me this message", or "stop asking me to rate this I don't want to". So, less about people taking issue with being asked, but rather about being asked over and over and over. This actually sounds like a bug, and not the way the feature was meant to work originally.

We should definitely fix that, not by redesigning this UI but rather by redesigning its behaviour: we should be ensuring that Firefox only shows this message to a user *once*.

The only way they should see this more than once is if they click a "maybe later" link, in which case we reset the clock and ask them again later.
As for your "growing backlash" articles, it strikes me that nothing we do short of completely removing this feature would answer what these authors are calling out for. Your proposed redesign doesn't actually hit the root of what they're after, which seems to be for apps to remove any form of "rate me" prompt altogether. 

And as I mentioned above in comment 10, until we have a stronger and more diverse set of marketing initiatives in place, I just don't think we can do that.
(In reply to Ian Barlow (:ibarlow) from comment #12)
> The only way they should see this more than once is if they click a "maybe
> later" link, in which case we reset the clock and ask them again later.

That's probably what's happening in all of these reports. Maybe they don't recognize that the prompt is a normal tab, and they think their options are limited the three choices listed there.
Maybe. We should still investigate though -- from the feedback it also looks like we're still showing users these screens after they go off to the play store and give a review, which is not good.
Depends on: 972844
(In reply to Ian Barlow (:ibarlow) from comment #12)
> Most of the feedback I'm reading in your Input links is more about "I
> already rated this, stop showing me this message", or "stop asking me to
> rate this I don't want to". So, less about people taking issue with being
> asked, but rather about being asked over and over and over.

Even excluding these, I still see at least a dozen other complaints per month about the feedback page.  I do recommend searching through our Play Store reviews too.

(In reply to Ian Barlow (:ibarlow) from comment #15)
> Maybe. We should still investigate though -- from the feedback it also looks
> like we're still showing users these screens after they go off to the play
> store and give a review, which is not good.

Perhaps the users "already rated" Firefox did it on their own, rather than through about:feedback -- so when about:feedback page shows up they try to dismiss it with "Maybe later," as theorized above.

I've filed bug 972844 to address the "Maybe later" problem.

(In reply to Ian Barlow (:ibarlow) from comment #13)
> As for your "growing backlash" articles, it strikes me that nothing we do
> short of completely removing this feature would answer what these authors
> are calling out for.

...except for the first link, which was specifically about pre-screening potential reviews.  (I've seen several other critiques along the same lines, though it'd take me some time to dig them back up.)  There's even discussion on our own community Yammer calling this a "dark pattern."

Again, I'm *not* calling for removing the feedback prompt.  If app store mechanics make five-star reviews a really important way for users to help us succeed, I'll accept that.  But long-term, our internal culture of doing right and our external trust and goodwill are more valuable than any stars.  I want to find a way to ask for users' help in a way that doesn't tarnish that in the eyes of some users and contributors.  I have to believe we can come up with a good design that meets both needs.
(In reply to Ian Barlow (:ibarlow) from comment #15)
> Maybe. We should still investigate though -- from the feedback it also looks
> like we're still showing users these screens after they go off to the play
> store and give a review, which is not good.

I asked in IRC wether there is a persistent profile preference remembering that feedback was submitted that might be accidentally be reset for some reason, perhaps during browser update? I missed any replies.
Margaret?
(In reply to Aaron Train [:aaronmt] from comment #17)
> I asked in IRC wether there is a persistent profile preference remembering
> that feedback was submitted that might be accidentally be reset for some
> reason, perhaps during browser update? I missed any replies.

We store a preference that counts the number of launches, and we show about:feedback only when this counter is exactly 15.  We reset the counter only if the user taps the "Maybe later" link on about:feedback.
Comment on attachment 8375879 [details] [diff] [review]
patch

Let's work with Ian to get something he is happy with too. I also want to see if bug 972844 has an impact.
Attachment #8375879 - Flags: review?(mark.finkle)
Blocks: 1211436
Assignee: mbrubeck → nobody
Status: ASSIGNED → NEW
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.
Priority: -- → P5
You need to log in before you can comment on or make changes to this bug.