1116503 - Yellow background on the startpage with donation solicitation: some worry it's malware, some cannot read text on it, some hate it.

Status: NEW

(Mozilla Foundation Communications :: Website, task)

Description

[ragnarok_zero]

Attachment: Unbenannt.png

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0
Build ID: 20141125180439

Steps to reproduce:

Turned the browser on


Actual results:

Startpage background appears yellow. On my Monitor this really hurts too look at, but what's worse I'm afraid, this could be a hint for some virus or a damaged profile.


Expected results:

Startpage background appears grey.

Comment 1

[[:Aleksej]]

I've seen it with a linux64 build.  I think it's drawing the user's attention to the donation UI.

Comment 2

[:Gijs (he/him)]

See the input.m.o link below for context. Using the product in this way seems extremely counterproductive to me. People say "Firefox has gone all yellow", not "oh, it's just the one page". Was any *user* testing done, rather than just checking that it 'worked'? (bug 1112340 talks about testing, but all I see is implementation testing)

It also appears on every startup, with no way for people to turn it off, and no way to distinguish people who have donated already. That's a pretty terrible experience. We should not be giving our users experiences that rank as "pretty terrible", ever.

Can we never do this again, please, or at least have a discussion about doing it in the appropriate forums (mozilla.governance, the all-community yammer, ...) ?

Comment 3

[:Gijs (he/him)]

Adam, you blog about the a/b testing for these - can you ensure that the right people are looking at this user feedback? I get that more "in your face" backgrounds and so on "work" in that they get more people to donate, but I worry that that doesn't take into account the other people leaving Firefox entirely because of the aggressive advertising (also without taking into account whether they have or have not already donated anyway).

Comment 4

[Adam Lofting (:adamlofting)]

Hi Gijs,

Thanks for flagging me on this. I can't answer all the questions on the planning part of this, but I'll ping it around to others who worked on the EOY campaign (my work is primarily on testing the fundraising performance).

Andrea's written a pretty detailed blog post about the snippet including the background which you might have seen here: https://fundraising.mozilla.org/saving-the-best-for-last/

The versions of the snippet with color backgrounds were saved for the last 48hrs of the campaign, though it was tested to a small % of users ahead of that time to see how it performed (the yellow, and the dark grey background variations tested both increased click-through rates to the donation process significantly).

Not being able to 'distinguish people who have already donated' is a really interesting issue for Mozilla, because improving that user experience would require tracking a user through to the donation and bringing that data back into the product. Which is not something we currently do for many good reasons. At the same time though, we'd love to be able to give donors a better 'thank you' experience if we could.

Thanks again for flagging this up. If there is any more data / info on the people who had problems with this (ie number of complaints for example) I can take this back to the team as well.

Comment 5

[:Gijs (he/him)]

(In reply to Adam Lofting (:adamlofting) from comment #4)
> Not being able to 'distinguish people who have already donated' is a really
> interesting issue for Mozilla, because improving that user experience would
> require tracking a user through to the donation and bringing that data back
> into the product.

Besides the obvious "cookies end up in Firefox, and therefore 'in the product'" which doesn't really make sense to me - after all, the donation page itself is 'in the product' in a much more visceral way - why does this need to be in the product?

> Which is not something we currently do for many good
> reasons.

Are those reasons on record somewhere?

> At the same time though, we'd love to be able to give donors a
> better 'thank you' experience if we could.

Why can't the snippet set a cookie (hasdonated=1 or something) specific to snippets.m.o when a donation is made? The final "success" page for the donation could load a special page on snippets.m.o in an iframe in order to set the cookie, after which the cookie would show up for snippet requests (I think? We'd need to test it, obviously, but it seems to me like that should be the case). It'd go away like other cookies if the user clears their cookies, or when it expires (expiry-time tbd - maybe 1 month?). It'd also be de-trackable or whatever by ghostery, abp and other anti-ad/tracking things, because it would just work the same way the web generally works. :-)

> Thanks again for flagging this up. If there is any more data / info on the
> people who had problems with this (ie number of complaints for example) I
> can take this back to the team as well.

Per https://input.mozilla.org/en-US/?q=yellow&date_end=2014-12-31&date_start=2014-12-24, at least 21 people went through the input.mozilla.org flow and specifically called out the word 'yellow' in that last week of december. From a quick skim, it seems 20 out of the 21 are related to the donations.

Comment 6

[Adam Lofting (:adamlofting)]

I'm not technically knowledgeable enough about Firefox Development and the snippets service to dig in to these questions in detail, but I *think* some of the complexity is because the snippet is loaded into the product rather than displaying a URL directly on snippets.m.o. I'm CC'ing :mkelly who may know more.

These are good questions to ask though, and perhaps we can turn some of these into specific tickets for development to the snippets service?

> Per
> https://input.mozilla.org/en-US/?q=yellow&date_end=2014-12-
> 31&date_start=2014-12-24, at least 21 people went through the
> input.mozilla.org flow and specifically called out the word 'yellow' in that
> last week of december. From a quick skim, it seems 20 out of the 21 are
> related to the donations.

Thanks, that's very useful. (And also sorry, I now see this was already linked in the bug).

Comment 7

[Osmose [:osmose, :mkelly]]

(In reply to Adam Lofting (:adamlofting) from comment #6)
> I'm not technically knowledgeable enough about Firefox Development and the
> snippets service to dig in to these questions in detail, but I *think* some
> of the complexity is because the snippet is loaded into the product rather
> than displaying a URL directly on snippets.m.o. I'm CC'ing :mkelly who may
> know more.
> 
> These are good questions to ask though, and perhaps we can turn some of
> these into specific tickets for development to the snippets service?

There's docs on how snippets are inserted into about:home[1] but you've got the gist of it; Firefox downloads HTML from snippets and embeds it on about:home, meaning direct use of cookies won't work.

We could, in theory, have a page on snippets.m.o that is opened in an iframe and use postMessage to communicate with it to get at cookies stored on the snippets.m.o domain. Persona does something very similar, in fact. My biggest concern would be how to prevent any site from reading the cookies stored via that page, as it's unclear whether the targetOrigin protections around postMessage would work with about:home.

As to whether we should've gone this far in the first place, my personal opinion is that changing the background color is too much, but I was told that we had the authority to change the background color without checking with anyone on the Firefox team or doing any user testing first, so I didn't block on the issue. Andrea's the right person to ask about that.

[1] https://abouthome-snippets-service.readthedocs.org/en/latest/overview.html

Comment 8

[Andrea Wood [:andreawood]]

Hi Gijs - sorry this is a late reply. We are still analyzing data from the campaign and I was intending to gather some of it to share with you here. The yellow background snippet was reviewed and approved at the exec level (both moco and mofo leadership weighed in). It raised over $300,000 from more than 30,000 users in the 48 hrs it was live. As for the mechanics, we agree completely:

1. I would LOVE to turn off the fundraising snippet for donors (it's a substandard experience as-is, has been for 2 fundraising campaigns).
2. I would LOVE to give people the chance to turn off a snippet if they find it annoying (that goes for any snippet; and for fundraising, they're not going to become donors anyway).

We are always open to ways we can improve the user experience - it's why we do so much user testing and why we monitor the donate@ inbox and other channels so carefully. We definitely do not want to harm our relationship with users -- that's the last thing we want to do. We have always been (and still are) willing to leave money on the table to preserve the user experience. 

That said, Mozilla is a nonprofit and we fundraise to stay sustainable (see this post for more on that: https://blog.mozilla.org/blog/2014/12/19/building-a-healthy-web-to-hand-to-future-generations/). 

I called for a way to turn off the campaign snippet for donors over a year ago, but as a feature, it wasn't prioritized. Here's another related bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1109100 

If we can find a solution and implement a product update that makes the experience better on the snippet, I'm definitely on board. 

I do have a Q: Is 21 comments to "Input" a high number? I don't have a frame of reference. For instance, when a new version of Firefox is released, how many negative "Input" comments come in, generally?

Comment 9

[:Gijs (he/him)]

(split off the 'turn off fundraising for donors' bit to bug 1125994 as it's technically a different issue)

(In reply to Andrea Wood [:andreawood] from comment #8)
> Hi Gijs - sorry this is a late reply. We are still analyzing data from the
> campaign and I was intending to gather some of it to share with you here.
> The yellow background snippet was reviewed and approved at the exec level
> (both moco and mofo leadership weighed in).

This makes me feel better already, thanks for sharing that. I think my main concerns here are (a) the visceral reactions this has created (even with me just viewing the screenshot; I didn't see the original as I was not online at all those days), and (b) the impact that it clearly has on the use of Firefox-the-product - not just about:home or the snippets, with (c) no obvious information about who was involved in the decision (and whether that included the Firefox Desktop stakeholders like Gavin/Chad/Madhava or the Firefox UI/UX team).

> It raised over $300,000 from
> more than 30,000 users in the 48 hrs it was live.

Fair, this is significant. My concern is, also with regards to the decision-making, that I think it should not just be guided by the "effectiveness" of the snippet in fundraising terms. If we get 10% more fundraising revenue over that timeperiod but lose 2% of our users over it, that doesn't sound like a great tradeoff (potentially not even financially, but I've not done the math there).

> As for the mechanics, we
> agree completely:
> 
> 1. I would LOVE to turn off the fundraising snippet for donors (it's a
> substandard experience as-is, has been for 2 fundraising campaigns).

I've filed a separate bug about this.

> 2. I would LOVE to give people the chance to turn off a snippet if they find
> it annoying (that goes for any snippet; and for fundraising, they're not
> going to become donors anyway).

This probably makes sense. Maybe we could have a separate bug about this; I'll leave that up to mkelly...

However, from the feedback on input, it seems at least for some people the page was so unusable (due to the colors and their eyesight) that UI to turn it off would be "mustard after dinner", ie it would come too late. We ideally shouldn't be using a design that's quite that unusable.

> We are always open to ways we can improve the user experience - it's why we
> do so much user testing and why we monitor the donate@ inbox and other
> channels so carefully.

Can you elaborate on how this snippet was user tested? As I noted in comment #2, I couldn't find data about this. I know that we sometimes rely on outside help for the testing, or promise confidentiality to individual testers, so I can understand that it might not be 100% public, but it would be great if at least summaries, methods, timelines and the people involved were public. As it was, I couldn't find any of that info - but maybe I've not looked in the right places?

> We definitely do not want to harm our relationship
> with users -- that's the last thing we want to do. We have always been (and
> still are) willing to leave money on the table to preserve the user
> experience. 
> 
> That said, Mozilla is a nonprofit and we fundraise to stay sustainable (see
> this post for more on that:
> https://blog.mozilla.org/blog/2014/12/19/building-a-healthy-web-to-hand-to-
> future-generations/). 

I'm afraid that this misses the mark - I'm well aware of what Mozilla is and does, having volunteered for about 10 years and being on the Firefox Desktop team for the last (nearly) two. I'm suggesting we should course-adjust this specific type of campaigning (and/or perhaps our testing methodologies for it; I don't know what they are!) for next year's end of year fundraising. :-)

> I called for a way to turn off the campaign snippet for donors over a year
> ago, but as a feature, it wasn't prioritized. Here's another related bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1109100 

OK. I think we can do this in a privacy-sensitive way, but we can have the larger discussion on the other bug.

> If we can find a solution and implement a product update that makes the
> experience better on the snippet, I'm definitely on board. 

Great!

> I do have a Q: Is 21 comments to "Input" a high number? I don't have a frame
> of reference. For instance, when a new version of Firefox is released, how
> many negative "Input" comments come in, generally?

And this is a great question. We get input continuously - not just for new versions. The link given is for a period of one week - Dec. 24 to 31, for all comments matching "yellow". By comparison, during the same week we got 216 messages matching "donation" ( https://input.mozilla.org/en-US/?q=donation&date_end=2014-12-31&date_start=2014-12-24 - most of them seem to be about not wanting to see the message after donating ) and 9825 overall ( https://input.mozilla.org/en-US/?q=&date_end=2014-12-31&date_start=2014-12-24 ).

So 20 or even 216 doesn't seem like that much (compared to the 9,825 overall) - but as you said, this particular snippet only ran some of the time (although there is 'yellow' feedback from other days... did you run a trial? Did it run at other times for beta/aurora/nightly users?). When comparing the number of donation/yellow-related comments with the overall comments, I'd also point out that only a comparatively small number of our ADUs see about:home (cf. bug 1097232).

To give an alternative example, one of the Firefox platform team's Q1 priorities includes flash video issues (esp. on youtube) as we know it's an important reason people ditch Firefox for other browsers, and in this week, "only" 154 people mentioned youtube: https://input.mozilla.org/en-US/?q=youtube&date_end=2014-12-31&date_start=2014-12-24 . That is, despite the limited number of people seeing the snippet, more people wrote about "donation" than "youtube".

As you can see, interpreting data from input is a bit of an inexact science (I've not even mentioned extrapolating to people who don't write on input but are happy/sad anyway - when Firefox 29 (Australis) launched, input was pretty negative, but our usage numbers showed people were generally happy with it) - maybe I'm wildly overestimating the number of people who are annoyed at this.

Comment 10

[Andrea Wood [:andreawood]]

All really good points, and there's a lot to share / unpack here. Too bad you're in GB, i'd love to grab coffee and discuss!

There are quite a few people right now interested in the data that could help answer this important Q:  "If we get 10% more fundraising revenue over that timeperiod but lose 2% of our users over it, that doesn't sound like a great tradeoff (potentially not even financially, but I've not done the math there)." 

So far I've not had luck finding data that could tell us whether a fundraising snippet has a negative / positive impact on Firefox user behavior over the long term (e.g. uninstalls, reduced ADU; etc). Do you have a suggestion for where we might look, or how we might go about that?

There are a few blog posts on fundraising.mozilla.org about our testing and optimization, worth checking out. We also have a few more posts planned about our test results. Some snippet toplines:

EN-US impressions 2014 EOY snippet: 830,413,360 (down from 1,318,227,300 in 2013)
Average click through on EOY snippets: 0.20% 
Users who donated in 2014: 382,583 (up from 119,751 in 2013)
Visits to our donor FAQ during the campaign: 34,971 (https://wiki.mozilla.org/Donate)
Emails received from users asking for our mailing address to send Mozilla a donation by paper check: 194

Comment 11

[:Gijs (he/him)]

(In reply to Andrea Wood [:andreawood] from comment #10)
> All really good points, and there's a lot to share / unpack here. Too bad
> you're in GB, i'd love to grab coffee and discuss!

I'd be happy to chat on vidyo or IRC sometime your morning / my afternoon/evening, which should be workable timezone-wise. Bring your own coffee. ;-)

> There are quite a few people right now interested in the data that could
> help answer this important Q:  "If we get 10% more fundraising revenue over
> that timeperiod but lose 2% of our users over it, that doesn't sound like a
> great tradeoff (potentially not even financially, but I've not done the math
> there)." 
> 
> So far I've not had luck finding data that could tell us whether a
> fundraising snippet has a negative / positive impact on Firefox user
> behavior over the long term (e.g. uninstalls, reduced ADU; etc). Do you have
> a suggestion for where we might look, or how we might go about that?
> 
> There are a few blog posts on fundraising.mozilla.org about our testing and
> optimization, worth checking out.

I don't know what data we have about why users are sad about / leave Firefox, beyond input.m.o and bugreports here. I expect that over the entire year, requests for donations aren't generally going to be the major issue that we lose on. I doubt that we have specific data about the timeframe around the EOY fundraiser, when that proportion would likely be higher, but I don't know. Dolske?

Regarding the testing: I did read/skim most of the posts before now (via planet) - but as best I can tell they are all about revenue optimization. I also can't find basic details like how many people the snippets were tested on, and on which release channel (by the screenshot I'm guessing firefox developer edition / aurora, but maybe that's just coincidence and/or what Adam uses as his main browser? :-) ).

The kind of testing I meant is gathering a much smaller set of users and actually (remotely or physically) go through a flow with them, and deducing what works and what doesn't, either by analyzing their physical behaviour ("this person hesitated at this step") or asking them afterwards ("I didn't think this form was clear", "I couldn't find this button, maybe it should be more obvious", ...). I don't know how feasible this is with something like a snippet where the largest drop-off rate is before taking the first action, but that's what I associate with user testing. If it sounds appealing, I'm sure our user experience folks would be happy to talk to you about what could and couldn't work; I'm not an expert regarding this type of research myself.

While it's true that you can use large-scale quantitative a/b testing to help make UX/UI decisions, it seems that at least in this case, the interest/outcome that's being maximized (namely: foundation revenue) doesn't align with what users want. That is, I would contend that nobody who sees about:home every day desperately wants to use it to maximize Mozilla's donation revenue... even if that's what we try to do with the snippets. So to do 'user testing' rather than 'revenue-oriented a/b testing', I think we would need to start from wanting/evaluating a different outcome than just revenue.

Comment 12

[Justin Dolske [:Dolske]]

Looked into this earlier, there have already been discussions to make sure Firefox product folks are adequately involved in the snippet process.

I presume there's nothing more to do here since it looks like this snippet has already finished running.