If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Replace 3.6 First Run with Winner of A/B Test

RESOLVED WONTFIX

Status

www.mozilla.org
General
RESOLVED WONTFIX
8 years ago
5 years ago

People

(Reporter: Blake Cutler, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

8 years ago
Winning design: http://www.mozilla.com/en-US/firefox/3.6/firstrun/d/

In this design, 35% more users interacted with the page, total interactions increased 104.4%, and 57.8% more users previewed Personas.  All of these results are statistically significant.

John, would you like to make any tweaks before pushing this live?
The new design (I like btw) doesn't have space for the rotating promos on the current page (RYF, Thunderbird, StumbleUpon, ReminderFox etc)

It also doesn't include Twitter, Blog, and Facebook links.  I could be missing other things that aren't included also.

What do we do about that stuff?  While the test page created more interaction with Personas and Addons, no one interacted with the removed promos.  Should that be seen as a negative impact?

Comment 2

8 years ago
To add to what Alex said, I think we should have a conversation with all the involved parties about the goals of this page before we move forward. 

For example, one of the most important strategic changes we made to this page with 3.6 was giving users the opportunity to engage with us through Twitter, Facebook and the Mozilla Blog. If we implemented this exact design without including those then I think that would be a big loss. Plus, the addition of those rotating promos has business value that needs to be considered.

But, this design obviously also has some compelling qualities that we should embrace and include. I'd like to find a way to keep those while also factoring in the issues mentioned above. My original understanding was that we'd do a 2nd round of testing based on that once we had an initial winner.

Anyway, I'll set up that meeting.

Comment 3

8 years ago
(In reply to comment #2)
> But, this design obviously also has some compelling qualities that we should
> embrace and include. I'd like to find a way to keep those while also factoring
> in the issues mentioned above. My original understanding was that we'd do a 2nd
> round of testing based on that once we had an initial winner.

I also should have mentioned that I'd like to polish up the design a little bit before we implement.
(Reporter)

Comment 4

8 years ago
@alex: I did consider those negative effects.  On the control, under 1% of visitors clicked any of the social media/blog links.  Likewise, under 1% of visitors clicked on the featured promotions.  

Conversely, with the new design, 7.9% of users interacted with the expander tabs, .7% of users installed an Add-on, and .9% either clicked on an support article or searched for help. 

I'd like to go ahead push the design and test adding additional features, such as social media.  I think the 57% Personas preview improvement and the support improvement justify making this change now.
(Reporter)

Comment 5

8 years ago
I agree with John about polishing the design (some elements look a little off).  How would you like to go about this?
(Reporter)

Comment 6

8 years ago
Partially for my reference, here are a few tests I want to run after pushing this (modified?) design live:

* Add social media links in right-hand column
* Add social media in expander tab 
* Alter the order of the expander tabs
* New variations that load faster (i.e. sprite images) 
* Add/remove expanders 
* Replace the current expanders (i.e. "Set your preferences")
* Add more/different support articles
* Swap column 1 and column 2 positions

I can't run these tests effectively until we replace the current design.

Comment 7

8 years ago
Let's meet on Monday and figure out what to do. My vote would be to polish the current design and push that soon, but not before we've had the conversation.

It's also worth noting that click-thru stats don't tell the whole story...even less than 1% are clicking on those engagement promos, that's still translated into a massive increase in the actual number fans and followers.

Bottom line, I want to make sure that we're considering the whole picture before we push new content live on such an important page. We can get this figured out soon.
(Reporter)

Comment 8

8 years ago
I hear what you're saying, but I would like to make this change before 3.6.2 is released (March 30th?).  If we miss this, it could be a month plus before we have another traffic spike that large.  

From a UX standpoint, I would arguing answering users' support questions is more important that getting them to connect with Firefox.
3.6.2 will likely be shipped today or tomorrow, due to a change in urgency around a security fix. I don't think you should tie this to that.

There are valid product design concerns to be resolved here before executing, too, as noted in comment 2 and comment 4. The a/b test told us where clicks were going, but wasn't engineered to understand where we want clicks to go, from what I'm seeing.

Also: next time I'd appreciate being cc'd on these sorts of bugs.

Comment 10

8 years ago
Quick update: cbeard, the various content stakeholders and I met yesterday and talked through this. The new plan is to have Sean rapidly iterate on some feedback that came out of yesterday's meeting, with the goal of having more versions to test that build on the learnings from the option shown in comment #0 with more polished design and content that's more consistent with our goals for this page.

More details to come...

Updated

8 years ago
Depends on: 555098
(Assignee)

Updated

6 years ago
Component: www.mozilla.org/firefox → www.mozilla.org
Product: Websites → Websites
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
(Assignee)

Updated

5 years ago
Component: www.mozilla.org → General
Product: Websites → www.mozilla.org
You need to log in before you can comment on or make changes to this bug.