Winning design: http://www.mozilla.com/en-US/firefox/3.6/firstrun/d/ In this design, 35% more users interacted with the page, total interactions increased 104.4%, and 57.8% more users previewed Personas. All of these results are statistically significant. John, would you like to make any tweaks before pushing this live?
The new design (I like btw) doesn't have space for the rotating promos on the current page (RYF, Thunderbird, StumbleUpon, ReminderFox etc) It also doesn't include Twitter, Blog, and Facebook links. I could be missing other things that aren't included also. What do we do about that stuff? While the test page created more interaction with Personas and Addons, no one interacted with the removed promos. Should that be seen as a negative impact?
To add to what Alex said, I think we should have a conversation with all the involved parties about the goals of this page before we move forward. For example, one of the most important strategic changes we made to this page with 3.6 was giving users the opportunity to engage with us through Twitter, Facebook and the Mozilla Blog. If we implemented this exact design without including those then I think that would be a big loss. Plus, the addition of those rotating promos has business value that needs to be considered. But, this design obviously also has some compelling qualities that we should embrace and include. I'd like to find a way to keep those while also factoring in the issues mentioned above. My original understanding was that we'd do a 2nd round of testing based on that once we had an initial winner. Anyway, I'll set up that meeting.
(In reply to comment #2) > But, this design obviously also has some compelling qualities that we should > embrace and include. I'd like to find a way to keep those while also factoring > in the issues mentioned above. My original understanding was that we'd do a 2nd > round of testing based on that once we had an initial winner. I also should have mentioned that I'd like to polish up the design a little bit before we implement.
@alex: I did consider those negative effects. On the control, under 1% of visitors clicked any of the social media/blog links. Likewise, under 1% of visitors clicked on the featured promotions. Conversely, with the new design, 7.9% of users interacted with the expander tabs, .7% of users installed an Add-on, and .9% either clicked on an support article or searched for help. I'd like to go ahead push the design and test adding additional features, such as social media. I think the 57% Personas preview improvement and the support improvement justify making this change now.
I agree with John about polishing the design (some elements look a little off). How would you like to go about this?
Partially for my reference, here are a few tests I want to run after pushing this (modified?) design live: * Add social media links in right-hand column * Add social media in expander tab * Alter the order of the expander tabs * New variations that load faster (i.e. sprite images) * Add/remove expanders * Replace the current expanders (i.e. "Set your preferences") * Add more/different support articles * Swap column 1 and column 2 positions I can't run these tests effectively until we replace the current design.
Let's meet on Monday and figure out what to do. My vote would be to polish the current design and push that soon, but not before we've had the conversation. It's also worth noting that click-thru stats don't tell the whole story...even less than 1% are clicking on those engagement promos, that's still translated into a massive increase in the actual number fans and followers. Bottom line, I want to make sure that we're considering the whole picture before we push new content live on such an important page. We can get this figured out soon.
I hear what you're saying, but I would like to make this change before 3.6.2 is released (March 30th?). If we miss this, it could be a month plus before we have another traffic spike that large. From a UX standpoint, I would arguing answering users' support questions is more important that getting them to connect with Firefox.
3.6.2 will likely be shipped today or tomorrow, due to a change in urgency around a security fix. I don't think you should tie this to that. There are valid product design concerns to be resolved here before executing, too, as noted in comment 2 and comment 4. The a/b test told us where clicks were going, but wasn't engineered to understand where we want clicks to go, from what I'm seeing. Also: next time I'd appreciate being cc'd on these sorts of bugs.
Quick update: cbeard, the various content stakeholders and I met yesterday and talked through this. The new plan is to have Sean rapidly iterate on some feedback that came out of yesterday's meeting, with the goal of having more versions to test that build on the learnings from the option shown in comment #0 with more polished design and content that's more consistent with our goals for this page. More details to come...