1150728 - Shape of the Web Snippet (Copy and icon) Testing

Status: RESOLVED FIXED

(Marketing :: Design, task)

Description

[Jean Collings]

>>Project/Request Title:
Shape of the Web Snippet (Copy and icon) Testing 

>>Project Overview:
During the soft launch of the Shape of the Web website, we'd like to test messaging and icons for the about:home snippets (this will help us optimize for the actual launch in the summer).



>> Creative Help Needed:
Copy:    Yes 
Design:  Yes 
Video:   No 
Other:  No

>>Creative Specs:
> Copy:
- Approximately 10 snippet messages. Greg has put together a draft and we would love to get feedback from Matej and Andrea, to pick the best options and some wordsmithing as well.
- Draft found here: https://docs.google.com/a/mozilla.com/document/d/1m4uzls5Wgj4y_iYmxmLuhi-JUTB3fNKAb7fnHgTf740/edit

> Visual:
- 1 standard SOTW icon (40 x 50px): should be representative of SOTW look & feel. We can test different options (e.g. sphere vs colored vs dotsanimated gif)... If possible, would be great to have Lee do this since he's working on SOTW already.

> Flight date
Testing during 10 days from 4-27 to 5/6 

> Deadline:
4/26


>>CTA and Design:
See above

>>Expected Launch Date:
2015-04-27

>>Mozilla Goal:
Shape of the Web

>>Mozilla Creative Collective:
No

>>Points of Contact:
[e-mail], [e-mail], [e-mail], [e-mail],

Comment 1

[Matej Novak [:matej]]

My comments and edits are in. Let me know if you have any questions. Thanks.

Comment 2

[Gregory Jost]

Thanks Matej and Andrea for your comments. I addressed most of them where I could, but this also gave me ideas for new variations to test.

Jean, are we limited in terms of the number of messages we can test? (I am at 20 right now). Can you provide some guidance on any limit and how much time/ how many impressions each message needs for the result to be statistically significant?

Comment 3

[Jean Collings]

(In reply to Gregory Jost from comment #2)
> Thanks Matej and Andrea for your comments. I addressed most of them where I
> could, but this also gave me ideas for new variations to test.
> 
> Jean, are we limited in terms of the number of messages we can test? (I am
> at 20 right now). Can you provide some guidance on any limit and how much
> time/ how many impressions each message needs for the result to be
> statistically significant?

Hi Greg,
I left some small comments in the doc. 

20 is a lot to test given that publishing and pulling metrics is a timely task. To answer your question about statistically significant, it all depends on our "expected conversion rate" as well the minimum relative change in the conversion rate we want to see. So for example if we expect 1% CTR (that's generous) and need to see a 5-10% change in order to make a decision then we'd need x impressions. Based on what we've done in the past, on a smaller rotation with about 4 snippets in testing, 3 days will suffice.

So with 10 days, and 4 snippets at a time - we can do about 12ish snippets total. If we can cut some down in the doc that would be great. Lmk what you think.

Comment 4

[Gregory Jost]

A fun idea for an animated snippet icon: http://cl.ly/image/1u0w0M041X43.
This is less urgent than bug 1150129 but also wanted to check on status. Let me know what/when seems reasonable!

Comment 5

[Jean Collings]

Attachment: snippet_tubes_40x50px.gif

We have this one that Foundation provided awhile back for something else. Kind of similar...

Comment 6

[Jean Collings]

(In reply to Jean Collings from comment #5)
> Created attachment 8592399 [details]
> snippet_tubes_40x50px.gif
> 
> We have this one that Foundation provided awhile back for something else.
> Kind of similar...

Now that I'm seeing the GOOD email, we should keep the look and feel consistent so ignore what I put in comment 5 :)

Comment 7

[Gregory Jost]

Hey Lee, were you able to make any progress on this?

Comment 8

[Lee Tom]

Attachment: sotw_snippet_40x50.gif

Hi all, here is an animated snippet for SoTW. I also have an @2x version for high resolution displays.  Below I will attach a static version showing the 3 circles

Comment 9

[Lee Tom]

Attachment: sotw_snippet_80x100_static.jpg

Static graphic @2x (80x100px). I will supply SVG upon approval

Comment 10

[Jean Collings]

Hey Greg,
For purposes of testing we should probably use the static one (40 x 50 svg needed) and then use the actual animated one when we do the full launch. These icons look fine to me - so I leave it to Greg to make the final decision.

Thanks!

Comment 11

[Gregory Jost]

Lee, do you realize your animated gif might be too efficient for our test? ;)

I hear you Jean, since we're trying to minimize the traffic, the simple static version might provide a better baseline. I would be interested in comparing the impact or using a static vs animated visual on one message if we can. Will be great to know and have for our public launch. Lee, the gif will not go unused!

I am setting up time to review all test assets with MEM but this will likely happen at the end of the week or beginning of next. Jean and I agreed to push the test back by a few days so we should good on timing.

Will post an update here when I hear back from MEM.

Comment 12

[Gregory Jost]

Quick update on this: reviewed this with MEM and she is fine with attachment 8598242 [details] but asked if we can try a version of the static visual with a container (circle/disk) or some added depth to act as anchor point.

Thanks Lee!

Comment 13

[Lee Tom]

Attachment: sotw_snippet_80x100_static_02.jpg

Heya Greg, here's another version of the static snippet - this time, with additional details similar to the GOOD email and Facebook graphics.  I think the subtle rings, gradient colors and solid halo behind the dots help give more weight to the cluster. LMK what you think, or if you want to see another version. Thanks

Comment 14

[Gregory Jost]

Thanks Lee.

Feels like the circle clashes a bit with the about:home grey. Do you think using another color could work here? Maybe a slightly darker grey than the about:home default? Let me know what you think.

If it doesn't work let's not bother with a circle at all and keep the dots free!

Comment 15

[Lee Tom]

Attachment: sotw_snippet_80x100_static_03.jpg

I agree. Going darker will be misleading (and inconsistent with the actual website) and make the graphic really crowded. I think we should leave it out and let the dots float freely!

Comment 16

[Gregory Jost]

It is a wrap then! Thanks Lee! Jean, if this works with you, here's our visual.

Comment 17

[Jean Collings]

(In reply to Lee Tom from comment #15)
> Created attachment 8602888 [details]
> sotw_snippet_80x100_static_03.jpg
> 
> I agree. Going darker will be misleading (and inconsistent with the actual
> website) and make the graphic really crowded. I think we should leave it out
> and let the dots float freely!

Lee can you provide the 40 x 50 png and svg version? Thanks!

Comment 18

[Lee Tom]

Hi Jean! 
Find the assets in the link: http://cl.ly/0m2l1H3i0g0g

Included there is: 
- 40x50 static .png
- 80x100 static .png (for retina displays)
- 40x50 scalable .svg


> 
> Lee can you provide the 40 x 50 png and svg version? Thanks!

Comment 19

[Jean Collings]

Thanks Lee!