1281568 - Setup Optimizely for Rebranded /new/ test (Sunrise in forest)

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Alex Davis [:adavis]]

All test details are here: 
https://mana.mozilla.org/wiki/display/FIREFOX/Download+page%3A+New+Design

Optimizely is being setup here:
https://app.optimizely.com/edit?experiment_id=6320381957

Related bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1280976

Tasks in Taiga:
https://tree.taiga.io/project/malexis-mozillaorg-durable-team/task/199

Comment 1

[Alex Davis [:adavis]]

:cmore

Please review the Optimizely test (link above) and sign off. I duplicated one of your past redirect tests so that the JS should be solid. I removed just one of your goals that didn't seem relevant anymore. Audience and targeting still seems accurate so I left it as is.

Comment 2

[Chris More [:cmore]]

(In reply to Alex Davis [:adavis] from comment #1)
> :cmore
> 
> Please review the Optimizely test (link above) and sign off. I duplicated
> one of your past redirect tests so that the JS should be solid. I removed
> just one of your goals that didn't seem relevant anymore. Audience and
> targeting still seems accurate so I left it as is.

Just made one change to the URL targeting for the ignore section:

Removed the (a|b) out of the end of the target since this redirect doesn't have v=1a as it is simply v=1

^.+www\.mozilla\.org/en\-.+\/firefox\/new\/\?v=[1-4].*$

The rest looks good to me.

Comment 3

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #2)
> ^.+www\.mozilla\.org/en\-.+\/firefox\/new\/\?v=[1-4].*$
> 
> The rest looks good to me.

Any reason why we still need the v=[1-4]? There's only one version in this test so we could probably call out ?v=1

Comment 4

[Alex Gibson [:agibson]]

We should also make sure this matches en-US only, rather than any en locale (the new design is shown for en-US only, not locales like en-GB).

Comment 5

[Alex Gibson [:agibson]]

Ah, I realize now that regex above is for the ignore section (I'm not yet too familiar with Optimizely). It should work fine as-is, but my suggestions above we're more specific to this experiment, rather than a general rule used in previous experiments. We don't have any other variation tests running at the minute fwiw, so this should be the only variant people hit.

Comment 6

[Alex Gibson [:agibson]]

I logged into Optimizely and took a quick look. The target URL is indeed set to:

^.+www\.mozilla\.org/en\-.+\/firefox\/new\/.*$

Please make sure this is en-US *only*, not any en variant.

Comment 7

[Chris More [:cmore]]

(In reply to Alex Gibson [:agibson] from comment #6)
> I logged into Optimizely and took a quick look. The target URL is indeed set
> to:
> 
> ^.+www\.mozilla\.org/en\-.+\/firefox\/new\/.*$
> 
> Please make sure this is en-US *only*, not any en variant.

URLs updated to be en-US and not en\-.+

Comment 8

[Alex Gibson [:agibson]]

The URL is now line on production:

https://www.mozilla.org/en-US/firefox/new/?v=1

Comment 9

[Alex Gibson [:agibson]]

Marking this bug as resolved. The two variations showed nearly identical performance, which gives us the confidence to move forward and make the forest theme default.

Thanks, all!