1338667 - Setup First Run Optimizely to Test Headlines instead of Almost Done

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Alex Davis [:adavis]]

Feeley, Verdi, Pdol and I are testing new copy on first run to see if we can replace Almost Done with something that performs somewhat similarly.

Normally it's a pretty simple copy test.

Test is being setup here:
https://app.optimizely.com/edit?experiment_id=8227473099

Test Recipe is being documented here:
https://mana.mozilla.org/wiki/display/FIREFOX/First+Run%3A+Headline+Tests

Comment 1

[Jonathan Petto [:jpetto]]

Sounds like this is an experiment we could possibly run with Traffic Cop [1]. I'm happy to do the dev for the performance win.

:adavis - What do you think?


[1] http://bedrock.readthedocs.io/en/latest/mozilla-traffic-cop.html

Comment 2

[Alex Davis [:adavis]]

I appreciate the offer but I've already setup the Optimizely test. I used it as an example to show the guys how Optimizely works.

I just wanted to complete the test recipe before submitting it to cmore and then you for sign-off.

I will however certainly read up about Traffic Cop. I'm unfamiliar with it.

For informational purposes, is there a type of test that traffic cop is better suited for? How does tracking work? For example, can GA tell the difference in version? Can it send different parameters to the FxA signup form so that additional tracking can happen there? Thanks in advance for answering my questions. I love to hear about new testing tools.

Comment 3

[Alex Davis [:adavis]]

:Cmore Please review this optimizely test. Note that we are just testing copy.

The only particularity of this test is that I want to exclude funnelcakes so I have conditions in URL targeting for this. Review there would be appreciated.

If it looks good to you. Please ni? :jpetto for his r+ .

Comment 4

[Chris More [:cmore]]

(In reply to Alex Davis [:adavis] [PM FxA+Sync] from comment #3)
> :Cmore Please review this optimizely test. Note that we are just testing
> copy.
> 
> The only particularity of this test is that I want to exclude funnelcakes so
> I have conditions in URL targeting for this. Review there would be
> appreciated.
> 
> If it looks good to you. Please ni? :jpetto for his r+ .

Couple things:

1) I removed the URL target for funnelcakes as it was redundant since your target url ended with /$. Nothing will come after the slash regardless if it was f= or foobar.

2) adjusted version b that had replacewith instead of html jquery object.

3) what is the purpose of the click to setup sync goal if there is nothing on the page except the iframe? can we remove that goal?

Other than to remove that unneeded goal, looks good!

Comment 5

[Alex Davis [:adavis]]

Thanks :cmore

I removed that event because I'm getting it from FxA event metrics anyways.

:jpetto can you sign off?

Comment 6

[Jonathan Petto [:jpetto]]

The code looks okay, but verifying that utm_content=fx-50.0 is okay (as the page is targeting 51.x).

Are the two params in the #fxa querysting that regularly need adjustment for experiments utm_campaign and utm_source? It would be nice if we could find a less heavy-handed way to change individual pieces of the #fxa src URL.

Comment 7

[Alex Davis [:adavis]]

Thanks. I changed it to 51. Will launch test now.

I'd love to explore options with you to more efficiently pass tracking to FxA iFrame. Did you have any ideas in mind for updating those 2 params?

Comment 8

[Jonathan Petto [:jpetto]]

I'd recommend something like the following:

// reference to #fxa element
var $fxa = $('#fxa');

// clone the original data-src attribute
var newSrc = $fxa.attr('data-src');

// replace specific query params in the cloned data-src value
newSrc = newSrc.replace('utm_campaign=fxa-embedded-form', 'utm_campaign=fxa-embedded-form-signup');
newSrc = newSrc.replace('utm_source=firstrun', 'utm_source=firstrun_20170214_B');

// update the data-src attribute
$fxa.attr('data-src', newSrc);

This way you're only changing targeted key/value pairs, and not replacing the entire URL. This method is much less error prone.

Comment 9

[Alex Davis [:adavis]]

Great. I've noted this for future reference. This looks pretty slick.

If we can minimize errors, I'm all for it. I'll make sure :cmore sees this too.

Comment 10

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #8)
> I'd recommend something like the following:
> 
> // reference to #fxa element
> var $fxa = $('#fxa');
> 
> // clone the original data-src attribute
> var newSrc = $fxa.attr('data-src');
> 
> // replace specific query params in the cloned data-src value
> newSrc = newSrc.replace('utm_campaign=fxa-embedded-form',
> 'utm_campaign=fxa-embedded-form-signup');
> newSrc = newSrc.replace('utm_source=firstrun',
> 'utm_source=firstrun_20170214_B');
> 
> // update the data-src attribute
> $fxa.attr('data-src', newSrc);
> 
> This way you're only changing targeted key/value pairs, and not replacing
> the entire URL. This method is much less error prone.

Awesome! Thanks, Jon. Good stuff!

Comment 11

[pgerman]

This looks like it is complete.