Closed Bug 1221328 Opened 9 years ago Closed 8 years ago

Set Winning Android Page as Default

Categories

(www.mozilla.org :: Pages & Content, defect)

Product:
www.mozilla.org
Component:
Pages & Content
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: adavis, Assigned: jpetto)

References

Details

(Whiteboard: [kb=1888137]) 

We recently ran an A/B test on the Android page with 2 different send to device widgets and one with just a GPlay button.

This version won: https://www.mozilla.org/en-US/firefox/android/?v=b

Please set as default.
Assignee: nobody → jon
Component: General → Pages & Content
Whiteboard: [kb=1888137]
Just confirming we can remove code for the losing variations and the current default/non-variant UI. Also confirming we no longer need to set a custom basket id.

Are there any other experiments running on this page? Can we disable Optimizely for now?
Flags: needinfo?(adavis)
Blocks: 1221338
(In reply to Jon Petto [:jpetto] from comment #1)
> Just confirming we can remove code for the losing variations and the current
> default/non-variant UI. Also confirming we no longer need to set a custom
> basket id.
> 
> Are there any other experiments running on this page? Can we disable
> Optimizely for now?

You can disable Optimizely for now. The next test which we are starting to prepare won't be immediately ready so we can re-enable it when needed.

As per the baskets, since we are preparing another test, perhaps we can re-use them to avoid having to create new ones (as well as avoid creating new email templates). Let me reflect on this a bit more because I want to make sure it doesn't impact tracking in any way. I'd hate to try to save time but cause other problems.

Finally, I will let Jenn answer as per removing the code of the losing variations. I don't think it will be needed again but I'd prefer for her to sign off on that. I don't know if there could be any other repercussions.
Flags: needinfo?(adavis) → needinfo?(jbertsch)
(In reply to Alex Davis [:adavis] from comment #2)
> (In reply to Jon Petto [:jpetto] from comment #1)
> > Just confirming we can remove code for the losing variations and the current
> > default/non-variant UI. Also confirming we no longer need to set a custom
> > basket id.
> > 
> > Are there any other experiments running on this page? Can we disable
> > Optimizely for now?
> 
> You can disable Optimizely for now. The next test which we are starting to
> prepare won't be immediately ready so we can re-enable it when needed.
> 
> As per the baskets, since we are preparing another test, perhaps we can
> re-use them to avoid having to create new ones (as well as avoid creating
> new email templates). Let me reflect on this a bit more because I want to
> make sure it doesn't impact tracking in any way. I'd hate to try to save
> time but cause other problems.
> 
> Finally, I will let Jenn answer as per removing the code of the losing
> variations. I don't think it will be needed again but I'd prefer for her to
> sign off on that. I don't know if there could be any other repercussions.

I don't think they will be needed either. And if we do need them, they still live somewhere in Git, right?

Let's remove. Thanks, jpetto!
Flags: needinfo?(jbertsch)
Yep, we'll have the Git history. I'll get the front-end all set, and wait to hear back about basket ids prior to issuing a PR.
(In reply to Jon Petto [:jpetto] from comment #4)
> Yep, we'll have the Git history. I'll get the front-end all set, and wait to
> hear back about basket ids prior to issuing a PR.

Alex - Any further thoughts on the basket ids?
Flags: needinfo?(adavis)
Just to clarify, here are the current basket ids:

SMS:

SMS_Android (default, modal, non-variant)
android-download-notembed (variation a, modal)
android-download-embed (variation b, embed)

Email:

download-firefox-android (default, modal, non-variant)
get-android-notembed (variation a, modal)
get-android-embed (variation b, embed)
(In reply to Jon Petto [:jpetto] from comment #6)
> Just to clarify, here are the current basket ids:
> 
> SMS:
> 
> SMS_Android (default, modal, non-variant)
> android-download-notembed (variation a, modal)
> android-download-embed (variation b, embed)
> 
> Email:
> 
> download-firefox-android (default, modal, non-variant)
> get-android-notembed (variation a, modal)
> get-android-embed (variation b, embed)

I imagine the default baskets are used somewhere else (like snippets) so I wouldn't touch those. (correct me if I'm wrong)
SMS: SMS_Android
Email: download-firefox-android

Keep the winning basket for this page only. It'll allow us to continue to measure the performance of this page.
SMS: android-download-embed
Email: get-android-embed

Delete the losers:
SMS: android-download-notembed
Email: get-android-notembed


For our new test, I'll make 2 new ones with appropriate names and trackers.
Flags: needinfo?(adavis)
:agibson brought up a good question while reviewing the PR:

Are the basket messages (android-download-embed, get-android-embed) translated? For the previous test, we restricted to en-US only, so this wasn't an issue. However, before we roll out to all audiences, we need to make sure we've got these messages localized.
Flags: needinfo?(adavis)
I think the basket only has English because it was created just for the test but I'm a little unclear on how baskets are setup and how they integrate to the widget in every language. 

Do we have to create a different basket for every language or do we add additional languages to a basket?

Normally, all the languages were integrated in the pre-test version of the widget.

Perhaps you or Ben can clarify this for me.
Thanks
Flags: needinfo?(jon)
Flags: needinfo?(bniolet)
Flags: needinfo?(adavis)
I am untrained in the ways of basket. I'll let Ben chime in here.
Flags: needinfo?(jon)
We have Android messages in all eight languages that our email program currently supports: EN, ES, DE, FR, PT-BR, RU, ID, PL. 

Basket is how we can add new language functionality to a form. For example, on this page: 

mozilla.org/newsletter 

A setting in basket determines which languages appear in the drop down window. That said, if we're doing testing variants for each language, the overhead in ExactTarget setup work becomes exponentially greater (2 splits x 8 languages x 2 formats (text and html) = 32 unique triggers and unique 32 emails to set up so be kind to your email team.
Flags: needinfo?(bniolet)
Ben - 

Sorry for my ignorance, but does that mean the basket ids we want to roll out (android-download-embed and get-android-embed) are ready for all locales supported by our email program?
Flags: needinfo?(bniolet)
Does the form include a language drop-down?
Flags: needinfo?(bniolet) → needinfo?(jon)
The form is the send to device widget, and does not include a language drop down. This widget is limited by a pre-defined list of locales [1]. If the user does not fall into that set of locales, they are not shown the form.

I believe we just need to make sure the two new basket messages (android-download-embed and get-android-embed) match the list of said locales.

[1] https://github.com/mozilla/bedrock/blob/master/bedrock/settings/base.py#L752
Flags: needinfo?(jon)
Flags: needinfo?(bniolet)
Okay, so not to make this any more complex, but let's lay out what's needed and what we have:

>> SMS: android-download-embed

We only ever do SMS in English, so this is good to go. 

Email: get-android-embed is currently live in EN.

This can *relatively* quickly be expanded to the following languages: ES, FR, DE, PL, RU, ID, PT-BR. Does the form allow users to opt into a TEXT only version or are we defaulting to HTML? 

Here's what needs to happen: 

Alex: Confirm for me the link you want to use in each of the versions localized versions. 

Ben: I'll need to make a very small change to basket to activate the other languages. I'll also need to set up triggers / messages for each of the languages (need to know if we're going to have TXT only versions before I do this).
Flags: needinfo?(jon)
Flags: needinfo?(bniolet)
Flags: needinfo?(adavis)
The form doesn't provide an option of text vs. HTML, and the messages come through as HTML, so I'd say that's the default. :)
Flags: needinfo?(jon)
Good to know. Saves me a ton of work that way. Still need the link from adavis.
Email triggers are live and should be working for: 

es, de, pl, pt-br, ru, id, fr. 

HTML only. 

Alex, if you want a unique link for each email, let me know. Otherwise, they're all using the same link. jpetto, let me know if you need help testing.
Flags: needinfo?(jon)
More testing is always good. :D

The updated form is on demo3 [1]. Pay no attention to the visual issues on that page - there are many branches smashed together on demo3 at the moment.

[1] https://www-demo3.allizom.org/firefox/android/
Flags: needinfo?(jon)
Checking back on this bug - are we ready to merge the changes?
Jpetto: 

I am not sure how to test whether language works since the back-end appears to auto-select language based on my browser settings (i.e. switching the form to /fr-fr/ still sends an EN message). 

Do we have a way (like VPNs from around the world maybe?) to test whether the language triggering works? 

Other than that, I'm fine. Never heard back from adavis on whether the links are what he wants them to be.
Catching up on all these bugs. 2 weeks of honeymoon and 1 week of Mozlando really makes this hard to jump back into.

(In reply to Ben Niolet from comment #18)
> Alex, if you want a unique link for each email, let me know. Otherwise,
> they're all using the same link. jpetto, let me know if you need help
> testing.

The English template should already have a tracking URL. Don't worry about the other languages. Nothing prevents me down the road to add a tracking URL to other languages. Let's keep it simple for now.

Let me know if there's anything else I've missed and that needs my attention. You guys seem to be covering all angles.
Flags: needinfo?(adavis)
(In reply to Ben Niolet from comment #18)
> Email triggers are live and should be working for: 
> 
> es, de, pl, pt-br, ru, id, fr. 
> 
> HTML only. 

Hm, according to the data [1], the 'get-android-embed' message is only set for 'en'. I *think* that's the issue here, as visiting a supported URL [2] should send the correct message.

:bniolet - Can you double check basket to make sure the new message is enabled for all necessary locales?

[1] https://basket.mozilla.org/news/newsletters/
[2] https://www-demo3.allizom.org/de/firefox/android/
Flags: needinfo?(bniolet)
ah, yes, user error on my part. 

Which instance of basket is demo3 pointed at? 

Dev, Stage or Prod?
Flags: needinfo?(bniolet)
^
Flags: needinfo?(jon)
:pmac checked last night and said demo3 is pointed at prod basket.
Flags: needinfo?(jon)
it works!!!

fixed two errors in how I set up basket. all langs are live now and tested!!!
Flags: needinfo?(jon)
Hurray! We'll shoot to get this out on Monday.
Flags: needinfo?(jon)
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/929f1238dc68b17bc37802ff4f4823ab8340c6ec
[fix bug 1221328] Update /firefox/android/ send to device widget.

https://github.com/mozilla/bedrock/commit/149b036c8fc22c9d71f6afe7a8921f2decf65cc3
Merge pull request #3534 from jpetto/bug-1221328-update-android-send-to-device

[fix bug 1221328] Update /firefox/android/ send to device widget.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.