Set Winning Android Page as Default

RESOLVED FIXED

Status

www.mozilla.org
Pages & Content
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: adavis, Assigned: jpetto)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [kb=1888137])

(Reporter)

Description

3 years ago
We recently ran an A/B test on the Android page with 2 different send to device widgets and one with just a GPlay button.

This version won: https://www.mozilla.org/en-US/firefox/android/?v=b

Please set as default.
(Assignee)

Updated

3 years ago
Assignee: nobody → jon
Component: General → Pages & Content
Whiteboard: [kb=1888137]
(Assignee)

Comment 1

3 years ago
Just confirming we can remove code for the losing variations and the current default/non-variant UI. Also confirming we no longer need to set a custom basket id.

Are there any other experiments running on this page? Can we disable Optimizely for now?
Flags: needinfo?(adavis)
(Reporter)

Updated

3 years ago
Blocks: 1221338
(Reporter)

Comment 2

3 years ago
(In reply to Jon Petto [:jpetto] from comment #1)
> Just confirming we can remove code for the losing variations and the current
> default/non-variant UI. Also confirming we no longer need to set a custom
> basket id.
> 
> Are there any other experiments running on this page? Can we disable
> Optimizely for now?

You can disable Optimizely for now. The next test which we are starting to prepare won't be immediately ready so we can re-enable it when needed.

As per the baskets, since we are preparing another test, perhaps we can re-use them to avoid having to create new ones (as well as avoid creating new email templates). Let me reflect on this a bit more because I want to make sure it doesn't impact tracking in any way. I'd hate to try to save time but cause other problems.

Finally, I will let Jenn answer as per removing the code of the losing variations. I don't think it will be needed again but I'd prefer for her to sign off on that. I don't know if there could be any other repercussions.
Flags: needinfo?(adavis) → needinfo?(jbertsch)
(In reply to Alex Davis [:adavis] from comment #2)
> (In reply to Jon Petto [:jpetto] from comment #1)
> > Just confirming we can remove code for the losing variations and the current
> > default/non-variant UI. Also confirming we no longer need to set a custom
> > basket id.
> > 
> > Are there any other experiments running on this page? Can we disable
> > Optimizely for now?
> 
> You can disable Optimizely for now. The next test which we are starting to
> prepare won't be immediately ready so we can re-enable it when needed.
> 
> As per the baskets, since we are preparing another test, perhaps we can
> re-use them to avoid having to create new ones (as well as avoid creating
> new email templates). Let me reflect on this a bit more because I want to
> make sure it doesn't impact tracking in any way. I'd hate to try to save
> time but cause other problems.
> 
> Finally, I will let Jenn answer as per removing the code of the losing
> variations. I don't think it will be needed again but I'd prefer for her to
> sign off on that. I don't know if there could be any other repercussions.

I don't think they will be needed either. And if we do need them, they still live somewhere in Git, right?

Let's remove. Thanks, jpetto!
Flags: needinfo?(jbertsch)
(Assignee)

Comment 4

3 years ago
Yep, we'll have the Git history. I'll get the front-end all set, and wait to hear back about basket ids prior to issuing a PR.
(In reply to Jon Petto [:jpetto] from comment #4)
> Yep, we'll have the Git history. I'll get the front-end all set, and wait to
> hear back about basket ids prior to issuing a PR.

Alex - Any further thoughts on the basket ids?
Flags: needinfo?(adavis)
(Assignee)

Comment 6

3 years ago
Just to clarify, here are the current basket ids:

SMS:

SMS_Android (default, modal, non-variant)
android-download-notembed (variation a, modal)
android-download-embed (variation b, embed)

Email:

download-firefox-android (default, modal, non-variant)
get-android-notembed (variation a, modal)
get-android-embed (variation b, embed)
(Reporter)

Comment 7

3 years ago
(In reply to Jon Petto [:jpetto] from comment #6)
> Just to clarify, here are the current basket ids:
> 
> SMS:
> 
> SMS_Android (default, modal, non-variant)
> android-download-notembed (variation a, modal)
> android-download-embed (variation b, embed)
> 
> Email:
> 
> download-firefox-android (default, modal, non-variant)
> get-android-notembed (variation a, modal)
> get-android-embed (variation b, embed)

I imagine the default baskets are used somewhere else (like snippets) so I wouldn't touch those. (correct me if I'm wrong)
SMS: SMS_Android
Email: download-firefox-android

Keep the winning basket for this page only. It'll allow us to continue to measure the performance of this page.
SMS: android-download-embed
Email: get-android-embed

Delete the losers:
SMS: android-download-notembed
Email: get-android-notembed


For our new test, I'll make 2 new ones with appropriate names and trackers.
Flags: needinfo?(adavis)
(Assignee)

Comment 8

3 years ago
:agibson brought up a good question while reviewing the PR:

Are the basket messages (android-download-embed, get-android-embed) translated? For the previous test, we restricted to en-US only, so this wasn't an issue. However, before we roll out to all audiences, we need to make sure we've got these messages localized.
Flags: needinfo?(adavis)
(Reporter)

Comment 9

3 years ago
I think the basket only has English because it was created just for the test but I'm a little unclear on how baskets are setup and how they integrate to the widget in every language. 

Do we have to create a different basket for every language or do we add additional languages to a basket?

Normally, all the languages were integrated in the pre-test version of the widget.

Perhaps you or Ben can clarify this for me.
Thanks
Flags: needinfo?(jon)
Flags: needinfo?(bniolet)
Flags: needinfo?(adavis)
(Assignee)

Comment 10

3 years ago
I am untrained in the ways of basket. I'll let Ben chime in here.
Flags: needinfo?(jon)

Comment 11

3 years ago
We have Android messages in all eight languages that our email program currently supports: EN, ES, DE, FR, PT-BR, RU, ID, PL. 

Basket is how we can add new language functionality to a form. For example, on this page: 

mozilla.org/newsletter 

A setting in basket determines which languages appear in the drop down window. That said, if we're doing testing variants for each language, the overhead in ExactTarget setup work becomes exponentially greater (2 splits x 8 languages x 2 formats (text and html) = 32 unique triggers and unique 32 emails to set up so be kind to your email team.
Flags: needinfo?(bniolet)
(Assignee)

Comment 12

3 years ago
Ben - 

Sorry for my ignorance, but does that mean the basket ids we want to roll out (android-download-embed and get-android-embed) are ready for all locales supported by our email program?
Flags: needinfo?(bniolet)

Comment 13

3 years ago
Does the form include a language drop-down?
Flags: needinfo?(bniolet) → needinfo?(jon)
(Assignee)

Comment 14

3 years ago
The form is the send to device widget, and does not include a language drop down. This widget is limited by a pre-defined list of locales [1]. If the user does not fall into that set of locales, they are not shown the form.

I believe we just need to make sure the two new basket messages (android-download-embed and get-android-embed) match the list of said locales.

[1] https://github.com/mozilla/bedrock/blob/master/bedrock/settings/base.py#L752
Flags: needinfo?(jon)
(Assignee)

Updated

3 years ago
Flags: needinfo?(bniolet)

Comment 15

3 years ago
Okay, so not to make this any more complex, but let's lay out what's needed and what we have:

>> SMS: android-download-embed

We only ever do SMS in English, so this is good to go. 

Email: get-android-embed is currently live in EN.

This can *relatively* quickly be expanded to the following languages: ES, FR, DE, PL, RU, ID, PT-BR. Does the form allow users to opt into a TEXT only version or are we defaulting to HTML? 

Here's what needs to happen: 

Alex: Confirm for me the link you want to use in each of the versions localized versions. 

Ben: I'll need to make a very small change to basket to activate the other languages. I'll also need to set up triggers / messages for each of the languages (need to know if we're going to have TXT only versions before I do this).
Flags: needinfo?(jon)
Flags: needinfo?(bniolet)
Flags: needinfo?(adavis)
(Assignee)

Comment 16

3 years ago
The form doesn't provide an option of text vs. HTML, and the messages come through as HTML, so I'd say that's the default. :)
Flags: needinfo?(jon)

Comment 17

3 years ago
Good to know. Saves me a ton of work that way. Still need the link from adavis.

Comment 18

3 years ago
Email triggers are live and should be working for: 

es, de, pl, pt-br, ru, id, fr. 

HTML only. 

Alex, if you want a unique link for each email, let me know. Otherwise, they're all using the same link. jpetto, let me know if you need help testing.
Flags: needinfo?(jon)
(Assignee)

Comment 19

3 years ago
More testing is always good. :D

The updated form is on demo3 [1]. Pay no attention to the visual issues on that page - there are many branches smashed together on demo3 at the moment.

[1] https://www-demo3.allizom.org/firefox/android/
Flags: needinfo?(jon)
(Assignee)

Comment 20

3 years ago
Checking back on this bug - are we ready to merge the changes?

Comment 21

3 years ago
Jpetto: 

I am not sure how to test whether language works since the back-end appears to auto-select language based on my browser settings (i.e. switching the form to /fr-fr/ still sends an EN message). 

Do we have a way (like VPNs from around the world maybe?) to test whether the language triggering works? 

Other than that, I'm fine. Never heard back from adavis on whether the links are what he wants them to be.
(Reporter)

Comment 22

3 years ago
Catching up on all these bugs. 2 weeks of honeymoon and 1 week of Mozlando really makes this hard to jump back into.

(In reply to Ben Niolet from comment #18)
> Alex, if you want a unique link for each email, let me know. Otherwise,
> they're all using the same link. jpetto, let me know if you need help
> testing.

The English template should already have a tracking URL. Don't worry about the other languages. Nothing prevents me down the road to add a tracking URL to other languages. Let's keep it simple for now.

Let me know if there's anything else I've missed and that needs my attention. You guys seem to be covering all angles.
Flags: needinfo?(adavis)
(Assignee)

Comment 23

3 years ago
(In reply to Ben Niolet from comment #18)
> Email triggers are live and should be working for: 
> 
> es, de, pl, pt-br, ru, id, fr. 
> 
> HTML only. 

Hm, according to the data [1], the 'get-android-embed' message is only set for 'en'. I *think* that's the issue here, as visiting a supported URL [2] should send the correct message.

:bniolet - Can you double check basket to make sure the new message is enabled for all necessary locales?

[1] https://basket.mozilla.org/news/newsletters/
[2] https://www-demo3.allizom.org/de/firefox/android/
Flags: needinfo?(bniolet)

Comment 24

3 years ago
ah, yes, user error on my part. 

Which instance of basket is demo3 pointed at? 

Dev, Stage or Prod?
Flags: needinfo?(bniolet)

Comment 25

3 years ago
^
Flags: needinfo?(jon)
(Assignee)

Comment 26

3 years ago
:pmac checked last night and said demo3 is pointed at prod basket.
Flags: needinfo?(jon)

Comment 27

3 years ago
it works!!!

fixed two errors in how I set up basket. all langs are live now and tested!!!

Updated

3 years ago
Flags: needinfo?(jon)
(Assignee)

Comment 28

3 years ago
Hurray! We'll shoot to get this out on Monday.
Flags: needinfo?(jon)

Comment 29

3 years ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/929f1238dc68b17bc37802ff4f4823ab8340c6ec
[fix bug 1221328] Update /firefox/android/ send to device widget.

https://github.com/mozilla/bedrock/commit/149b036c8fc22c9d71f6afe7a8921f2decf65cc3
Merge pull request #3534 from jpetto/bug-1221328-update-android-send-to-device

[fix bug 1221328] Update /firefox/android/ send to device widget.

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.