1198516 - Implement Send to Device user testing variations on Android product page

Status: RESOLVED FIXED

(www.mozilla.org :: General, defect)

Description

[Holly Habstritt Gaal [:Habber]]

Test objective: 
Compare engagement, conversion and user sentiment between the Send to Device form and Google Play Store button on Android product page. We will be using usertesting.com for qualitative and GA for quantitative testing. A unique URL will be needed for each test variation.

Android product page:
https://www.mozilla.org/en-US/firefox/android/

Users:
en-us, desktop users only

3 variations:
- Control (current Android product page layout, but will need unique URL so that Send to Device form can send email specific to this test)

- v1: Embedded Send to Device Form http://cl.ly/image/2Q1v0Z1u1D0v (w/ updates to mockup as discussed in meeting - embed form in top area of page, to the left of product images)

- v2: Google Play Store button - http://cl.ly/image/1B2J2o3v0n3N


Further details and notes:
https://docs.google.com/document/d/1ja5R1CBAVmIQe59vMT-hzngI4l7UmEzyPLB9PNcaCF8/edit#

Comment 1

[Alex Davis [:adavis]]

I've started the detail report (recipe) of for this A/B test. It's still a work in progress but I've attached the link for future reference.

Comment 2

[Alex Davis [:adavis]]

Update: Email templates and triggers were created (yay, no more error messages!). 

BenN will be sending us new basket ids for the email and sms messages for both variations.

Reminder: It's important we use these basket ids because they each have their own Adjust tracking urls to measure the performance of both variations on the product page.

Comment 3

[Jonathan Petto [:jpetto]]

For the B/C variations (embedded send to device widget, Google Play button), what should be done with the "Get Firefox for Android" buttons in the sticky family nav and footer? Both of these buttons open the send to device modal. Should they remain unchanged?

Comment 4

[Holly Habstritt Gaal [:Habber]]

(In reply to Jon Petto [:jpetto] from comment #3)
> For the B/C variations (embedded send to device widget, Google Play button),
> what should be done with the "Get Firefox for Android" buttons in the sticky
> family nav and footer? Both of these buttons open the send to device modal.
> Should they remain unchanged?

Ideally the buttons in the sticky nav and footer would reflect the method being focused on in the test. (all CTAs on page point to play store or all CTAs point to sned to device form)

Are the header, sticky nav and footer buttons tracked separately and uniquely in GA?  


If the buttons are being tracked separately/uniquely, this is a test, so if it takes significant time to implement, it isn't a blocker. The main thing is that we can uniquely track the button clicks across each variation in the header part of the page and that the result of those clicks aren't combined with any of the similar actions on the page (ie. in sticky and footer).

Comment 5

[Jonathan Petto [:jpetto]]

Hm, I don't think tracking will be an issue. I believe we can fit the Google Play button in the header and footer, but my concern is with the embedded send to device form. What would the header and footer buttons do? Scroll the page to the form?

Comment 6

[Alex Davis [:adavis]]

(In reply to Jon Petto [:jpetto] from comment #5)
> Hm, I don't think tracking will be an issue. I believe we can fit the Google
> Play button in the header and footer, but my concern is with the embedded
> send to device form. What would the header and footer buttons do? Scroll the
> page to the form?

I like that proposal. Scrolling the page to the form sounds pretty simple and efficient.

Comment 7

[Holly Habstritt Gaal [:Habber]]

(In reply to Alex Davis [:adavis] from comment #6)
> (In reply to Jon Petto [:jpetto] from comment #5)
> > Hm, I don't think tracking will be an issue. I believe we can fit the Google
> > Play button in the header and footer, but my concern is with the embedded
> > send to device form. What would the header and footer buttons do? Scroll the
> > page to the form?
> 
> I like that proposal. Scrolling the page to the form sounds pretty simple
> and efficient.

+1

Comment 8

[Jonathan Petto [:jpetto]]

(In reply to Alex Davis [:adavis] from comment #2)
> Update: Email templates and triggers were created (yay, no more error
> messages!). 
> 
> BenN will be sending us new basket ids for the email and sms messages for
> both variations.
> 
> Reminder: It's important we use these basket ids because they each have
> their own Adjust tracking urls to measure the performance of both variations
> on the product page.

Do we have the new basket ids yet?

Comment 9

[Alex Davis [:adavis]]

(In reply to Jon Petto [:jpetto] from comment #8)
> (In reply to Alex Davis [:adavis] from comment #2)
> > Update: Email templates and triggers were created (yay, no more error
> > messages!). 
> > 
> > BenN will be sending us new basket ids for the email and sms messages for
> > both variations.
> > 
> > Reminder: It's important we use these basket ids because they each have
> > their own Adjust tracking urls to measure the performance of both variations
> > on the product page.
> 
> Do we have the new basket ids yet?

I got them from Ben this morning. Apparently there was a basket outage so it took longer than planned:

Embedded version
Email: get-android-embed
SMS: android-download-embed

Not embedded version
Email: get-android-notembed
SMS: android-download-notembed

Comment 10

[Jonathan Petto [:jpetto]]

Excellent! The new basket ids are in.

In terms of the Google Play button variation (calling this one variation C), do we need custom/alternate tracking from the default Google Play button, both for Google Play store link [1] and GTM [2]? 

[1] https://play.google.com/store/apps/details?id=org.mozilla.firefox&referrer=utm_source%3Dmozilla%26utm_medium%3DReferral%26utm_campaign%3Dmozilla-org
[2] data-button-type="download" data-interaction="download click" data-download-version="Firefox for Android"

Comment 11

[Alex Davis [:adavis]]

(In reply to Jon Petto [:jpetto] from comment #10)
> Excellent! The new basket ids are in.
> 
> In terms of the Google Play button variation (calling this one variation C),
> do we need custom/alternate tracking from the default Google Play button,
> both for Google Play store link [1] and GTM [2]? 
> 
> [1]
> https://play.google.com/store/apps/details?id=org.mozilla.
> firefox&referrer=utm_source%3Dmozilla%26utm_medium%3DReferral%26utm_campaign%
> 3Dmozilla-org
> [2] data-button-type="download" data-interaction="download click"
> data-download-version="Firefox for Android"

Since the test is targeting only desktop users, there is no special GP link. Google doesn't track conversions through the web version of the Google Play store. (or they don't provide it to people)

As per GTM, we should be good since each variation is on its own page.
-jpetto, just confirm that each variation is on its own page.
-Garethc, once jpetto confirms, can you also confirm that GTM should be ok to track clicks to Google Play store. I'm presuming that they will have the same event names and we will just filter the events by page.

Comment 12

[Jonathan Petto [:jpetto]]

The branch has been put on demo3:

https://www-demo3.allizom.org/en-US/firefox/android/ (original)
https://www-demo3.allizom.org/en-US/firefox/android/?v=a (same as original, with updated basket ids)
https://www-demo3.allizom.org/en-US/firefox/android/?v=b (embedded widget, updated basket ids)
https://www-demo3.allizom.org/en-US/firefox/android/?v=c (google play button/no widget)

Comment 13

[Holly Habstritt Gaal [:Habber]]

Thanks, Jon. 

Glad the embedded form was able to fit in the header in version B. Looks good. Would you mind reducing the height of the "Go to Google Play" area? http://cl.ly/image/0k0V3N453H0x It starts to compete with the main Send CTA.

For version C could you grab the Google Play button that is one size larger so that it is closer in prominence to our Send CTA in the other variations? https://developer.android.com/distribute/tools/promote/badges.html

Are these the URLs that I can send users to in the user tests?

Comment 14

[Alex Davis [:adavis]]

(In reply to Jon Petto [:jpetto] from comment #12)
> The branch has been put on demo3:
> 
> https://www-demo3.allizom.org/en-US/firefox/android/ (original)
> https://www-demo3.allizom.org/en-US/firefox/android/?v=a (same as original,
> with updated basket ids)
> https://www-demo3.allizom.org/en-US/firefox/android/?v=b (embedded widget,
> updated basket ids)
> https://www-demo3.allizom.org/en-US/firefox/android/?v=c (google play
> button/no widget)

Hey Jon, 
1) we're getting error with the send to device form. Is this normal in demo3?

2) As long as you keep ?v=X in the URL Gareth and I confirm that the GA works for the GPlay buttons. 

3) Due to the form error, we can't double check tracking for "Send to device" since it's setup to fire on submit.

Comment 15

[Jonathan Petto [:jpetto]]

Apparently the demo servers are pointed to a dev instance of basket, which did not have the new ids setup. pmac lent a hand and pointed demo3 to the prod instance of basket, and it now appears that demo3 is working.

Give it another try?

Comment 16

[Alex Davis [:adavis]]

(In reply to Alex Davis [:adavis] from comment #14)
> (In reply to Jon Petto [:jpetto] from comment #12)
> > The branch has been put on demo3:
> > 
> > https://www-demo3.allizom.org/en-US/firefox/android/ (original)
> > https://www-demo3.allizom.org/en-US/firefox/android/?v=a (same as original,
> > with updated basket ids)
> > https://www-demo3.allizom.org/en-US/firefox/android/?v=b (embedded widget,
> > updated basket ids)
> > https://www-demo3.allizom.org/en-US/firefox/android/?v=c (google play
> > button/no widget)
> 
> Hey Jon, 
> 1) we're getting error with the send to device form. Is this normal in demo3?
> 
> 2) As long as you keep ?v=X in the URL Gareth and I confirm that the GA
> works for the GPlay buttons. 
> 
> 3) Due to the form error, we can't double check tracking for "Send to
> device" since it's setup to fire on submit.

Gareth,
the send to device now works. (I confirm)

Can you double check the tracking in GA?

Comment 17

[Gareth Cull [:garethc]]

I can see the send to device event coming through in GA: http://cl.ly/image/110z1j3r3k3L

Comment 18

[Jonathan Petto [:jpetto]]

demo3 has just been updated. See below for details:

(In reply to Holly Habstritt Gaal [:Habber] from comment #13)
> Thanks, Jon. 
> 
> Glad the embedded form was able to fit in the header in version B. Looks
> good. Would you mind reducing the height of the "Go to Google Play" area?
> http://cl.ly/image/0k0V3N453H0x It starts to compete with the main Send CTA.

I've dialed down the prominence of the "Go to Google Play" link for ?v=b. Let me know if you're happy with the size. We also have the option of hiding that link entirely.

> For version C could you grab the Google Play button that is one size larger
> so that it is closer in prominence to our Send CTA in the other variations?
> https://developer.android.com/distribute/tools/promote/badges.html

For ?v=c, I've increased the size of the Google Play button from 129 x 45 to 200 x 69. Looking better? Too big?

> Are these the URLs that I can send users to in the user tests?

As long as :adavis and :garethc are good with the URLs as they are on demo3 (and it sounds like they are), then these will be the final URLs.

Comment 19

[Holly Habstritt Gaal [:Habber]]

(In reply to Jon Petto [:jpetto] from comment #18)
> demo3 has just been updated. See below for details:
> 
> (In reply to Holly Habstritt Gaal [:Habber] from comment #13)
> > Thanks, Jon. 
> > 
> > Glad the embedded form was able to fit in the header in version B. Looks
> > good. Would you mind reducing the height of the "Go to Google Play" area?
> > http://cl.ly/image/0k0V3N453H0x It starts to compete with the main Send CTA.
> 
> I've dialed down the prominence of the "Go to Google Play" link for ?v=b.
> Let me know if you're happy with the size. We also have the option of hiding
> that link entirely.
> 
> > For version C could you grab the Google Play button that is one size larger
> > so that it is closer in prominence to our Send CTA in the other variations?
> > https://developer.android.com/distribute/tools/promote/badges.html
> 
> For ?v=c, I've increased the size of the Google Play button from 129 x 45 to
> 200 x 69. Looking better? Too big?
> 
> > Are these the URLs that I can send users to in the user tests?
> 
> As long as :adavis and :garethc are good with the URLs as they are on demo3
> (and it sounds like they are), then these will be the final URLs.

Looks good to me. Thanks!

Comment 20

[Jennifer Bertsch [:jbertsch]]

(In reply to Holly Habstritt Gaal [:Habber] from comment #19)
> (In reply to Jon Petto [:jpetto] from comment #18)
> > demo3 has just been updated. See below for details:
> > 
> > (In reply to Holly Habstritt Gaal [:Habber] from comment #13)
> > > Thanks, Jon. 
> > > 
> > > Glad the embedded form was able to fit in the header in version B. Looks
> > > good. Would you mind reducing the height of the "Go to Google Play" area?
> > > http://cl.ly/image/0k0V3N453H0x It starts to compete with the main Send CTA.
> > 
> > I've dialed down the prominence of the "Go to Google Play" link for ?v=b.
> > Let me know if you're happy with the size. We also have the option of hiding
> > that link entirely.
> > 
> > > For version C could you grab the Google Play button that is one size larger
> > > so that it is closer in prominence to our Send CTA in the other variations?
> > > https://developer.android.com/distribute/tools/promote/badges.html
> > 
> > For ?v=c, I've increased the size of the Google Play button from 129 x 45 to
> > 200 x 69. Looking better? Too big?
> > 
> > > Are these the URLs that I can send users to in the user tests?
> > 
> > As long as :adavis and :garethc are good with the URLs as they are on demo3
> > (and it sounds like they are), then these will be the final URLs.
> 
> Looks good to me. Thanks!

+1

Comment 21

[Jonathan Petto [:jpetto]]

Sounds like we're good for a PR here. I'll open one today.

Comment 22

[Jonathan Petto [:jpetto]]

Attachment: GitHub PR

Comment 23

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/dfda5d359f4185baf08d6de265a45c385e9729c8
[fix bug 1198516] Implement send to device tests on /firefox/android

https://github.com/mozilla/bedrock/commit/8cc0721351c228cc84c8f63ca684a5bbe83ab5ae
Merge pull request #3291 from mozilla/bug-1198516-send-to-device-testing-fx-android

[fix bug 1198516] Implement send to device tests on /firefox/android

Comment 24

[Alex Davis [:adavis]]

Hi Jon,

I was going over the performance of each variation and noticed that I'm seeing installs for:

Embedded:
-SMS
-Email

Pop-up:
-SMS

The volume is high enough now that we should have gotten email installs through email.

I double checked the email templates and their tracking URLs. I then tested sending myself messages.
I did not get the email from https://www.mozilla.org/en-US/firefox/android/?v=b .

Please double check that everything looks good.
Thanks

Comment 25

[Jonathan Petto [:jpetto]]

Hm, I just ran a test for ?v=b and successfully received emails at my personal, business, and school addresses. However, I submitted my mozilla.com address multiple times, but never received an email at that address.

I forwarded the email sent to my school address over to you.

No errors are being thrown as far as I can tell. Do you have an alternate email to test against to verify my results?

Comment 26

[Alex Davis [:adavis]]

(In reply to Jon Petto [:jpetto] from comment #25)
> Hm, I just ran a test for ?v=b and successfully received emails at my
> personal, business, and school addresses. However, I submitted my
> mozilla.com address multiple times, but never received an email at that
> address.
> 
> I forwarded the email sent to my school address over to you.
> 
> No errors are being thrown as far as I can tell. Do you have an alternate
> email to test against to verify my results?

Thanks for forwarding the email! I'm glad you got it.

I noticed that the URL in it was the URL from the embedded form (not the pop-up). This means that variation A & B are sending the same emails with the same tracking URL.

Can you check to see if the baskets are properly setup? (see comment 9) I can already confirm that the URLs in the email templates are in fact different. So my guess right now is something around baskets is wrong.

If the baskets look good on your end, we'll have to check with BNiolet to see if they are properly mapped out on his end.

Thanks!

Comment 27

[Alex Davis [:adavis]]

It was resolved over IRC. Closing ticket.