1316161 - [SHIELD] Study Validation Review for [Contextual notifications test]

Status: RESOLVED INVALID

(Shield :: Shield Study, defect)

Description

[Fabio Rios [:frios]]

The purpose of this test is to understand if offering mobile versions of Firefox to desktop users in a more timely and relevant manner will increase installs.

Test plan with more detail here: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=63931297

Comment 1

[Fabio Rios [:frios]]

Hey Gregg, trying to follow instructions on this sheet: https://docs.google.com/document/d/1xNpc7GuyJ4gwXhf7NaG5BW0BThCCtXjp-3ATIpJAny4/edit#

Requesting f+

Comment 2

[Chris More [:cmore]]

gregglind: any update here?

Comment 3

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

F+, with these caveats, as discussed in the previous meeting on this:

1.  Bookmarking Threshold too high.   I think 5 bookmarks is too many.  I think 1 bookmark is plenty.  I would even prefer to see it "when bookmarking", but I know that might be a technical challenge.  [This is a hunch.  I would accept counterevidence from UT]. 

2.  Packet should including 'which' notification it is.  The immediate/24/48-ness of the message should be part of what is recorded about that offer.  I want to know the success of the immediate message, the 24 conditional on not having taken the immediate, etc.

In simpler terms, there should be a field in the recorded packet saying which message timing the growl had, for analysis.  I am most interested in uptake from 1st message, and don't want that polluted :)

3.  Control group.  There is an assumption that conversion for mobile is essential 0 without this.  I am actually OK with this being uncontrolled.  However, I would like to see some measure of the negative impact of the Growl / notification.  We have estimates of 8:1 ratio of like / dislike of notifications from firefox from other studies.  Ideas very welcome.


Matt_G, if you have other comments, they are welcome.

Comment 4

[Chris More [:cmore]]

(In reply to Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat ) from comment #3)
> F+, with these caveats, as discussed in the previous meeting on this:
> 
> 1.  Bookmarking Threshold too high.   I think 5 bookmarks is too many.  I
> think 1 bookmark is plenty.  I would even prefer to see it "when
> bookmarking", but I know that might be a technical challenge.  [This is a
> hunch.  I would accept counterevidence from UT]. 

What if we did a few different arms? A control, and two variations. One variation could be set to 1 bookmark and another variation set to 5. That would help understand the impact. The volume and conversion may could likely be different for each threshold. 

Also, we were unable to find a way via the add-on to listen for exactly when the bookmark is done. We inquired and heard that it is not technically possible now, but in 2017 event tracking will land in Firefox to allow this and more. The counting was only done because the event tracking wasn't available. Ideally, the notification would happen immediately and we didn't want to count every second and keep take up some cpu cycles.

> 
> 2.  Packet should including 'which' notification it is.  The
> immediate/24/48-ness of the message should be part of what is recorded about
> that offer.  I want to know the success of the immediate message, the 24
> conditional on not having taken the immediate, etc.

Agree. :espressive: can you add a &utm_content=notification-X where X which if the 3 notification they took to the URL to the page?

What's the current URL you are using?

> 
> In simpler terms, there should be a field in the recorded packet saying
> which message timing the growl had, for analysis.  I am most interested in
> uptake from 1st message, and don't want that polluted :)
> 
> 3.  Control group.  There is an assumption that conversion for mobile is
> essential 0 without this.  I am actually OK with this being uncontrolled. 
> However, I would like to see some measure of the negative impact of the
> Growl / notification.  We have estimates of 8:1 ratio of like / dislike of
> notifications from firefox from other studies.  Ideas very welcome.
> 

Yeah, the control is not the best because you can organically convert. We are just mostly interested in the overall conversion rate to taking the notification when compared to the population of the experiment arm. It's not a great A/B test given the control is not really controlled.

> 
> Matt_G, if you have other comments, they are welcome.

Comment 5

[Schalk Neethling [:espressive]]

(In reply to Chris More [:cmore] from comment #4)
> (In reply to Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )
> from comment #3)
> > F+, with these caveats, as discussed in the previous meeting on this:
> > 
> > 1.  Bookmarking Threshold too high.   I think 5 bookmarks is too many.  I
> > think 1 bookmark is plenty.  I would even prefer to see it "when
> > bookmarking", but I know that might be a technical challenge.  [This is a
> > hunch.  I would accept counterevidence from UT]. 
> 
> What if we did a few different arms? A control, and two variations. One
> variation could be set to 1 bookmark and another variation set to 5. That
> would help understand the impact. The volume and conversion may could likely
> be different for each threshold. 
> 
> Also, we were unable to find a way via the add-on to listen for exactly when
> the bookmark is done. We inquired and heard that it is not technically
> possible now, but in 2017 event tracking will land in Firefox to allow this
> and more. The counting was only done because the event tracking wasn't
> available. Ideally, the notification would happen immediately and we didn't
> want to count every second and keep take up some cpu cycles.
> 
> > 
> > 2.  Packet should including 'which' notification it is.  The
> > immediate/24/48-ness of the message should be part of what is recorded about
> > that offer.  I want to know the success of the immediate message, the 24
> > conditional on not having taken the immediate, etc.
> 
> Agree. :espressive: can you add a &utm_content=notification-X where X which
> if the 3 notification they took to the URL to the page?

Will do.

> 
> What's the current URL you are using?

https://www.mozilla.org/firefox/sync/?utm_source=get-sycned-addon&utm_medium=firefox-browser&utm_campaign=get-synced-v-10

Comment 6

[Schalk Neethling [:espressive]]

This is what the URL will look like now, after adding the variation info:

https://www.mozilla.org/en-US/firefox/sync/?utm_campaign=get-synced-v-10&utm_content=notification-3&utm_medium=firefox-browser&utm_source=get-sycned-addon

Comment 7

[Schalk Neethling [:espressive]]

Attachment: @get-synced-1.0.0.xpi

Here is the XPI to upload to AMO

Comment 8

[Chris More [:cmore]]

Gregg: does the XPI have to uploaded to the AMO site as a listed add-on? There's two type of add-ons: self-signed that I can do right now that is not listed on AMO and then listed, which is reviewed by the AMO team and then posted to addons.mozilla.org. Which one do we need to do for this experiment?

Comment 9

[Matt Grimes [:Matt_G]]

At this time it needs to be a publicly listed add-on on AMO. This is for technical reasons that we can explain later, but it's a requirement right now. 

I might have missed it in the previous meetings/docs, but are you taking advantage of the built in survey tools provided by Shield Studies? This allows you to automatically launch a survey when a user leaves the study or when the experiment expires. I would argue that these attitudinal measures can be as important as any behavioral data you'd get from the study.

Comment 10

[Chris More [:cmore]]

I've uploaded the XPI to AMO for approval.

Comment 11

[Chris More [:cmore]]

Attachment: get_synced-1.0.0-fx-windows.xpi (xpi uploaded to amo)

AMO listing: https://addons.mozilla.org/en-US/firefox/addon/get-synced-v1-0/

gregg: it's uploaded to AMO and approved. 

What's the next steps to launch the experiment?

Comment 12

[Matt Grimes [:Matt_G]]

@cmore - Take a look at the Shield study documentation. You've got a few more steps and bugs to file. 

1. We still need a data review bug for rweiss. Rather than posting a sample data packet, you'll need to post an actual packet retrieved from your QA. We've had issues in the past of sample packets not matching actual packets. 

2. At then end of every study, we do a debrief with participants. This captures attitudinal changes between cohorts. This will tell you if people found the experiment creepy, annoying, beneficial etc. You can either create these yourself or we have resources on my team that can help. The debriefs need to be reviewed by legal.

3. You'll also need a consent page that will also go in the legal bug. If you want help with the consent page, let me know. Once the consent page has been approved you'll have to check it in to the AMO repo. Check in must happen on a Monday for the page to be live on a Thurs. 

4. At that point you'd file your launch bug and we'd be ready to go live. We're no longer launching on Thurs since it is so close to the weekend, so you'd be looking at the following Monday. 

Step 3 above will be significantly improved once the system add-on lands, so you'll save a great deal of time in future studies.

Comment 13

[Chris More [:cmore]]

Hey Matt.

This is starting to feel like overkill for a very small experiment that has no additional data collection and just triggers a desktop notification that links to a www.mozilla.org page and does nothing else. 

Can your team handle the logistics (bugs, pages, etc,) of the experiment given that we built the add-on?

We have coordinated these things already:

"You (the Reader) are doing / coordinating these aspects:

    ui design variations
    engineering the addon
    testing / qa of addon"

Thank you.

Comment 14

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

We agree that all this machinery is some work, and we want it to be easier in the future, especially for small studies that are low risk.  We aren't quite there yet, and thus consent and risks are needful.  We are also quite slammed with studies, but we can help land it.

At a minimum we need 
- text for the consent page describing your experiment and risks (copy this from a previous study)
- and that text needs to be approved by legal to allow recruiting to begin
- the consent page needs to be hosted at AMO.  

I will personally own landing that text at amo.

Not sure of what your specific blockers are on this, but email us and we will sort them out.

Comment 15

[Chris More [:cmore]]

Thanks, Gregg. I had a Q4 2016 OKR to get this experiment launched and it looks like we are going to miss it now. Is anyone in next week at all to get this add-on closer to launching?

No blockers from us other than needing help getting this experiment launched. Thanks

Comment 16

[Chris More [:cmore]]

(In reply to Chris More [:cmore] from comment #15)
> Thanks, Gregg. I had a Q4 2016 OKR to get this experiment launched and it
> looks like we are going to miss it now. Is anyone in next week at all to get
> this add-on closer to launching?
> 
> No blockers from us other than needing help getting this experiment
> launched. Thanks

Gregg: Any update here? We need to get this experiment launched, so whatever we can learn can be applied to something better going forward.

Comment 17

[Chris More [:cmore]]

(p.s. Happy New Year!)

Comment 18

[Schalk Neethling [:espressive]]

I wonder if perhaps Shield is a bit much for what we are trying to accomplish. We are not taking advantage of much of what shield provides so, perhaps we can have three funnelcake builds with the different bookmark counts and push that out as we do All Aboard.

Thoughts?

Comment 19

[Chris More [:cmore]]

Good question. At this point, 3 funnelcakes would been easier and we could have had results back by now, but I thought that Shield would be a perfect platform to do experiments like this. Not sure if it is or isn't a good fit. Let's see what Gregg/Matt have to say and next steps.

Comment 20

[Matt Grimes [:Matt_G]]

Hey Chris. Sorry for the delay. Let's set up some time to chat this week and we can get you launched. Will be easier than hashing everything out over bugzilla.

Comment 21

[Chris More [:cmore]]

(In reply to Matt Grimes [:Matt_G] from comment #20)
> Hey Chris. Sorry for the delay. Let's set up some time to chat this week and
> we can get you launched. Will be easier than hashing everything out over
> bugzilla.

I will send an invite for Friday AM.

Can you all suggest some copy for the AMO page that has been used before? The only information collected with this study is what Shield does out of the box and whatever Google Analytics already goes on www.mozilla.org (which we already have a privacy policy for). So, the AMO page and getting legal approval should be straightforward. An example or not starting from scratch would be useful.

Comment 22

[Chris More [:cmore]]

templates from Gregg:

https://addons.allizom.org/en-US/firefox/shield_study_8
 
https://addons.allizom.org/en-US/firefox/shield_study_7

Comment 23

[Chris More [:cmore]]

Fabio: here's two templates examples. Let's work on this today.

Actions below

1) Copy this template:

https://docs.google.com/document/d/1AXtaBJQ7NZ7faUmKXASXtRsDZ6OvMaxadQHRUbHzbg4/edit#

Examples in comment 22

2) copy: button copy, sentence up top, and basically all copy edits down to "your privacy".
3) AMO: ensure all platforms, check URL
4) Privacy: needinfo Elvin after the new doc is written.

Comment 24

[Chris More [:cmore]]

(and names at very bottom too)

Comment 25

[Chris More [:cmore]]

TBD if we will do a post-survey

Comment 26

[Chris More [:cmore]]

Schedule a tech review early next week with Gregg/Schalk.

Comment 27

[Chris More [:cmore]]

Here is what Fabio came up with for the AMO page:

https://docs.google.com/document/d/1_lmFxOgjR_jT1jYUq-PVsVvPu0FuaLlEM605i8EXB-8/edit#

Gregg/Matt: Check it out and please provide comments within the doc.

Thanks!

Comment 28

[Matt Grimes [:Matt_G]]

Looks good. I made some comments in the doc and filed the legal bug. Review should be quick as very little data is being collected.

Comment 29

[Chris More [:cmore]]

(In reply to Matt Grimes [:Matt_G] from comment #28)
> Looks good. I made some comments in the doc and filed the legal bug. Review
> should be quick as very little data is being collected.

Any update here?

Comment 30

[Chris More [:cmore]]

This bug is no longer valid and I am going to close it.