1523781 - [SHIELD] Pref-Flip Study: Mid-65 Default-On Cookie Restrictions for Existing Users, v65, release

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[:shell escalante]

On 2019/2/11 we plan to study the effect of default-on cookie restrictions for 1.5% of existing users (plus 1.5% control). The ad-click probe and churn will be monitored during the rollout to inform a 2019/3/11 decision whether to scale up to 100% of users.

more information: https://experimenter.services.mozilla.com/experiments/mid-65-default-on-cookie-restrictions-for-existing-users/

Comment 1

[Tony Cinotto [:tcinotto] (UTC-8)]

Hi Robert, I just wanted to see if you think this experiment needed Data Steward sign off?

Comment 2

[Robert (rrayborn, he/him)]

Hi Tony,

I assume that the reference to ad clicks is with regards to this previous implementation bug? https://bugzilla.mozilla.org/show_bug.cgi?id=1495548

If we're referencing a previous probe and this experiment doesn't somehow modify the behavior of that probe then there's no need for additional data review. We can use existing probes for branch analysis without issue.

If this is incorrect please flag me and provide a description and link to any relevant bugs/docs.

Comment 3

[Tony Cinotto [:tcinotto] (UTC-8)]

(In reply to Rob from comment #2)

Hi Tony,

I assume that the reference to ad clicks is with regards to this previous implementation bug? https://bugzilla.mozilla.org/show_bug.cgi?id=1495548

If we're referencing a previous probe and this experiment doesn't somehow modify the behavior of that probe then there's no need for additional data review. We can use existing probes for branch analysis without issue.

If this is incorrect please flag me and provide a description and link to any relevant bugs/docs.

Great! I think we should be okay then. Thank you for the confirmation!

Comment 4

[Arthur Edelstein [:arthur]]

[Tracking Requested - why for this release]: We are pushing out this pref-flip study on 2019/2/11 (mid-65).

Comment 5

[Carmen Fat [:cfat] - Ecosystem QA]

Pref-Flip Study: Mid-65 Default-On Cookie Restrictions for New Users
Pref-Flip Study: Mid-65 Default-On Cookie Restrictions for Existing Users
Targeted: Firefox Release 65.x

We have finished testing the "Pref-Flip Study: Mid-65 Default-On Cookie Restrictions for New Users & Existing Users" experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

• We haven’t found any issues during testing.

Testing Summary:

• Full Functional test suite: TestRail (https://tinyurl.com/y9tyv5yl)
• Verified that the "browser.search.ad_clicks" probe registers click on SERP ads.

Tested Platforms:

• Windows 10 x64
• Ubuntu 16.04 x64
• Mac 10.14.2

Tested Firefox versions:

• Firefox Release 65 (en-US, de, fr)

Comment 6

[Josh Gaunt [:jgaunt]]

Recipe 682 has been disabled. This experiment has concluded.