Closed Bug 1532678 Opened 6 years ago Closed 5 years ago

[Shield] Add-On Study: Feature Revenue Impact study for Enhanced Tracking Protection

Categories

(Shield :: Shield Study, enhancement)

Product:
Shield
Component:
Shield Study
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox66- affected)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox66 - affected

People

(Reporter: experimenter, Assigned: mconnor)

Details

Attachments

(2 files, 1 obsolete file)

Final add-on, r+, QA+, for signing
107.98 KB, application/zip
Details
etp-search-volume-study@shield.mozilla.org-1.2-signed.xpi
112.36 KB, application/x-xpinstall
Details

Feature Revenue Impact study for Enhanced Tracking Protection

In Firefox 66 we have a major new feature (Enhanced Tracking Protection) that we believe may impact search revenue. We want to test the revenue impact of this feature to inform a final ship decision for the feature.

Due to concerns about the long term (vs. immediate) revenue impact of the feature, we would like to run this experiment for as long as 180 days to ensure we continue to monitor the impact even if we choose to deploy more broadly. As a result, we need to ensure that it continues to stay live through Firefox 69 at minimum. This may require special handling.

More information: https://experimenter.services.mozilla.com/experiments/feature-revenue-impact-study-for-enhanced-tracking-protection/

Could you review this please?

https://github.com/0c0w3/cookie-restrictions-strict-list-study

Flags: needinfo?(nhnt11)
Flags: needinfo?(mconnor)
Flags: needinfo?(jhofmann)
Flags: needinfo?(ewright)

Erica looked over the repo (thanks!) and I addressed her comments. According to https://wiki.mozilla.org/Firefox/Shield/Shield_Studies, the code needs to be approved by a Firefox peer, so I'll leave the needinfos on Nihanth and Johann. I'll also leave the needinfo on Mike to make sure that it's testing what it's supposed to be testing.

Flags: needinfo?(ewright)

According to that page, we'll need QA and Shield-team approval too. Judging by bug 1522309, it would be better to wait on that until the core review is done (which makes sense imo).

fyi I renamed the repo to properly reflect the study (it had the same name as the earlier study repo I forked): https://github.com/0c0w3/enhanced-tracking-protection-study

The old link above redirects to that one, so it still works too.

I reviewed the commits made on top of the cookie restrictions study. The cookie behavior pref needs to be capitalized properly through variations.json (cookieBehavior vs cookiebehavior) but other than that, r+!

Please double-check the names and values of all the prefs being touched.

Flags: needinfo?(nhnt11)

Whoops, I must have hit Command-L and lowercased it. Thanks for the quick review Nihanth!

I'll package this up into a xpi, post it here, and request that it be signed. Mike looked at the code too offline, so I'll clear his needinfo.

Flags: needinfo?(mconnor)
Flags: needinfo?(jhofmann)

Er, actually Erica asked me to wait for an issue in the cookie study (which I forked) to be resolved before continuing with release, so I'll wait for that before packaging the xpi and attaching it here and requesting signing, QA testing, etc.

More info:

https://github.com/0c0w3/enhanced-tracking-protection-study/commit/2817ca8cca26392a880a1c68b396d3a460d19bb2#commitcomment-32642238

https://github.com/mozilla/cookie-restrictions-strict-list-study/issues/8

A new version of the add-on has been developed that works around several issues we identified between default-branch prefs, multipreffer, and the search service.

This new code lives in https://github.com/mozilla/etp-search-volume-study/

The code has been reviewed by Johann and Erica.

I've uploaded a signed version of the add-on for QA testing as a release, here: https://github.com/mozilla/etp-search-volume-study/releases/tag/v1.0

New release up on GitHub with the category-pref-recomputing code removed. https://github.com/mozilla/etp-search-volume-study/releases/tag/v1.1

It seems that the repository is private, therefore we don't have access to download the build.
Can you please give the Experiments QA team the necessary rights to access the repository?

If the build is not signed with dev key, for us to be able to install it properly, please also attach it here and ask mythmon to sign it.
Thank you.

Flags: needinfo?(nhnt11)

(In reply to Carmen Fat [:carmenf] - Experiments QA from comment #10)

It seems that the repository is private, therefore we don't have access to download the build.
Can you please give the Experiments QA team the necessary rights to access the repository?

If the build is not signed with dev key, for us to be able to install it properly, please also attach it here and ask mythmon to sign it.
Thank you.

Thanks for the heads up, I granted the team access to the repo. Sorry I missed this. The build uploaded in the GitHub release is signed.

Flags: needinfo?(nhnt11)

We have finished testing the Add-On Study: Feature Revenue Impact study for Enhanced Tracking Protection experiment on Firefox 66 release.
Considering the fact that the study will run for 6 months, in order to cover all the targeted Firefox versions we will run our test suite on the latest build of each Beta cycle and will send an individual sign off for the remaining versions.

QA’s recommendation: GREEN - SHIP IT

Reasoning:

  • During testing we have found one issue #6: the study expires after 365 days even if it’s mentioned in the Experimenter page that will expire after 180 days. However, one of the engineers informed us that Normandy will be taking care of the expiry of the addon, so this issue will not affect the study.

Testing Summary:

  • Test Suite: Test Rail;
  • Verified that the created recipe by mythmon for the experiment on the Normandy stage server works as expected.

Tested Platforms:

  • Windows 10 x64;
  • Windows 7 x64;
  • Mac 10.14;
  • Ubuntu 16.04 x64;

Tested Firefox versions:

  • Firefox 66.0.2 (US);
  • Firefox 66.0.2 (DE);
  • Firefox 66.0.2 (PL);
  • Firefox 66.0.2 (FR);

Regards,
Cosmin

Flags: shield-qa+

Untracking since we have this information easily findable now in Experimenter.

tracking-firefox66: + → -

This is the final version of the add-on, ready for signing before deployment.

Attachment #9056242 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)

We have finished testing the Add-On Study: Feature Revenue Impact study for Enhanced Tracking Protection experiment on Firefox 67.0b16.
The study is already live, but since it's running for 6 months, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.

QA’s recommendation: GREEN

Reasoning:

  • No new issues have been found during testing the 1.2 build on Firefox 67.0b16.

Testing Summary:

  • Test Suite: Test Rail;
  • Verified that the created recipe by mythmon for the experiment on the Normandy stage server still works as expected with Fx67.

Tested Platforms:

  • Windows 10 x64;
  • Mac 10.14;
  • Arch Linux 4.14.3 x64;

Tested Firefox versions:

  • Firefox 67.0b16 (US);
  • Firefox 67.0b16 (DE);
  • Firefox 67.0b16 (FR);
  • Firefox 67.0b16 (PL);

We have finished testing the Add-On Study: Feature Revenue Impact study for Enhanced Tracking Protection experiment on Firefox 68.0b13.
The study is already live, but since it's running for 6 months, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.

QA’s recommendation: YELLOW

Reasoning:

  • No new issues have been found during testing the 1.2 build on Firefox 68.0b13.
  • However, while verifying that the created recipe for the experiment works as expected on the Normandy stage server using Fx68, we found a new Webextension issue: Bug 1562212. The "user-disable" and "exit" telemetry pings are no longer sent after manually removing the study from "about:studies" page on Beta 68 and Nightly 69. This issue seems to be a regression of Bug 1549192 that affects all existing and future experiments.

Testing Summary:

  • Test Suite: Test Rail.
  • Verified that the created recipe for the experiment still works as expected on the Normandy stage server and on Fx68.

Tested Platforms:

  • Windows 10 x64;
  • Windows 7 x64;
  • Mac 10.14;
  • Arch Linux 4.14.3 x64;

Tested Firefox versions:

  • Firefox 68.0b13 (US);
  • Firefox 68.0b13 (DE);
  • Firefox 68.0b13 (FR);
  • Firefox 68.0b13 (PL);

We have finished testing the Add-On Study: Feature Revenue Impact study for Enhanced Tracking Protection experiment on Firefox 69.0b12.

The study is already live, but since it's running for 6 months, we are running our test suite on the latest build of each Beta cycle to make sure that there are no new issues between Firefox versions.

QA’s recommendation: YELLOW

Reasoning:

  • One new issue (#5) has been found during testing the 1.2 build on Firefox 69.0b12. But, based on the discussion from the issue, it should not affect the experiment and the study data.
  • However, the YELLOW sign off still stands as for the previous sign off since bug 1562212 is still reproducible on Firefox 69 and affects the "user-disable" and "exit" telemetry pings of the experiment.

Testing Summary:

  • Test Suite: Test Rail.
  • Verified that the created recipe for the experiment still works as expected on the Normandy stage server and on Fx69.

Tested Platforms:

  • Windows 10 x64;
  • Windows 7 x64;
  • Mac 10.14;
  • Arch Linux 4.14.3 x64;

Tested Firefox versions:

  • Firefox 69.0b12 (US);
  • Firefox 69.0b12 (DE);
  • Firefox 69.0b12 (FR);
  • Firefox 69.0b12 (PL);
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: