1444366 - [Shield] The "TAAR Experiment V2" add-on can wrongly be installed on profiles younger than 2 or older than 9 days

Status: RESOLVED WONTFIX

(Shield :: Shield Study, defect)

Description

[Marius Coman [:mcoman], Ecosystem QA]

Attachment: rec of the issue.gif

[Affected Versions]:
- Firefox Beta (59.0) Build ID:20180307211018

[Affected Platforms]:
- All Windows
- All Mac
- All Linux

[Prerequisites]:
- Have a new Firefox profile.
- Have the following prefs. in the "about:config" page (If a preference does not exist, please create it):
	- “extensions.legacy.enabled” pref set to “true”;
	- “shield.test.variation” pref set to “linear-taar”;
	- “extensions.taarexpv2.profile-age-in-days-test-override” pref set to any value lower than 2 or higher than 9.

[Steps to reproduce]:
1. Open the browser with the profile from prerequisites.
2. Install the latest "TAAR Experiment" (v1.0.13) from https://goo.gl/mg6ubo.
3. Navigate to the "about:addons>Extensions" and observe the dispalyed add-ons.

[Expected result]:
- There are no add-ons displayed.

[Actual result]:
- The "TAAR Experiment V2" add-on is displayed.

[Notes]:
- The install pings are also displayed in the browser console and in the "about:telemetry" page.
- Attached a screen recording of the issue.

Comment 1

[mlopatka]

This is expected behavior purposefully introduced in order to allow the study to target a wider cohort in light of launch delays. 

We will now rely on The SHIELD [Normandy](https://github.com/mozilla/normandy) targeting for study enrollment.

The changes have been commented in the repo for transparency, [here](https://github.com/motin/taar-experiment-v2-shield-study/blob/9bb04008e0c70b03c49db5d093356f903ef50840/addon/Config.jsm#L140)

And the study design document has been updated to reflect these changes.

Comment 2

[mlopatka]

Also please note, profile age related test cases have been removed from [TESTPLAN.md](https://github.com/motin/taar-experiment-v2-shield-study/blob/develop/TESTPLAN.md) to reflect expected functionality of the latest version.