Open Bug 1350907 Opened 3 years ago Updated 3 years ago

Validate telemetry experiment annotations using SHIELD

Categories

(Toolkit :: Telemetry, enhancement, P3)

enhancement

Tracking

()

Tracking Status
firefox55 --- affected

People

(Reporter: Dexter, Assigned: glind)

References

Details

(Whiteboard: [measurement:client])

User Story

Before PUBLICIZING the existence of these new TelemetryEnvironment api's, an EXPERIMENT DEPLOYER should have confidence that they work such that the CEP / ETL / data platform can see and collect the packets.
We should validate the Telemetry experiment annotation using a limited experiment, in order to make sure that the API works and that the data we collect looks good.
Depends on: 1348748
Priority: -- → P1
Whiteboard: [measurement:client]
Experiments plan for evaluation:


Experiment 1 :: Really do nothing, test collection and sampling

- in NIGHTYLY|AURORA|BETA (as far as uplifted),  ALL LOCALES
- sample the 'smallest' sample O(100)
- for a NON-EXISTENT PREF
- set it to (null) in both branches
- VERIFY
  - data collection


Experiment 2 :: Test Value Setting

- in NIGHTYLY|AURORA|BETA (as far as uplifted),  ALL LOCALES
- sample the 'smallest' sample O(100)
- for a NON-EXISTENT PREF
* NEW!  set it to ARBITRARY VALUES in both branches
- VERIFY
  - data collection
  - settings
User Story: (updated)
Assignee: nobody → glind
This should be available in all pre-release channels, up to Beta 53 from Thu, 3/30.
Flags: needinfo?(glind)
Priority: P1 → P2
Per:  https://github.com/mozilla/normandy/issues/734

We are doing a test as soon as release-drivers says yes, which we expect by monday 8 May.
Flags: needinfo?(glind)
Can you confirm that the first round of experiment will be on Nightly55 only?
:Ritu, updated test plan below

1. I confirm this is in Nightly55 only.  If the test is successful, we can discuss next test in beta and release.
2. This pref doesn't exist.  Normandy can set prefs that do not already exist.  There is No Need for any additional patching or uplift to the tree to do this test.

Updated test plan (replaces comment 1):

----

see:  https://github.com/mozilla/normandy/issues/734

Description: After POSITIVE OKAY ("explicit yes") from RELEASE-DRIVERS we intend to ship a TEST of the Shield (Normandy) Pref-flip experiments system.

Population Affected: 1% of Nightly Channel (only), all locales and user configurations.

Expected Effect:  the default pref  ` test.normandy.prefflip.defaultpref` will be set to one of 3 values [0,1,2], with unequal probability.  This preference has no known user facing effect.  This preference doesn't otherwise exist.

Cleanup:  after we determine data is coming in correctly (within 2 days), will close/end the test.  Closing the test will reset the preference.  Within 24 hours (when profiles next contact the server), all running profiles should be reset.

Goals:

    Per  https://bugzilla.mozilla.org/show_bug.cgi?id=1350907, CONFIRM that experiment annotations are working in a real population
    Per https://github.com/mozilla/normandy/issues/734, CONFIRM that these work
        sampling ratio
        pref-flip experiments
        rollback/cancellation
    Confirm/iterate on template language for notifying Release-Drivers about pref-flips.


Uplift needed:  None.  Pref doesn't exist in tree, which is okay with Shield/Normandy. 'experiment tagging' code is already in release.  Shield updates as a system addon.

GL
Fx S+I team.
Flags: needinfo?(rkothari)
The plan in comment 5 looks good. Let's do it!
Flags: needinfo?(rkothari)
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.