Closed Bug 1372237 Opened 3 years ago Closed 3 years ago

Better control group for plugin rollout to release

Categories

(Core :: Plug-ins, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla56
Tracking Status
firefox55 + fixed
firefox56 --- fixed

People

(Reporter: benjamin, Assigned: benjamin)

References

Details

Attachments

(1 file)

When we roll out plugin click-to-activate to release (bug 1365714) the code as currently written will divide users up like this:

95% control
5% test

This will make it quite difficult to compare the test and control populations.

What I'd like to do instead is have test and control be the same size:

5% test
5% control
90% excluded

I think a relatively small tweak to the system addon will achieve this goal. I'm going to try this. In order to be able to grow the cohorts, I'm going to make test and control "grow" towards the middle: e.g.

[0, 0.05) - test
[0.05, 0.95) - excluded
[0.95, 1) - control
This does not affect beta 55 since we're rolling out 50/50 and there are no excluded users.
Priority: -- → P1
Comment on attachment 8876750 [details]
Bug 1372237 - Balance the test/control groups for Flash rollout. The test group grows up from 0 as it always has. The control group grows down from 1, and everyone else is "exlucded"

https://reviewboard.mozilla.org/r/148084/#review152418
Attachment #8876750 - Flags: review?(felipc) → review+
Pushed by bsmedberg@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9828f9dac14b
Balance the test/control groups for Flash rollout. The test group grows up from 0 as it always has. The control group grows down from 1, and everyone else is "excluded" r=felipe
https://hg.mozilla.org/mozilla-central/rev/9828f9dac14b
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
Comment on attachment 8876750 [details]
Bug 1372237 - Balance the test/control groups for Flash rollout. The test group grows up from 0 as it always has. The control group grows down from 1, and everyone else is "exlucded"

Approval Request Comment
[Feature/Bug causing the regression]: N/A
[User impact if declined]: Engineering impact only: won't have a comparable control group for Flash CtP rollout
[Is this code covered by automated tests?]: no
[Has the fix been verified in Nightly?]: manual verification (patch has no effect on nightly)
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no. Isolated Flash CtP deployment code
[String changes made/needed]: none
Attachment #8876750 - Flags: approval-mozilla-beta?
Comment on attachment 8876750 [details]
Bug 1372237 - Balance the test/control groups for Flash rollout. The test group grows up from 0 as it always has. The control group grows down from 1, and everyone else is "exlucded"

adjust control group for flash ctp rollout to be the same size as the test group, beta55+
Attachment #8876750 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.