1341259 - Run a Telemetry experiment to enable Intersection Observer API

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, defect, P3)

Description

[Tobias Schneider [:tobytailor]]

We want to A/B test enabling the Intersection API t on all platforms. The idea is that Nightly users who qualify will have the API enabled (via pref dom.IntersectionObserver.enabled); We want to switch half of these users to compare the stability between the two different groups.

Proposed Criteria:

>      Product: Firefox Nightly
>      Version: 54.0a1
>           OS: all
>      Channel: nightly
>     Build ID: 2017-02-23+
>     Language: en-US
> Sample ratio: 50%
>   Start Date: February 23, 2017
>     End Date: March 9, 2017

Comment 1

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-nightly54.diff

Comment 2

[Tobias Schneider [:tobytailor]]

Attachment: experiment.xpi

Comment 3

[Tobias Schneider [:tobytailor]]

Felipe, any more information needed to get this started?

Comment 4

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-nightly54.diff

Addressed comments on IRC.

Comment 5

[Tobias Schneider [:tobytailor]]

Attachment: experiment.xpi

Updated add-on.

Comment 6

[:Felipe Gomes (needinfo for replies!)]

Looks good. You should get the xpi signed and do some manual testing (and of course get approval from release-drivers to run the experiment) before landing.

After you do some manual testing I'll give r+ and we can push to staging to do a quick QA and them push to prod.

Comment 7

[Tobias Schneider [:tobytailor]]

Jason, can you please sign the add-on for me?

Comment 8

[Jason Thomas [:jason]]

Attachment: experiment.xpi signed

Please see attached.

Comment 9

[Tobias Schneider [:tobytailor]]

Manually tested everything and can confirm it works. Still waiting for release-drivers approval.

Comment 10

[Tobias Schneider [:tobytailor]]

Felipe, can you please give your r+ so we can land this as soon as I heard back from release-drivers?

Comment 11

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8840147 [details] [diff] [review]
intersection-observer-nightly54.diff

Review of attachment 8840147 [details] [diff] [review]:
-----------------------------------------------------------------

Sure. Actually, you can land the patch now, as we won't update the release_tag yet. That will put it in the staging server (https://telemetry-experiment-dev.allizom.org/) where we can ask QA to test it (usually a requirement to release an experiment.. not sure if that applies to Nightly but it wouldn't hurt to do some simple testing).

Comment 12

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment

Final patch to land. Includes signed add-on.

Comment 13

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/5ff9169cf01eab2598dd1cc05919e0efcf45c795

Comment 14

[Tobias Schneider [:tobytailor]]

Attachment: Manifest

Updated manifest with new dates and sample rate.

Comment 15

[:Felipe Gomes (needinfo for replies!)]

I pushed the manifest update:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/068e2fb1e8e3229441a10ad50f1712d805dd8ab2

and tagged it as release_tag:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/da6ccc855030760dc95a081261f70ed34f896a36

Comment 16

[Tobias Schneider [:tobytailor]]

Attachment: Manifest

One last change to the manifest. Now includes nightly version 55a1, to avoid the experiment being disabled for some people after merge today.

Comment 17

[Tobias Schneider [:tobytailor]]

Attachment: experiment.xpi

Updated add-on.

Comment 18

[Tobias Schneider [:tobytailor]]

jason, can you please sign the add-on on more time? Thanks.

Comment 19

[Jason Thomas [:jason]]

Attachment: experiment.xpi signed

Please see attached.

Comment 20

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment 2

Updated patch with new signed add-on.

Comment 21

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/6e38436bc5d2
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/13bf28e08b4b

Comment 22

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment 3

Running the experiment on 10% of the Nightly user population showed no stability issues. Lets do a final run on 50% before shipping.

Comment 23

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment 3

Created a new copy of the experiment to make sure we get enough new activations.

Comment 24

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/40a11e1fa5c17cb934656d26728675d29eff1ee1
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/88b6307803ba6683d9000ef9d3acdc071e88e238

Now you just need to file a new deployment bug like the previous one. Please make sure you have clearance from release-drivers before doing so!

Comment 25

[Tobias Schneider [:tobytailor]]

Attachment: experiment.xpi

Updated add-on to target FF 55.0a1. Jason, please sign.

Comment 26

[Jason Thomas [:jason]]

Attachment: experiment.xpi signed

Please see attached.

Comment 27

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment 4

Running the experiment on 10% and 50% of the Nightly user population showed no stability issues. Lets enable it to 100% before shipping.

Comment 28

[Tobias Schneider [:tobytailor]]

Attachment: intersection-observer-experiment 4

Correct patch this time.

Comment 29

[Tobias Schneider [:tobytailor]]

Felipe, can you please land the latest patch?

Comment 30

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/3c9acc1b7f5bfbbcd0ff137bceddc9ff2820e935
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/dc0e815061ca276a9c2c5b018f5f7580b58c51a1