Last Comment Bug 1341259 - Run a Telemetry experiment to enable Intersection Observer API
: Run a Telemetry experiment to enable Intersection Observer API
Status: RESOLVED FIXED
:
Product: Core
Classification: Components
Component: DOM (show other bugs)
: 53 Branch
: Unspecified Unspecified
P3 normal (vote)
: ---
Assigned To: Tobias Schneider [:tobytailor]
:
: Andrew Overholt [:overholt]
Mentors:
Depends on: 1344881 1348144 1351837
Blocks: intersection-observer
  Show dependency treegraph
 
Reported: 2017-02-21 04:44 PST by Tobias Schneider [:tobytailor]
Modified: 2017-05-22 11:35 PDT (History)
5 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
wontfix
wontfix
wontfix
affected


Attachments
intersection-observer-nightly54.diff (4.35 KB, patch)
2017-02-21 04:47 PST, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
experiment.xpi (1.40 KB, application/zip)
2017-02-21 04:49 PST, Tobias Schneider [:tobytailor]
no flags Details
intersection-observer-nightly54.diff (4.35 KB, patch)
2017-02-22 14:28 PST, Tobias Schneider [:tobytailor]
felipc: review+
Details | Diff | Splinter Review
experiment.xpi (1.40 KB, application/zip)
2017-02-22 14:29 PST, Tobias Schneider [:tobytailor]
no flags Details
experiment.xpi signed (5.30 KB, application/x-xpinstall)
2017-02-23 09:20 PST, Jason Thomas [:jason]
no flags Details
intersection-observer-experiment (11.46 KB, patch)
2017-02-23 12:29 PST, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
Manifest (1009 bytes, patch)
2017-03-03 10:20 PST, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
Manifest (1009 bytes, patch)
2017-03-06 09:33 PST, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
experiment.xpi (5.30 KB, application/zip)
2017-03-06 09:41 PST, Tobias Schneider [:tobytailor]
no flags Details
experiment.xpi signed (5.30 KB, application/x-xpinstall)
2017-03-06 12:16 PST, Jason Thomas [:jason]
no flags Details
intersection-observer-experiment 2 (8.34 KB, patch)
2017-03-06 12:23 PST, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
intersection-observer-experiment 3 (897 bytes, patch)
2017-03-16 14:24 PDT, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
intersection-observer-experiment 3 (1.83 KB, patch)
2017-03-16 14:35 PDT, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
experiment.xpi (1.40 KB, application/zip)
2017-03-28 13:18 PDT, Tobias Schneider [:tobytailor]
no flags Details
experiment.xpi signed (5.30 KB, application/x-xpinstall)
2017-03-28 14:12 PDT, Jason Thomas [:jason]
no flags Details
intersection-observer-experiment 4 (897 bytes, patch)
2017-03-28 14:26 PDT, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review
intersection-observer-experiment 4 (9.97 KB, patch)
2017-03-28 14:27 PDT, Tobias Schneider [:tobytailor]
no flags Details | Diff | Splinter Review

Description User image Tobias Schneider [:tobytailor] 2017-02-21 04:44:58 PST
We want to A/B test enabling the Intersection API t on all platforms. The idea is that Nightly users who qualify will have the API enabled (via pref dom.IntersectionObserver.enabled); We want to switch half of these users to compare the stability between the two different groups.

Proposed Criteria:

>      Product: Firefox Nightly
>      Version: 54.0a1
>           OS: all
>      Channel: nightly
>     Build ID: 2017-02-23+
>     Language: en-US
> Sample ratio: 50%
>   Start Date: February 23, 2017
>     End Date: March 9, 2017
Comment 1 User image Tobias Schneider [:tobytailor] 2017-02-21 04:47:09 PST
Created attachment 8839438 [details] [diff] [review]
intersection-observer-nightly54.diff
Comment 2 User image Tobias Schneider [:tobytailor] 2017-02-21 04:49:39 PST
Created attachment 8839439 [details]
experiment.xpi
Comment 3 User image Tobias Schneider [:tobytailor] 2017-02-22 10:51:09 PST
Felipe, any more information needed to get this started?
Comment 4 User image Tobias Schneider [:tobytailor] 2017-02-22 14:28:27 PST
Created attachment 8840147 [details] [diff] [review]
intersection-observer-nightly54.diff

Addressed comments on IRC.
Comment 5 User image Tobias Schneider [:tobytailor] 2017-02-22 14:29:05 PST
Created attachment 8840149 [details]
experiment.xpi

Updated add-on.
Comment 6 User image :Felipe Gomes (needinfo me!) 2017-02-22 14:32:58 PST
Looks good. You should get the xpi signed and do some manual testing (and of course get approval from release-drivers to run the experiment) before landing.

After you do some manual testing I'll give r+ and we can push to staging to do a quick QA and them push to prod.
Comment 7 User image Tobias Schneider [:tobytailor] 2017-02-22 14:58:02 PST
Jason, can you please sign the add-on for me?
Comment 8 User image Jason Thomas [:jason] 2017-02-23 09:20:55 PST
Created attachment 8840496 [details]
experiment.xpi signed

Please see attached.
Comment 9 User image Tobias Schneider [:tobytailor] 2017-02-23 10:12:28 PST
Manually tested everything and can confirm it works. Still waiting for release-drivers approval.
Comment 10 User image Tobias Schneider [:tobytailor] 2017-02-23 10:45:24 PST
Felipe, can you please give your r+ so we can land this as soon as I heard back from release-drivers?
Comment 11 User image :Felipe Gomes (needinfo me!) 2017-02-23 11:04:24 PST
Comment on attachment 8840147 [details] [diff] [review]
intersection-observer-nightly54.diff

Review of attachment 8840147 [details] [diff] [review]:
-----------------------------------------------------------------

Sure. Actually, you can land the patch now, as we won't update the release_tag yet. That will put it in the staging server (https://telemetry-experiment-dev.allizom.org/) where we can ask QA to test it (usually a requirement to release an experiment.. not sure if that applies to Nightly but it wouldn't hurt to do some simple testing).
Comment 12 User image Tobias Schneider [:tobytailor] 2017-02-23 12:29:23 PST
Created attachment 8840578 [details] [diff] [review]
intersection-observer-experiment

Final patch to land. Includes signed add-on.
Comment 14 User image Tobias Schneider [:tobytailor] 2017-03-03 10:20:58 PST
Created attachment 8843376 [details] [diff] [review]
Manifest

Updated manifest with new dates and sample rate.
Comment 16 User image Tobias Schneider [:tobytailor] 2017-03-06 09:33:09 PST
Created attachment 8844049 [details] [diff] [review]
Manifest

One last change to the manifest. Now includes nightly version 55a1, to avoid the experiment being disabled for some people after merge today.
Comment 17 User image Tobias Schneider [:tobytailor] 2017-03-06 09:41:46 PST
Created attachment 8844054 [details]
experiment.xpi

Updated add-on.
Comment 18 User image Tobias Schneider [:tobytailor] 2017-03-06 09:42:26 PST
jason, can you please sign the add-on on more time? Thanks.
Comment 19 User image Jason Thomas [:jason] 2017-03-06 12:16:36 PST
Created attachment 8844130 [details]
experiment.xpi signed

Please see attached.
Comment 20 User image Tobias Schneider [:tobytailor] 2017-03-06 12:23:55 PST
Created attachment 8844135 [details] [diff] [review]
intersection-observer-experiment 2

Updated patch with new signed add-on.
Comment 22 User image Tobias Schneider [:tobytailor] 2017-03-16 14:24:37 PDT
Created attachment 8848251 [details] [diff] [review]
intersection-observer-experiment 3

Running the experiment on 10% of the Nightly user population showed no stability issues. Lets do a final run on 50% before shipping.
Comment 23 User image Tobias Schneider [:tobytailor] 2017-03-16 14:35:10 PDT
Created attachment 8848255 [details] [diff] [review]
intersection-observer-experiment 3

Created a new copy of the experiment to make sure we get enough new activations.
Comment 24 User image :Felipe Gomes (needinfo me!) 2017-03-16 14:56:29 PDT
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/40a11e1fa5c17cb934656d26728675d29eff1ee1
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/88b6307803ba6683d9000ef9d3acdc071e88e238

Now you just need to file a new deployment bug like the previous one. Please make sure you have clearance from release-drivers before doing so!
Comment 25 User image Tobias Schneider [:tobytailor] 2017-03-28 13:18:41 PDT
Created attachment 8852179 [details]
experiment.xpi

Updated add-on to target FF 55.0a1. Jason, please sign.
Comment 26 User image Jason Thomas [:jason] 2017-03-28 14:12:10 PDT
Created attachment 8852201 [details]
experiment.xpi signed

Please see attached.
Comment 27 User image Tobias Schneider [:tobytailor] 2017-03-28 14:26:18 PDT
Created attachment 8852208 [details] [diff] [review]
intersection-observer-experiment 4

Running the experiment on 10% and 50% of the Nightly user population showed no stability issues. Lets enable it to 100% before shipping.
Comment 28 User image Tobias Schneider [:tobytailor] 2017-03-28 14:27:27 PDT
Created attachment 8852211 [details] [diff] [review]
intersection-observer-experiment 4

Correct patch this time.
Comment 29 User image Tobias Schneider [:tobytailor] 2017-03-28 14:28:52 PDT
Felipe, can you please land the latest patch?

Note You need to log in before you can comment on or make changes to this bug.