This is bug tracks running a Telemetry Experiment to A/B test animated image caching as implemented in bug 686905.
Created attachment 8862608 [details] [diff] [review] animated-image-discarding-nightly55-v1.patch Timothy, can you please look at this and send me your feedback? This patch implements the TelEx A/B experiment for animated image discarding by disabling image.mem.animated.discardable for 50% of Nightly users (the pref is currently enabled by default). Note, the following start/end values are for testing purposes only. Once we go to production these will need to be updated to reflect a 10-day window starting from the go-live date. > + "startTime" : 1493241697, > + "endTime" : 1493510400, Also, the experiment.xpi file will need to be signed before this goes live. Thanks
Comment on attachment 8862608 [details] [diff] [review] animated-image-discarding-nightly55-v1.patch Seems fine to me, although I don't really know about these kinds of things. Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. Does the experiment handle that?
Thanks Timothy. > Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. > Does the experiment handle that? Not by itself. To account for this we'll need to run the experiment long enough to allow *most* users in the disabled cohort to restart. This experiment will run for 10 days so I'm fairly confident this won't be an issue. In addition, we'll get some crash reports from the disabled cohort which actually happened while the pref was enabled and should therefore be counted toward the enabled cohort. In my experience this usually accounts for 1-2% of the data and doesn't tend to throw off the results, as long as you account for it.
Comment on attachment 8862608 [details] [diff] [review] animated-image-discarding-nightly55-v1.patch Hi Felipe, Can you please review this patch for my telemetry experiment? Proposed Criteria: > Product: Firefox Nightly > Version: 55.0a1 > OS: All > Channel: nightly > Build ID: All > Language: en-US > Sample ratio: 50% > Start Date: April 28, 2017 > End Date: May 7, 2017 > Duration: 10 days Metrics: > Overall crash rate & count by cohort and cpu_arch via Telemetry > OOM crash rate & count by cohort and cpu_arch via Socorro > Topcrash signature delta by cohort and cpu_arch via Socorro Note, before we push to staging the start/end dates need to be changed as per the proposed criteria above and I'll need to get the xpi signed. Note, before we push to production I will need to test on staging then send the intent-to-ship email.
Created attachment 8862618 [details] experiment.xpi Jason, can you please sign this xpi?
Created attachment 8862647 [details] attachment.8862618.signed.zip The signed file is attached. Please test.
:wezhou, thank you for signing the xpi. :felipe, thank you for reviewing the patch. Please push to staging with the following changes: * startDate = 1493856000 (May 4th) * endDate = 1494720000 (May 14th) * sampleRate = 25%
Pushed: https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/e6409d179187c67b8397abf44b98f6de6ec09904 With a sample rate of 50% (so that 25% will be in the control group, 25% in the experiment group) This needs to be tagged with release_tag before deployment. Once relman approves it, ping me and I can tag it.
Thanks for the help. I have tested this on staging and it's working correctly. I will now send an email to RelMan.
Looks good. Please go ahead with the experiment on 55 Nightly.
Marking this resolved since testing is completed on staging and I've received the go-ahead from Release Management. Bug 1361782 has been filed to deploy.