1359939 - Run a Telemetry experiment to vet animated image caching

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect)

Description

[u279076]

This is bug tracks running a Telemetry Experiment to A/B test animated image caching as implemented in bug 686905.

Comment 1

[u279076]

Attachment: animated-image-discarding-nightly55-v1.patch

Timothy, can you please look at this and send me your feedback?

This patch implements the TelEx A/B experiment for animated image discarding by disabling image.mem.animated.discardable for 50% of Nightly users (the pref is currently enabled by default).

Note, the following start/end values are for testing purposes only. Once we go to production these will need to be updated to reflect a 10-day window starting from the go-live date.
> +    "startTime"        : 1493241697,
> +    "endTime"          : 1493510400,

Also, the experiment.xpi file will need to be signed before this goes live.

Thanks

Comment 2

[Timothy Nikkel (:tnikkel)]

Comment on attachment 8862608 [details] [diff] [review]
animated-image-discarding-nightly55-v1.patch

Seems fine to me, although I don't really know about these kinds of things.

Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. Does the experiment handle that?

Comment 3

[u279076]

Thanks Timothy.

> Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. 
> Does the experiment handle that?

Not by itself. 

To account for this we'll need to run the experiment long enough to allow *most* users in the disabled cohort to restart. This experiment will run for 10 days so I'm fairly confident this won't be an issue.  

In addition, we'll get some crash reports from the disabled cohort which actually happened while the pref was enabled and should therefore be counted toward the enabled cohort. In my experience this usually accounts for 1-2% of the data and doesn't tend to throw off the results, as long as you account for it.

Comment 4

[u279076]

Comment on attachment 8862608 [details] [diff] [review]
animated-image-discarding-nightly55-v1.patch

Hi Felipe,

Can you please review this patch for my telemetry experiment?

Proposed Criteria:
>      Product: Firefox Nightly
>      Version: 55.0a1
>           OS: All
>      Channel: nightly
>     Build ID: All
>     Language: en-US
> Sample ratio: 50%
>   Start Date: April 28, 2017
>     End Date: May 7, 2017
>     Duration: 10 days

Metrics:
> Overall crash rate & count by cohort and cpu_arch via Telemetry
> OOM crash rate & count by cohort and cpu_arch via Socorro
> Topcrash signature delta by cohort and cpu_arch via Socorro

Note, before we push to staging the start/end dates need to be changed as per the proposed criteria above and I'll need to get the xpi signed.

Note, before we push to production I will need to test on staging then send the intent-to-ship email.

Comment 5

[u279076]

Attachment: experiment.xpi

Jason, can you please sign this xpi?

Comment 6

[:wezhou]

Attachment: attachment.8862618.signed.zip

The signed file is attached. Please test.

Comment 7

[u279076]

:wezhou, thank you for signing the xpi.

:felipe, thank you for reviewing the patch. 

Please push to staging with the following changes:
* startDate = 1493856000 (May 4th)
* endDate = 1494720000 (May 14th)
* sampleRate = 25%

Comment 8

[:Felipe Gomes (needinfo for replies!)]

Pushed: https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/e6409d179187c67b8397abf44b98f6de6ec09904

With a sample rate of 50% (so that 25% will be in the control group, 25% in the experiment group)

This needs to be tagged with release_tag before deployment. Once relman approves it, ping me and I can tag it.

Comment 9

[u279076]

Thanks for the help. I have tested this on staging and it's working correctly. I will now send an email to RelMan.

Comment 10

[Liz Henry (:lizzard) (relman/hg->git project)]

Looks good. Please go ahead with the experiment on 55 Nightly.

Comment 11

[u279076]

Marking this resolved since testing is completed on staging and I've received the go-ahead from Release Management. Bug 1361782 has been filed to deploy.