Run a Telemetry experiment to vet animated image caching

RESOLVED FIXED in Firefox 55

Status

()

Core
ImageLib
RESOLVED FIXED
27 days ago
20 days ago

People

(Reporter: ashughes, Assigned: ashughes)

Tracking

55 Branch
mozilla55
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox55+ fixed)

Details

Attachments

(3 attachments)

This is bug tracks running a Telemetry Experiment to A/B test animated image caching as implemented in bug 686905.
Created attachment 8862608 [details] [diff] [review]
animated-image-discarding-nightly55-v1.patch

Timothy, can you please look at this and send me your feedback?

This patch implements the TelEx A/B experiment for animated image discarding by disabling image.mem.animated.discardable for 50% of Nightly users (the pref is currently enabled by default).

Note, the following start/end values are for testing purposes only. Once we go to production these will need to be updated to reflect a 10-day window starting from the go-live date.
> +    "startTime"        : 1493241697,
> +    "endTime"          : 1493510400,

Also, the experiment.xpi file will need to be signed before this goes live.

Thanks
Attachment #8862608 - Flags: feedback?(tnikkel)
Comment on attachment 8862608 [details] [diff] [review]
animated-image-discarding-nightly55-v1.patch

Seems fine to me, although I don't really know about these kinds of things.

Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. Does the experiment handle that?
Attachment #8862608 - Flags: feedback?(tnikkel) → feedback+
Thanks Timothy.

> Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. 
> Does the experiment handle that?

Not by itself. 

To account for this we'll need to run the experiment long enough to allow *most* users in the disabled cohort to restart. This experiment will run for 10 days so I'm fairly confident this won't be an issue.  

In addition, we'll get some crash reports from the disabled cohort which actually happened while the pref was enabled and should therefore be counted toward the enabled cohort. In my experience this usually accounts for 1-2% of the data and doesn't tend to throw off the results, as long as you account for it.
Comment on attachment 8862608 [details] [diff] [review]
animated-image-discarding-nightly55-v1.patch

Hi Felipe,

Can you please review this patch for my telemetry experiment?

Proposed Criteria:
>      Product: Firefox Nightly
>      Version: 55.0a1
>           OS: All
>      Channel: nightly
>     Build ID: All
>     Language: en-US
> Sample ratio: 50%
>   Start Date: April 28, 2017
>     End Date: May 7, 2017
>     Duration: 10 days

Metrics:
> Overall crash rate & count by cohort and cpu_arch via Telemetry
> OOM crash rate & count by cohort and cpu_arch via Socorro
> Topcrash signature delta by cohort and cpu_arch via Socorro

Note, before we push to staging the start/end dates need to be changed as per the proposed criteria above and I'll need to get the xpi signed.

Note, before we push to production I will need to test on staging then send the intent-to-ship email.
Attachment #8862608 - Flags: review?(felipc)
Created attachment 8862618 [details]
experiment.xpi

Jason, can you please sign this xpi?
Attachment #8862618 - Flags: review?(jthomas)

Updated

26 days ago
Flags: needinfo?(wezhou)

Comment 6

26 days ago
Created attachment 8862647 [details]
attachment.8862618.signed.zip

The signed file is attached. Please test.
Flags: needinfo?(wezhou)
Attachment #8862618 - Flags: review?(jthomas)
Attachment #8862608 - Flags: review?(felipc) → review+
:wezhou, thank you for signing the xpi.

:felipe, thank you for reviewing the patch. 

Please push to staging with the following changes:
* startDate = 1493856000 (May 4th)
* endDate = 1494720000 (May 14th)
* sampleRate = 25%
Pushed: https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/e6409d179187c67b8397abf44b98f6de6ec09904

With a sample rate of 50% (so that 25% will be in the control group, 25% in the experiment group)

This needs to be tagged with release_tag before deployment. Once relman approves it, ping me and I can tag it.
Thanks for the help. I have tested this on staging and it's working correctly. I will now send an email to RelMan.
Looks good. Please go ahead with the experiment on 55 Nightly.
status-firefox55: --- → affected
tracking-firefox55: --- → +
Blocks: 1361782
Marking this resolved since testing is completed on staging and I've received the go-ahead from Release Management. Bug 1361782 has been filed to deploy.
Status: NEW → RESOLVED
Last Resolved: 20 days ago
Resolution: --- → FIXED
status-firefox55: affected → fixed
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.