Run a Telemetry experiment to vet animated image caching

RESOLVED FIXED in Firefox 55



2 years ago
2 years ago


(Reporter: ashughes, Assigned: ashughes)


55 Branch
Dependency tree / graph

Firefox Tracking Flags

(firefox55+ fixed)



(3 attachments)



2 years ago
This is bug tracks running a Telemetry Experiment to A/B test animated image caching as implemented in bug 686905.

Comment 1

2 years ago
Timothy, can you please look at this and send me your feedback?

This patch implements the TelEx A/B experiment for animated image discarding by disabling image.mem.animated.discardable for 50% of Nightly users (the pref is currently enabled by default).

Note, the following start/end values are for testing purposes only. Once we go to production these will need to be updated to reflect a 10-day window starting from the go-live date.
> +    "startTime"        : 1493241697,
> +    "endTime"          : 1493510400,

Also, the experiment.xpi file will need to be signed before this goes live.

Attachment #8862608 - Flags: feedback?(tnikkel)
Comment on attachment 8862608 [details] [diff] [review]

Seems fine to me, although I don't really know about these kinds of things.

Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. Does the experiment handle that?
Attachment #8862608 - Flags: feedback?(tnikkel) → feedback+

Comment 3

2 years ago
Thanks Timothy.

> Any change to the pref image.mem.animated.discardable doesn't take affect until we restart the browser. 
> Does the experiment handle that?

Not by itself. 

To account for this we'll need to run the experiment long enough to allow *most* users in the disabled cohort to restart. This experiment will run for 10 days so I'm fairly confident this won't be an issue.  

In addition, we'll get some crash reports from the disabled cohort which actually happened while the pref was enabled and should therefore be counted toward the enabled cohort. In my experience this usually accounts for 1-2% of the data and doesn't tend to throw off the results, as long as you account for it.

Comment 4

2 years ago
Comment on attachment 8862608 [details] [diff] [review]

Hi Felipe,

Can you please review this patch for my telemetry experiment?

Proposed Criteria:
>      Product: Firefox Nightly
>      Version: 55.0a1
>           OS: All
>      Channel: nightly
>     Build ID: All
>     Language: en-US
> Sample ratio: 50%
>   Start Date: April 28, 2017
>     End Date: May 7, 2017
>     Duration: 10 days

> Overall crash rate & count by cohort and cpu_arch via Telemetry
> OOM crash rate & count by cohort and cpu_arch via Socorro
> Topcrash signature delta by cohort and cpu_arch via Socorro

Note, before we push to staging the start/end dates need to be changed as per the proposed criteria above and I'll need to get the xpi signed.

Note, before we push to production I will need to test on staging then send the intent-to-ship email.
Attachment #8862608 - Flags: review?(felipc)

Comment 5

2 years ago
Posted file experiment.xpi
Jason, can you please sign this xpi?
Attachment #8862618 - Flags: review?(jthomas)
Flags: needinfo?(wezhou)

Comment 6

2 years ago
The signed file is attached. Please test.
Flags: needinfo?(wezhou)


2 years ago
Attachment #8862618 - Flags: review?(jthomas)
Attachment #8862608 - Flags: review?(felipc) → review+

Comment 7

2 years ago
:wezhou, thank you for signing the xpi.

:felipe, thank you for reviewing the patch. 

Please push to staging with the following changes:
* startDate = 1493856000 (May 4th)
* endDate = 1494720000 (May 14th)
* sampleRate = 25%

With a sample rate of 50% (so that 25% will be in the control group, 25% in the experiment group)

This needs to be tagged with release_tag before deployment. Once relman approves it, ping me and I can tag it.

Comment 9

2 years ago
Thanks for the help. I have tested this on staging and it's working correctly. I will now send an email to RelMan.
Looks good. Please go ahead with the experiment on 55 Nightly.
status-firefox55: --- → affected
tracking-firefox55: --- → +


2 years ago
Blocks: 1361782

Comment 11

2 years ago
Marking this resolved since testing is completed on staging and I've received the go-ahead from Release Management. Bug 1361782 has been filed to deploy.
Last Resolved: 2 years ago
Resolution: --- → FIXED
status-firefox55: affected → fixed
Target Milestone: --- → mozilla55
You need to log in before you can comment on or make changes to this bug.