1350092 - Run a Telemetry experiment to vet GPU Process on Windows prior to release in Firefox 53

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[u279076]

Similar to bug 1319206 (GPU Process A/B test on Windows in Nightly 53) we want to A/B test GPU Process in Beta 53 on Windows in support of bug 1264543. Currently Beta users who qualify have GPU Process on by default (~35% of users); we want to turn GPU Process off for half of these users to compare the stability. All other factors being equal this should help us determine the impact of GPU Process prior to release.

Proposed Criteria:
>      Product: Firefox Beta
>      Version: 53.0
>           OS: Windows (all)
>      Channel: beta
>     Build ID: 2017-03-07+ (53.0b1+)
>     Language: en-US
> Sample ratio: 50%
>   Start Date: March 27, 2017
>     End Date: April 10, 2017

The proposed dates will give us roughly 2 weeks of testing and leave just over a week for pre-release stabilization.

Comment 1

[u279076]

Attachment: bug1350092-v1.patch

David, can you please review this patch? It's essentially the same as the experiment we ran on Nightly updated for Beta. Basically it should take a random selection of Beta users with who have GPU Process enabled and flip that pref.

Thanks

Comment 2

[u279076]

Attachment: experiment.xpi

Jason,

Can you please sign the xpi?

Comment 3

[Jason Thomas [:jason]]

Attachment: experiment.xpi signed

Please see attached.

Comment 4

[u279076]

Thanks Jason and David for the quick follow up.

Cory, :ekr suggested you might be able to push this to staging for me. Can you take care of this? 

If not, I'll ask Felipe in the morning.

Comment 5

[u279076]

Felipe, can you please take care of the push to staging? I've not heard back from :ckprice yet.

Comment 6

[u279076]

Sorry for the noise, Felipe. I neglected to check IRC scroll back and it looks like :ckprice is looking in to this.

Comment 7

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

This needs RelMan approval. Usually via an "Intent to Ship" email: https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process#Intent_to_Ship_and_RelMan_Approval

I've also NI'd Julien

Comment 8

[Julien Cristau [:jcristau]]

302 → liz

Comment 9

[:Felipe Gomes (needinfo for replies!)]

This is a telemetry experiment, right? Not a system add-on? The process is different here, pushing to staging is just a matter of landing this on http://hg.mozilla.org/webtools/telemetry-experiment-server/ as long as it is reviewed

Comment 10

[u279076]

Yes, this is just a Telemetry Experiment. I'd like this pushed to staging so I can test it. Assuming it passes my tests I will then send I2S email to RelMan for approval to push live.

Comment 11

[u279076]

(In reply to Anthony Hughes (:ashughes) [GFX][QA][Mentor] from comment #10)
> Yes, this is just a Telemetry Experiment. I'd like this pushed to staging so
> I can test it. Assuming it passes my tests I will then send I2S email to
> RelMan for approval to push live.

To clarify I am following the instructions starting from https://wiki.mozilla.org/QA/Telemetry/Developing_a_Telemetry_Experiment#Testing_on_Beta. 

I've tested this in a local build, had the code reviewed, had the XPI signed. If I am not mistaken the next step should be push to staging, test on staging, email RelMan, then push live.

Comment 12

[Liz Henry (:lizzard) (relman/hg->git project)]

Yes, it's fine to get this onto staging and then send the Intent to Ship before deployment (according to the guidelines here: https://wiki.mozilla.org/Telemetry/Experiments ). Thanks!

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Pushed to staging:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/8caa6ce7abe680000b2b8b76f0fca9b99f65fc7a

Should be in https://telemetry-experiment-dev.allizom.org/ in a couple of minutes.

I think Cory was following the process for system add-ons.

Comment 14

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

> I think Cory was following the process for system add-ons.
Yes, my mistake.

Comment 15

[u279076]

Thanks everyone!

Comment 16

[u279076]

I've signed off the staged experiment and sent an email to RelMan for approval.

Comment 17

[Liz Henry (:lizzard) (relman/hg->git project)]

Thanks, looks good to me! Please go ahead and deploy.

Comment 18

[u279076]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #17)
> Thanks, looks good to me! Please go ahead and deploy.

Thanks Liz, I sent the deploy request via bug 1350100.

Comment 19

[u279076]

This is now live and tested as working in production.