Last Comment Bug 1350092 - Run a Telemetry experiment to vet GPU Process on Windows prior to release in Firefox 53
: Run a Telemetry experiment to vet GPU Process on Windows prior to release in ...
Status: RESOLVED FIXED
[gfx-noted]
:
Product: Core
Classification: Components
Component: Graphics (show other bugs)
: 53 Branch
: All Windows
-- normal (vote)
: ---
Assigned To: Anthony Hughes (:ashughes) [GFX][QA][Mentor]
:
: Milan Sreckovic [:milan]
Mentors:
Depends on:
Blocks: e10s-gpu 1350100
  Show dependency treegraph
 
Reported: 2017-03-23 14:47 PDT by Anthony Hughes (:ashughes) [GFX][QA][Mentor]
Modified: 2017-04-07 14:49 PDT (History)
8 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed


Attachments
bug1350092-v1.patch (4.28 KB, patch)
2017-03-23 14:55 PDT, Anthony Hughes (:ashughes) [GFX][QA][Mentor]
dvander: review+
Details | Diff | Splinter Review
experiment.xpi (1.48 KB, application/x-xpinstall)
2017-03-23 15:00 PDT, Anthony Hughes (:ashughes) [GFX][QA][Mentor]
no flags Details
experiment.xpi signed (5.35 KB, application/x-xpinstall)
2017-03-23 15:08 PDT, Jason Thomas [:jason]
no flags Details

Description User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-23 14:47:38 PDT
Similar to bug 1319206 (GPU Process A/B test on Windows in Nightly 53) we want to A/B test GPU Process in Beta 53 on Windows in support of bug 1264543. Currently Beta users who qualify have GPU Process on by default (~35% of users); we want to turn GPU Process off for half of these users to compare the stability. All other factors being equal this should help us determine the impact of GPU Process prior to release.

Proposed Criteria:
>      Product: Firefox Beta
>      Version: 53.0
>           OS: Windows (all)
>      Channel: beta
>     Build ID: 2017-03-07+ (53.0b1+)
>     Language: en-US
> Sample ratio: 50%
>   Start Date: March 27, 2017
>     End Date: April 10, 2017

The proposed dates will give us roughly 2 weeks of testing and leave just over a week for pre-release stabilization.
Comment 1 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-23 14:55:05 PDT
Created attachment 8850730 [details] [diff] [review]
bug1350092-v1.patch

David, can you please review this patch? It's essentially the same as the experiment we ran on Nightly updated for Beta. Basically it should take a random selection of Beta users with who have GPU Process enabled and flip that pref.

Thanks
Comment 2 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-23 15:00:25 PDT
Created attachment 8850734 [details]
experiment.xpi

Jason,

Can you please sign the xpi?
Comment 3 User image Jason Thomas [:jason] 2017-03-23 15:08:14 PDT
Created attachment 8850738 [details]
experiment.xpi signed

Please see attached.
Comment 4 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-23 16:09:15 PDT
Thanks Jason and David for the quick follow up.

Cory, :ekr suggested you might be able to push this to staging for me. Can you take care of this? 

If not, I'll ask Felipe in the morning.
Comment 5 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 09:14:18 PDT
Felipe, can you please take care of the push to staging? I've not heard back from :ckprice yet.
Comment 6 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 09:17:16 PDT
Sorry for the noise, Felipe. I neglected to check IRC scroll back and it looks like :ckprice is looking in to this.
Comment 7 User image Cory Price [:ckprice] (bugmail disabled, NI me!) 2017-03-24 09:18:06 PDT
This needs RelMan approval. Usually via an "Intent to Ship" email: https://wiki.mozilla.org/Firefox/Go_Faster/System_Add-ons/Process#Intent_to_Ship_and_RelMan_Approval

I've also NI'd Julien
Comment 8 User image Julien Cristau [:jcristau] 2017-03-24 09:21:19 PDT
302 → liz
Comment 9 User image :Felipe Gomes (needinfo me!) 2017-03-24 09:21:41 PDT
This is a telemetry experiment, right? Not a system add-on? The process is different here, pushing to staging is just a matter of landing this on http://hg.mozilla.org/webtools/telemetry-experiment-server/ as long as it is reviewed
Comment 10 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 09:47:12 PDT
Yes, this is just a Telemetry Experiment. I'd like this pushed to staging so I can test it. Assuming it passes my tests I will then send I2S email to RelMan for approval to push live.
Comment 11 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 09:55:14 PDT
(In reply to Anthony Hughes (:ashughes) [GFX][QA][Mentor] from comment #10)
> Yes, this is just a Telemetry Experiment. I'd like this pushed to staging so
> I can test it. Assuming it passes my tests I will then send I2S email to
> RelMan for approval to push live.

To clarify I am following the instructions starting from https://wiki.mozilla.org/QA/Telemetry/Developing_a_Telemetry_Experiment#Testing_on_Beta. 

I've tested this in a local build, had the code reviewed, had the XPI signed. If I am not mistaken the next step should be push to staging, test on staging, email RelMan, then push live.
Comment 12 User image Liz Henry (:lizzard) (needinfo? me) 2017-03-24 09:58:28 PDT
Yes, it's fine to get this onto staging and then send the Intent to Ship before deployment (according to the guidelines here: https://wiki.mozilla.org/Telemetry/Experiments ). Thanks!
Comment 13 User image :Felipe Gomes (needinfo me!) 2017-03-24 09:59:17 PDT
Pushed to staging:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/8caa6ce7abe680000b2b8b76f0fca9b99f65fc7a

Should be in https://telemetry-experiment-dev.allizom.org/ in a couple of minutes.

I think Cory was following the process for system add-ons.
Comment 14 User image Cory Price [:ckprice] (bugmail disabled, NI me!) 2017-03-24 09:59:57 PDT
> I think Cory was following the process for system add-ons.
Yes, my mistake.
Comment 15 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 10:03:32 PDT
Thanks everyone!
Comment 16 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-24 10:45:21 PDT
I've signed off the staged experiment and sent an email to RelMan for approval.
Comment 17 User image Liz Henry (:lizzard) (needinfo? me) 2017-03-27 11:46:45 PDT
Thanks, looks good to me! Please go ahead and deploy.
Comment 18 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-27 11:51:58 PDT
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #17)
> Thanks, looks good to me! Please go ahead and deploy.

Thanks Liz, I sent the deploy request via bug 1350100.
Comment 19 User image Anthony Hughes (:ashughes) [GFX][QA][Mentor] 2017-03-27 12:47:09 PDT
This is now live and tested as working in production.

Note You need to log in before you can comment on or make changes to this bug.