1384032 - determine hardware capacity requirements for migrating win 10 gpu tests to hw

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations :: RelOps: General, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

in the event that we fail to solve bug 1382625 for gpu tests, we need to determine what capacity would be required to migrate windows 10 gpu tests to hardware.

these test suites run on gpu instances:
- mochitest-gpu
- mochitest-webgl
- reftest
- reftest-gpu*
- reftest-no-accel

*: reftest-gpu doesn't yet run anywhere on tc (on any os/platform), see bug 1356240

it's possible that only some part of the test suites are problematic (eg: mochitest-webgl, chunk 1 & reftest-gpu, chunk 1, always cause impaired state on ec2 gpu instances, the others normally succeed/pass, if the kill script is running)

jmaher: do you have any guess what capacity we would need on hardware for either scenario (all gpu suites or alternatively if we can isolate the tests from the two problematic chunks mentioned above)?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

mochitest-gpu - 8 minutes opt/pgo/debug (25 total)
mochitest-webgl - 85 minutes debug, 50 minutes opt/pgo (185 total)
reftest - 27 minutes opt, 37 minutes debug, 25 minutes pgo (90 total)
reftest-gpu - 77 minutes debug, 30 minutes opt/pgo (140 total)
reftest-no-accel - 27 minutes opt, 37 minutes debug, 25 minutes pgo (90 total)

Given that we have about 900 commits/week on trunk, and that is about half of our overall load we should plan for a range of 2000 commits/week.  We do not necessarily need to run every test on every config on every push, but the reality is we will run it most of the time.

Max time would be 530 minutes/push (9 hours/push) and at a rate of 2000 pushes/week, we would need an upper bound of 18000/hours each week or translated into machines, we would need 107.14 machines, or 110 to account for reboots needed.  If I were requesting this, I would start at a minimum of 125.

In reality we don't run tests on all pushes, especially try.  Overall we can probably estimate about half of the load, so given the upper bound of 110 machines, 55 machines would be the number, and to account for loaners, dead machines, etc. you would need 65-70 machines.

---------------
scenario 2 is interesting where we cherry pick out specific tests that need hardware or are not stable on VM.  This is effectively what we did for mochitest-clipboard and have other HW based tests in there :)  I really don't know the scope of this, but I would image we would need 100 engineering hours to identify and properly split this up.  What would remain is probably 1/3 of the test load on HW and the rest on VM, so maybe 20-25 HW machines needed in this scenario.