Requirements: * want snow leopard + win7 * ts only * 4200 addons * How many addons to do you expect to test a day? Likely around 50, with peaks around 100. * test against latest release (Firefox3.6 currently) Ts = 5 minutes per run, assume 85 runs a day (to handle peak reasonably well) = ~7 hours of processor time per-OS. I propose creating two snow leopard and two win7 talos boxes and segregating them to only do addon performance testing. The win here is that addon testing requests will no place any more load on the existing pool of slaves and will also get through the system at a reasonable rate. Nick will be responsible for the hardware cost, but I'm filing this in releng as they will be treated like any other talos box in terms of maintenance/upkeep.
Will discuss prioritization of this with joduinn when he returns next week.
I'm not a huge fan of segregating the pool, every time we do it we increase the maintenance burden. Catlee recently implemented a thing that allows us to run jobs during idle time (define as "more than N slaves idle"), how would you feel about doing something like that instead? Regardless, more hardware may be necessary.
We have agreed to purchase extra machines to handle this load - comment 1 indicates two new snow leopard and two new win7. I don't mind having the addon tests just going into the general pool, but they can't be totally prioritized down as we need to guarantee that we can get through 50 addons a day with peaks of 100.
I can't imagine we'd have a problem meeting that based on how idle we get late at night and early in the morning.
The expectation is to add 7 hours per-OS of processor time - would the idle times ensure that this wouldn't bog things down?
Also, Nick - you should figure out your timeline for purchasing so that this can get rolling.
(In reply to comment #5) > The expectation is to add 7 hours per-OS of processor time - would the idle > times ensure that this wouldn't bog things down? 7 hours per OS isn't very much in the grand scheme of the pool. If we had 25 machines idle per OS (roughly half of the current pool) we'd finish everything in 16.8 minutes of wall clock time. Overnight, there's times when we have far more than idle, too. I think we'd be running few or no add-on jobs during the PDT day/evening, but it doesn't seem like that matters for this kind of testing.
I'm not worried about some lag in processing. We can keep track of the time it takes to process and reevaluate resources once we get a processing flow done.
Summary of meeting with Nick. 1) The incremental new job load is low enough that we believe we can handle this during idle times. There is a lot of addon-tester jobs to do as we first start this up. However, we can do those gradually. 2) These talos machines need to be identical to the existing production pool of minis, which are no longer easily available. Buying new / different talos machines for a new separate pool would complicate trying to do comparison between talos-on-plain-firefox vs talos-on-firefox-with-addon. Therefore, nick and i agree to not setup this additional pool, close this bug as WONTFIX and instead queue up these jobs as idle time jobs in the regular production test pool.