820243 - [tracking] testing infrastructure refresh

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[Chris Cooper [:coop] (he/him)]

We're moving away from minis to 4-node, 2U iX systems for all of our non-Mac test platforms. We had been using bug 758624 to track this effort, but it was never all-encompassing. I'll start collecting the various new and existing dependencies here. 

Here's a (slightly outdated) etherpad that lists the status of the various platforms:
https://etherpad.mozilla.org/ix-node-eval

Comment 1

[Armen [:armenzg]]

* bug 826050 is for running Windows 8 tests on the multi-node iX hardware.
* bug 770578 is for running Windows 7 tests on the multi-node iX hardware.
* bug 770598 is for running Windows XP tests on the multi-node iX hardware.

Comment 2

[Armen [:armenzg]]

> * bug 770598 is for running Windows XP tests on the multi-node iX hardware.

TYPO:
* bug 770579 is for running Windows XP tests on the multi-node iX hardware.

Comment 3

[Armen [:armenzg]]

* bug 826050 is for running Windows 8 tests on the multi-node iX hardware.
* bug 770578 is for running Windows 7 tests on the multi-node iX hardware.
* bug 770579 is for running Windows XP tests on the multi-node iX hardware.
* bug 849984 is for running Linux32 tests on the multi-node iX hardware.
* bug 849985 is for running Linux64 tests on the multi-node iX hardware.

Comment 4

[Armen [:armenzg]]

I want to raise up that 100 win8 machines will not be able to keep up with the on-going load growth of our test pool.

(From a low day: Wait: 36743/78.33% (testpool))
win8: 3315
  0:     3020    91.10%
 15:       44     1.33%
 30:      160     4.83%
 45:       91     2.75%

Comment 5

[Armen [:armenzg]]

I've created this etherpad to keep the status of things:
https://releng.etherpad.mozilla.org/testing-infrastructure-refresh-2013

If you need access please let me know.

Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=820243
* Ubuntu32 - https://bugzilla.mozilla.org/show_bug.cgi?id=852680
    chassis delivered: 25/25
    chassis racked and cabled: 22/25 (DCops to finish on week of 4/22)
    machines imaged and given to releng: 85/100
    machines in production: 76/100
* Win7 - https://bugzilla.mozilla.org/show_bug.cgi?id=770578
    chassis delivered: 9/25 (next batch on 4/22)
    chassis racked and cabled: 0/25
    machines imaged and given to releng: 0/100
    machines in production: 0/100
* WinXp - https://bugzilla.mozilla.org/show_bug.cgi?id=770579
    chassis delivered: 0/25 (first batch on 4/22)
    chassis racked and cabled: 0/25
    machines imaged and given to releng: 0/100
    machines in production: 0/100
* Ubuntu64 - https://bugzilla.mozilla.org/show_bug.cgi?id=849985
    99% completed :)
    chassis delivered: 25/25
    chassis racked and cabled: 25/25
    machines imaged and given to releng: 99/100
    machines in production: 99/100

Other notes:
    We don't have an imaging process ironed out for WinXp

Comment 6

[Armen [:armenzg]]

Ubuntu32/Ubuntu64 have been completed.

Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=820243&hide_resolved=1

Next delivery of 29 chassis:   4/26 (tomorrow) or early next week
Rack mount kits delivery:      4/26 (tomorrow)

NOTE: We're going to wait a bit on racking remaining 29 chassis until we get a proper cabling method. Racking earlier than that is useless until our imaging processes are ironed so please don't rush racking.

Win7 - https://bugzilla.mozilla.org/show_bug.cgi?id=770578
* 18 machines have been racked
** waiting on 2 imaging issues which will be figured out by next week
*** armenzg to re-verify and to Cedar
* 48 hosts are waiting on building cabinets
** cabinets are waiting on delivery of rack mount kits
* remaining 29 chassis are expected by tomorrow or early next week

WinXp - https://bugzilla.mozilla.org/show_bug.cgi?id=770579
* 18 machines have been racked
* We need imaging process (long pole)

Comment 7

[Armen [:armenzg]]

Win7 schedule:
* week of 4/29 - Have some machines on Cedar and green things out
* week of 5/06 - Enable more branches. Have remaining win7 machines in production
* week of 5/13 - Disable some load on Win7 rev3 minis
* week of 5/20 - Disable everything including talos on Win7 rev3 minis

WinXP schedule:
* week of 4/29 - armenzg to have all releng patches ready
* mid-June     - Get some machines for armenzg to test through staging
** the XP imaging process is on research/development period
** the graphics card solution for win7 might not port as easily to WinXP

Let's update the WinXP schedule in 2-3 weeks once we can get a better estimate.

Comment 8

[Armen [:armenzg]]

Attachment: builders being adjusted

Comment 9

[Armen [:armenzg]]

Attachment: disable rev3 for FF24 (m-c) & FF23 (m-a)

Comment 10

[Armen [:armenzg]]

Attachment: disable rev3 for FF24 (m-c) & FF23 (m-a)

Comment 11

[Armen [:armenzg]]

Attachment: disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other

Comment 12

[Armen [:armenzg]]

Attachment: builders being adjusted

Comment 13

[Chris Cooper [:coop] (he/him)]

Comment on attachment 754563 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other

Review of attachment 754563 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit fixed.

::: mozilla-tests/config.py
@@ +1468,5 @@
> +        for s in ('dirtypaint', 'other',):
> +            tests_key = '%s_tests' % s
> +            if tests_key in BRANCHES[branch]:
> +                tests = list(BRANCHES[branch]['%s_tests' % s])
> +                tests[3] = tests[3] + ['xp', 'win7']

Please either use a var here instead of 3 or insert a comment explaining why you're using 3. Hard-coded stuff scares me.

Comment 14

[Chris Cooper [:coop] (he/him)]

Comment on attachment 754563 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other

Review of attachment 754563 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit fixed.

Comment 15

[Armen [:armenzg]]

(In reply to Chris Cooper [:coop] from comment #13)
> Comment on attachment 754563 [details] [diff] [review]
> disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other
> 
> Review of attachment 754563 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with nit fixed.
> 
> ::: mozilla-tests/config.py
> @@ +1468,5 @@
> > +        for s in ('dirtypaint', 'other',):
> > +            tests_key = '%s_tests' % s
> > +            if tests_key in BRANCHES[branch]:
> > +                tests = list(BRANCHES[branch]['%s_tests' % s])
> > +                tests[3] = tests[3] + ['xp', 'win7']
> 
> Please either use a var here instead of 3 or insert a comment explaining why
> you're using 3. Hard-coded stuff scares me.

Lucky enough we won't this block after all. Sheriffs says that they don't care about running it side by side.

Comment 16

[Armen [:armenzg]]

Tomorrow we're disabling the rev3 jobs for XP and Win7 up to FF23.
We will disable it for m-b, m-r and m-esr17 in the following bugs:
* bug 877336
* bug 877337
* bug 877342

After the deployment tomorrow I will close this bug and follow the trains from there on.

Comment 17

[Armen [:armenzg]]

Live in production.

Comment 18

[Armen [:armenzg]]

Comment on attachment 753992 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a)

I ended up landing this version.

Comment 19

[Armen [:armenzg]]

http://armenzg.blogspot.com/2013/05/kiss-old-testing-infra-revision-3-mac.html

We still run Rev3 jobs for m-b, m-r, esr17 and b2g release branches but those will go away as we ride the trains.