bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

[tracking] testing infrastructure refresh

RESOLVED FIXED

Status

Infrastructure & Operations
CIDuty
P2
normal
RESOLVED FIXED
6 years ago
2 months ago

People

(Reporter: coop, Assigned: armenzg)

Tracking

Details

Attachments

(2 attachments, 3 obsolete attachments)

(Reporter)

Description

6 years ago
We're moving away from minis to 4-node, 2U iX systems for all of our non-Mac test platforms. We had been using bug 758624 to track this effort, but it was never all-encompassing. I'll start collecting the various new and existing dependencies here. 

Here's a (slightly outdated) etherpad that lists the status of the various platforms:
https://etherpad.mozilla.org/ix-node-eval

Updated

6 years ago
Blocks: 831495

Updated

5 years ago
No longer depends on: 731280

Updated

5 years ago
Depends on: 826050
(Assignee)

Updated

5 years ago
No longer depends on: 826050
(Assignee)

Comment 1

5 years ago
* bug 826050 is for running Windows 8 tests on the multi-node iX hardware.
* bug 770578 is for running Windows 7 tests on the multi-node iX hardware.
* bug 770598 is for running Windows XP tests on the multi-node iX hardware.
Depends on: 826050, 770578, 770579
(Assignee)

Comment 2

5 years ago
> * bug 770598 is for running Windows XP tests on the multi-node iX hardware.

TYPO:
* bug 770579 is for running Windows XP tests on the multi-node iX hardware.
(Assignee)

Updated

5 years ago
Depends on: 849984
(Assignee)

Updated

5 years ago
Depends on: 849985
(Assignee)

Updated

5 years ago
No longer depends on: 816078
(Assignee)

Updated

5 years ago
No longer depends on: 820238
(Assignee)

Updated

5 years ago
No longer depends on: 820235
(Assignee)

Updated

5 years ago
No longer depends on: 780024
(Assignee)

Updated

5 years ago
No longer depends on: 780025
(Assignee)

Updated

5 years ago
No longer depends on: 803595
(Assignee)

Comment 3

5 years ago
* bug 826050 is for running Windows 8 tests on the multi-node iX hardware.
* bug 770578 is for running Windows 7 tests on the multi-node iX hardware.
* bug 770579 is for running Windows XP tests on the multi-node iX hardware.
* bug 849984 is for running Linux32 tests on the multi-node iX hardware.
* bug 849985 is for running Linux64 tests on the multi-node iX hardware.
(Assignee)

Comment 4

5 years ago
I want to raise up that 100 win8 machines will not be able to keep up with the on-going load growth of our test pool.

(From a low day: Wait: 36743/78.33% (testpool))
win8: 3315
  0:     3020    91.10%
 15:       44     1.33%
 30:      160     4.83%
 45:       91     2.75%
Depends on: 855554
(Assignee)

Comment 5

5 years ago
I've created this etherpad to keep the status of things:
https://releng.etherpad.mozilla.org/testing-infrastructure-refresh-2013

If you need access please let me know.

Tracking bug: https://bugzilla.mozilla.org/show_bug.cgi?id=820243
* Ubuntu32 - https://bugzilla.mozilla.org/show_bug.cgi?id=852680
    chassis delivered: 25/25
    chassis racked and cabled: 22/25 (DCops to finish on week of 4/22)
    machines imaged and given to releng: 85/100
    machines in production: 76/100
* Win7 - https://bugzilla.mozilla.org/show_bug.cgi?id=770578
    chassis delivered: 9/25 (next batch on 4/22)
    chassis racked and cabled: 0/25
    machines imaged and given to releng: 0/100
    machines in production: 0/100
* WinXp - https://bugzilla.mozilla.org/show_bug.cgi?id=770579
    chassis delivered: 0/25 (first batch on 4/22)
    chassis racked and cabled: 0/25
    machines imaged and given to releng: 0/100
    machines in production: 0/100
* Ubuntu64 - https://bugzilla.mozilla.org/show_bug.cgi?id=849985
    99% completed :)
    chassis delivered: 25/25
    chassis racked and cabled: 25/25
    machines imaged and given to releng: 99/100
    machines in production: 99/100

Other notes:
    We don't have an imaging process ironed out for WinXp
(Assignee)

Updated

5 years ago
Blocks: 863236
(Assignee)

Updated

5 years ago
No longer blocks: 831495
(Assignee)

Comment 6

5 years ago
Ubuntu32/Ubuntu64 have been completed.

Dependency tree:
https://bugzilla.mozilla.org/showdependencytree.cgi?id=820243&hide_resolved=1

Next delivery of 29 chassis:   4/26 (tomorrow) or early next week
Rack mount kits delivery:      4/26 (tomorrow)

NOTE: We're going to wait a bit on racking remaining 29 chassis until we get a proper cabling method. Racking earlier than that is useless until our imaging processes are ironed so please don't rush racking.

Win7 - https://bugzilla.mozilla.org/show_bug.cgi?id=770578
* 18 machines have been racked
** waiting on 2 imaging issues which will be figured out by next week
*** armenzg to re-verify and to Cedar
* 48 hosts are waiting on building cabinets
** cabinets are waiting on delivery of rack mount kits
* remaining 29 chassis are expected by tomorrow or early next week

WinXp - https://bugzilla.mozilla.org/show_bug.cgi?id=770579
* 18 machines have been racked
* We need imaging process (long pole)
Assignee: coop → armenzg
Priority: -- → P4
Whiteboard: status on comment 6 - Ubuntu32/64 completed - Win7/WinXP waiting on imaging and chassis deliveries
No longer depends on: 855554
(Assignee)

Comment 7

5 years ago
Win7 schedule:
* week of 4/29 - Have some machines on Cedar and green things out
* week of 5/06 - Enable more branches. Have remaining win7 machines in production
* week of 5/13 - Disable some load on Win7 rev3 minis
* week of 5/20 - Disable everything including talos on Win7 rev3 minis

WinXP schedule:
* week of 4/29 - armenzg to have all releng patches ready
* mid-June     - Get some machines for armenzg to test through staging
** the XP imaging process is on research/development period
** the graphics card solution for win7 might not port as easily to WinXP

Let's update the WinXP schedule in 2-3 weeks once we can get a better estimate.
(Assignee)

Updated

5 years ago
Whiteboard: status on comment 6 - Ubuntu32/64 completed - Win7/WinXP waiting on imaging and chassis deliveries → left: switch over from Rev3 minis to iX infra
(Assignee)

Comment 8

5 years ago
Created attachment 753990 [details] [diff] [review]
builders being adjusted
(Assignee)

Comment 9

5 years ago
Created attachment 753992 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a)
(Assignee)

Comment 10

5 years ago
Created attachment 753993 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a)
Attachment #753992 - Attachment is obsolete: true
(Assignee)

Comment 11

5 years ago
Created attachment 754563 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other
Attachment #753993 - Attachment is obsolete: true
Attachment #754563 - Flags: review?(coop)
(Assignee)

Comment 12

5 years ago
Created attachment 754565 [details] [diff] [review]
builders being adjusted
(Reporter)

Comment 13

5 years ago
Comment on attachment 754563 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other

Review of attachment 754563 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit fixed.

::: mozilla-tests/config.py
@@ +1468,5 @@
> +        for s in ('dirtypaint', 'other',):
> +            tests_key = '%s_tests' % s
> +            if tests_key in BRANCHES[branch]:
> +                tests = list(BRANCHES[branch]['%s_tests' % s])
> +                tests[3] = tests[3] + ['xp', 'win7']

Please either use a var here instead of 3 or insert a comment explaining why you're using 3. Hard-coded stuff scares me.
(Reporter)

Comment 14

5 years ago
Comment on attachment 754563 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other

Review of attachment 754563 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with nit fixed.
Attachment #754563 - Flags: review?(coop) → review+
(Assignee)

Comment 15

5 years ago
(In reply to Chris Cooper [:coop] from comment #13)
> Comment on attachment 754563 [details] [diff] [review]
> disable rev3 for FF24 (m-c) & FF23 (m-a) except talos dirtypaint and other
> 
> Review of attachment 754563 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with nit fixed.
> 
> ::: mozilla-tests/config.py
> @@ +1468,5 @@
> > +        for s in ('dirtypaint', 'other',):
> > +            tests_key = '%s_tests' % s
> > +            if tests_key in BRANCHES[branch]:
> > +                tests = list(BRANCHES[branch]['%s_tests' % s])
> > +                tests[3] = tests[3] + ['xp', 'win7']
> 
> Please either use a var here instead of 3 or insert a comment explaining why
> you're using 3. Hard-coded stuff scares me.

Lucky enough we won't this block after all. Sheriffs says that they don't care about running it side by side.
(Assignee)

Comment 16

5 years ago
Tomorrow we're disabling the rev3 jobs for XP and Win7 up to FF23.
We will disable it for m-b, m-r and m-esr17 in the following bugs:
* bug 877336
* bug 877337
* bug 877342

After the deployment tomorrow I will close this bug and follow the trains from there on.
Whiteboard: left: switch over from Rev3 minis to iX infra
(Assignee)

Updated

5 years ago
Blocks: 877337
(Assignee)

Updated

5 years ago
Blocks: 877342
(Assignee)

Updated

5 years ago
Blocks: 877336
(Assignee)

Updated

5 years ago
Priority: P4 → P2
(Assignee)

Comment 17

5 years ago
Live in production.
(Assignee)

Comment 18

5 years ago
Comment on attachment 753992 [details] [diff] [review]
disable rev3 for FF24 (m-c) & FF23 (m-a)

I ended up landing this version.
Attachment #753992 - Flags: review+
Attachment #753992 - Flags: checked-in+
(Assignee)

Updated

5 years ago
Attachment #753992 - Attachment is obsolete: false
(Assignee)

Updated

5 years ago
Attachment #754563 - Attachment is obsolete: true
(Assignee)

Updated

5 years ago
Attachment #754565 - Attachment is obsolete: true
(Assignee)

Comment 19

5 years ago
http://armenzg.blogspot.com/2013/05/kiss-old-testing-infra-revision-3-mac.html

We still run Rev3 jobs for m-b, m-r, esr17 and b2g release branches but those will go away as we ride the trains.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.