keep (non-ESR support) rev2 minis in mtv1 as hardware spares for ESR support until firefox 17

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: dustin, Assigned: arich)

Tracking

Details

(Whiteboard: [sjc1 evac])

These hosts should be retired when we have r5 mini builders up and running (e.g., bug 688632)

| moz2-darwin10-slave01 |
| moz2-darwin10-slave02 |
| moz2-darwin10-slave40 |
| moz2-darwin10-slave41 |
| moz2-darwin10-slave42 |
| moz2-darwin10-slave43 |
| moz2-darwin10-slave44 |
| moz2-darwin10-slave45 |
| moz2-darwin10-slave53 |
| moz2-darwin10-slave54 |
| moz2-darwin10-slave55 |
| moz2-darwin10-slave56 |
| try-mac64-slave27     |
| try-mac64-slave29     |
| try-mac64-slave30     |
| try-mac64-slave31     |
| moz2-darwin10-slave46 |
| moz2-darwin10-slave47 |
| moz2-darwin10-slave48 |
| moz2-darwin10-slave49 |
| moz2-darwin10-slave50 |
| try-mac64-slave28     |

They can't move much of anywhere - they do not handle 208v or cold-aisle containment, so are unsuitable for scl3.  scl1 doesn't have space.
(Assignee)

Updated

7 years ago
Assignee: server-ops-releng → arich
colo-trip: --- → mtv1
tweaking summary after RelEng/IT meeting yesterday. Depending on how long it takes to get rev5 minis up in production, these rev2 machines may need to be kept in use in production for a while, and IT has a contingency plan for this.
Summary: retire rev2 minis in mtv1 → relocate or retire rev2 minis in mtv1
I think you've confused this with the rev5's currently in sjc1.  The minis in comment 0 are in mtv1 and will never move to scl1 -- if we can't retire them, they'll stay put in mtv1 until we can (and that day will come eventually, if only by the hosts themselves dying).

I don't have a bug for the rev2's in sjc1, but I will file one today and connect it to bug 712457.
Blocks: 712457
No longer blocks: 672969
Summary: relocate or retire rev2 minis in mtv1 → retire rev2 minis in mtv1
*rev2's currently in sjc1.  So many rev's!

Comment 4

7 years ago
If I understand correctly we currently do 10.6 builds (rev2) from mtv1 (#c0) and sjc1 (bug 715337).

I think joduinn wants to make sure that this won't happen until we have 10.7 builders and we switch over.

I guess if we added "retiring rev2 builders from *sjc*" (bug 715337) and "building from 10.7" as dependencies it would not sound as "there is nothing blocking us to do this work". I think adding the whiteboard is also good enough. Feel free to adjust.

In other words what you say in here:
(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> -- if we can't retire
> them, they'll stay put in mtv1 until we can (and that day will come
> eventually, if only by the hosts themselves dying).
Whiteboard: if we can't retire them, they'll stay put in mtv1 until we can
You are correct that there is nothing actionable here.  John asked me to open a bug to track it until it is actionable.  I dig get the blocker/blocking relationship wrong.
No longer blocks: 712457
Depends on: 712457
(Assignee)

Comment 6

7 years ago
In order to support the ESR release, we need to keep 10 moz2-darwin10-slaveXX machines in mtv1 along with their current buildbot master.  That means that our list of r2 minis to decommission when the r5 minis are in production is:

| moz2-darwin10-slave01 |
| moz2-darwin10-slave02 |
| try-mac64-slave27     |
| try-mac64-slave29     |
| try-mac64-slave30     |
| try-mac64-slave31     |
| moz2-darwin10-slave46 |
| moz2-darwin10-slave47 |
| moz2-darwin10-slave48 |
| moz2-darwin10-slave49 |
| moz2-darwin10-slave50 |
| try-mac64-slave28     |
Summary: retire rev2 minis in mtv1 → retire (non-ESR support) rev2 minis in mtv1 when r5 minis are in production
Whiteboard: if we can't retire them, they'll stay put in mtv1 until we can
(Assignee)

Updated

7 years ago
Depends on: 715411
(Assignee)

Updated

7 years ago
No longer depends on: 715411
(Assignee)

Updated

7 years ago
Priority: -- → P2
Whiteboard: [sjc1 evac]
(Assignee)

Updated

7 years ago
No longer depends on: 712457
(Assignee)

Comment 7

7 years ago
Since beta and release will still be building on r2 minis when we evac sjc1, we are going to hold off on decommissioning any of these minis until we are certain that the r2 builders can handle the load of ESR+beta+release.  If all goes according to plan, the r2 minis in mtv1 will only be building ESR (nightly builds and anything required for a security release) from the time firefox 14 is released till the time firefox 17 is released.  

We will keep the above minis as hardware spares in case those building ESR fail.
Summary: retire (non-ESR support) rev2 minis in mtv1 when r5 minis are in production → keep (non-ESR support) rev2 minis in mtv1 as hardware spares for ESR support until firefox 17
I'm getting DCOps to set aside 20 extra darwin10 minis - moz2-darwin10-slave[03-22] - as part of the evacuation from sjc. These minis will be used to backfill in the case of failures in the mtv pool. Some may also be required to prop up Thunderbird Mac builds until Thunderbird can build on 10.7 (bug 748628).
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.