704585 - deploy remaining tegras

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Comment 1

[Amy Rich [:arr] [:arich]]

Parts ordered:

* pigtails (ETA less than a week)
* cat5e (ETA less than a week)
* CW-32VD1-C20 CDUs (ETA 10 days)
* sonnet chassis for foopies (ETA less than a week)
* mac minis for foopies (ETA next week)

Once the parts get in, we need to install/configure the switches and foopies, and then attach the tegras (already imaged).  We're looking at the week of December 12th to do deployment.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting w/arr,dustin on Friday (2nd):

* long pole is waiting for last PDU; its ETA is 10dec.
* given all prep work already in place, expect to have these tegras visible to RelEng early in week (12-14th) assuming PDU arrives on schedule.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

This is mostly finished.  80 tegras are ready to go.  The exceptions are:
 * foopies not all configured yet (hwine is working on this)
 * war-room tegras are not swapped out yet
 * there are three more tegras which we need to look for space/power for
 * several of the new tegras aren't pingable and need some TLC - bug 710515
the latter three points will be addressed Thursday.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Dustin J. Mitchell [:dustin] from comment #3)
> This is mostly finished.  80 tegras are ready to go.  The exceptions are:
>  * foopies not all configured yet (hwine is working on this)
...in bug#704597.

>  * war-room tegras are not swapped out yet (bug 707584)
...added to tracking bug.

>  * there are three more tegras which we need to look for space/power for
there are 7 spare shoe racks; from irc w/arr, it seems the 7 empty slots are not adjacent to empty power. arr recommends keeping these 3 spare in reserve.


>  * several of the new tegras aren't pingable and need some TLC - bug 710515
...added to tracking bug.

> the latter three points will be addressed Thursday.
Status update?

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

> > the latter three points will be addressed Thursday.
> Status update?

They'll be addressed today (Thursday).

Comment 6

[Hal Wine [:hwine] use NI!]

found 2 doa tegras so far (only configuring for foopy18 at this point)

tegra-209
tegra-215

Neither is pingable, even after PDU power cycle.

I'll proceed with checking other new tegras as I bring up their controlling foopies.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Hal, those are in bug 710515 already.  Please add any others, that aren't already in that bug, there.

Comment 8

[Hal Wine [:hwine] use NI!]

Dustin - sorry I missed the ref to bug 710515 - that list matches mine exactly

Comment 9

[Mike Taylor [:bear]]

tegras are being moved to production and any "problem childs" are being tracked in bug 710515