669415 - [tracking bug] bring 40 new tegras online

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Mike Taylor [:bear]]

bring online as many of the new 40 tegras that are ready for use.

per https://bugzilla.mozilla.org/show_bug.cgi?id=665892#c3 we have 40 waiting for setup.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per meeting with IT yesterday:

* no update; zandr will talk to desktop Tuesday (16th)

Comment 2

[Chris Cooper [:coop] (he/him)]

per meeting with IT yesterday:

zandr is not getting sufficient response from desktop, going to tackle himself this week.

Comment 3

[Chris Cooper [:coop] (he/him)]

per meeting with IT today:

* The tegras were being furiously imaged last night. ETA is tomorrow AM for power hook-up and handover.

* need increased foopy capacity to handle new tegras (at least 2 more foopys, possibly up to 5). bear can fit some new tegras onto existing foopys, but zandr will see how many minis he can rustle up to use as new foopys.

* bear to look into NRPE on foopys to avoid future disk space crunches.

Comment 4

[Chris Cooper [:coop] (he/him)]

Any progress update here? This has been a top-3 priority (now #1) since the beginning of August.

Comment 6

[Zandr Milewski [:zandr]]

Replacements for -018,-021,-053,-058 have been imaged, inventory and DHCP has been updated. -18 and -21 have been deployed, -53 and -58 Nagios has not been updated.

tegra-094 through 123 have been added to inventory, DHCP and DNS. Nagios has not been updated.

tegra-094 through 118 are deployed. All except -106 and -107 are up and responding to 20701.

Tuesday I'll deploy the remaining replacements, image and deploy -119 through -123, and hand off to :arr for nagios.

The remaining boards will be used for bug 681369 and bug 682125.

Foopies mentioned in comment 3 are bug 684253.

Comment 7

[Zandr Milewski [:zandr]]

(In reply to Zandr Milewski [:zandr] from comment #6)
>  -18 and -21 have been deployed, -53 and -58 

...have not been deployed.

Comment 8

[Zandr Milewski [:zandr]]

tegra-094 through -123 are on pdu4:

pdu4.build.mozilla.org               IN  CNAME   pdu4.df202-1.build.mtv1.mozilla.com
pdu4.df202-1.build.mtv1.mozilla.com  IN  A       10.250.48.159

Comment 9

[Amy Rich [:arr] [:arich]]

I've added the returning MIA tegras and all of the new ones to nagios as well as the pdu.

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

tweaking summary to reduce confusion with bug#669421 after talking with zandr.

Comment 11

[Zandr Milewski [:zandr]]

bug 685630 has consumed all undeployed tegras, so there's no work to do here until after the boards are returned from the allhands.

leaving this open to recover those machines.

Comment 12

[Amy Rich [:arr] [:arich]]

zandr will be racking 119 - 122 today and handing off 5 for iron monkey and 1 to another developer.  Those are all the tegras we have.

Comment 13

[Zandr Milewski [:zandr]]

119 to 121, actually. All the other tegras have been placed in the hands of devs in other bugs, so these are up.