759968 - Cleanup foopies and tegras.json/foopies.sh

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Justin Wood (:Callek)]

Attachment: WIP changes

So we have an array of offline-* across foopies when we also either have an up tegra-*, and in a few places tegras assigned where they need not be, etc.

This is to track/fix all that.

Referenced Bugs

* Bug 727345 -- Tegra-024 (WebQA pull)
* Bug 703010 -- Tegra-030 (replaced)
* Bug 754406 -- Tegra-049 (handed to clint)
* Bug 759101 -- Tegra-065 (handed to clint)
* Bug 701766 -- Tegra-122 (on bob's bench being looked at)
* ^^^ ^^^^^^ -- Tegra-122 (on bob's bench being looked at)
* Bug 754647 -- Tegra-161 (given to bmoss and pulled from nagios)
* Bug 741684 -- Tegra-222 (recovery, swapping)
* Bug 758085 -- Tegra-226 (not even sure if its powered)
* Bug 738422 -- Tegra-268 (Pulled for Ateam)

Added tegra-178 to foopy17 (like we thought it was there already)

tegra-239 was down (offline on foopy20), but nagios claims its up, and no bug I can find for why its down, so bringing it back online

TODO: Figure out what tegra's to bring the balance back up as we'd like.

TODO have arr/dcops/relops verify if the PDU assignments for these are still active with anything/them, or if we should pull them from tegras.json as well.

Comment 1

[Justin Wood (:Callek)]

tegra-178 apparently was pulled from the "safe slave" list, so I added it back with r=catlee over IRC. double-landed in prod and default

http://hg.mozilla.org/build/buildbot-configs/rev/e871835577ea
http://hg.mozilla.org/build/buildbot-configs/rev/5899dd7eb4f7

Comment 2

[Justin Wood (:Callek)]

Attachment: [tools] shuffling v1

Once the patch[es] here land, this is the tegra shuffle that is "yet to do"

* Remove:
  * tegra-145 from foopy15

* Add:
  * tegra-078 to foopy07
  * tegra-090 to foopy08
  * tegra-091 to foopy10
  * tegra-145 to foopy14 [** moving from foopy15 which is over committed]
  * tegra-157 to foopy14
  * tegra-171 to foopy16
  * tegra-172 to foopy19
  * tegra-194 to foopy19
  * tegra-195 to foopy23

tegras-per-foopy output after these changes is [note the comment stuff will be fixed up as a followup if its ok]:

PRODUCTION:
  foopy07 contains 11 tegras
  foopy08 contains 11 tegras
  foopy09 contains 11 tegras
  foopy10 contains 11 tegras
  foopy11 contains 11 tegras
  foopy12 contains 13 tegras
  foopy13 contains 13 tegras
  foopy14 contains 13 tegras
  foopy15 contains 13 tegras
  foopy16 contains 13 tegras
  foopy17 contains 13 tegras
  foopy18 contains 13 tegras
  foopy19 contains 13 tegras
  foopy20 contains 13 tegras
  foopy22 contains 13 tegras
  foopy23 contains 13 tegras
  foopy24 contains 13 tegras
We have 211 tegras in 17 foopies which means a ratio of 12 tegras per foopy

STAGING:
  foopy05 contains 8 tegras
  foopy06 contains 15 tegras
We have 23 tegras in 2 foopies which means a ratio of 11 tegras per foopy

UNASSIGNED
   2 With Comment: Bug 701766, on bob's bench being looked at
   1 With Comment: Bug 703010 - replaced
   1 With Comment: Bug 727345: Assigned to WebQA
   1 With Comment: Bug 736630, pulled for clint
   1 With Comment: Bug 738422, Pulled for ATeam
   1 With Comment: Bug 741684 (recovery)
   5 With Comment: Bug 749637: Assigned to Sec-Team
   1 With Comment: Bug 754406, pulled for clint
   1 With Comment: Bug 758085, recovery
   1 With Comment: Bug 759101, pulled for clint
   5 (With no Comment)

Comment 3

[Justin Wood (:Callek)]

Comment on attachment 628884 [details] [diff] [review]
[tools] shuffling v1

pushed this to tools.

Updated foopies on foopy05->foopy10, including its tools checkout.

Proceeded with manual edit of /builds/create_dirs.sh to match the TEGRAS list in this bug, then ran create_dirs.sh and then a ./start_cp.sh to start clientproxy for these new ones.

Comment 4

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #3)
> Updated foopies on foopy05->foopy10, including its tools checkout.

Rest of them are done