684253 - need more 2 foopies for current tegra pool in addition to 4 for new foopies

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Mike Taylor [:bear]]

With the increase in test and project coverage, the working directory for a tegra build has increased from an average of 300M to 450M.

The foopies have always had small(ish) drives, so this increase means that the 13 tegras per foopy are now running out of space regularly (even with an aggressive cronjob to cull logs and temp files)

I would like to reduce the tegra count to 10 per foopy and that would mean find a new home for 18 Tegras.

18 Tegras would be 1.8 foopies
40 new Tegras would be 4 foopies

So we would need to bring online 6 foopies to handle the new and current requirements.

Comment 1

[Zandr Milewski [:zandr]]

Going to pull 6 rev4's from scl1 for this. This will also give me a sense of how much load a foopy puts on modern hardware.

Comment 2

[Amy Rich [:arr] [:arich]]

Matt, bring back 6 minis from scl1, and we'll set the up in mtv1 (dns, dhcp, nagios) for this.

Comment 3

[Matthew Larrain[:MaRu]]

I have brought the 6 mini's back to my desk in MTV as soon as I have back in the office I will rack them on the second floor.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

These are racked and running.  They have IPs from DHCP and are in inventory.  They should appear in DNS shortly (I just committed).

Bear, you can start setting these up, but I'll reassign to Amy for the moment to get nagios running.

Comment 5

[Mike Taylor [:bear]]

Amy I don't think has any known good setup for nagios on foopies because nagios doesn't run on foopies and that's a bug i'm supposed to be fixing ;)

but yea, I'll start chewing on this - exact time to be posted when I figure out my time availability

Comment 6

[Amy Rich [:arr] [:arich]]

Bear: When it comes to installing and configuring nrpe on OS X, it's already done via puppet on other releng servers, so it's just a matter of getting the foopies puppetized and using that bit of code, I think.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I added ping checks to nagios.

I also ssh'd to all 6, so they're all up :)

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Oh, I missed a critical detail, the hostnames:

 foopy12.build.mozilla.org
 foopy13.build.mozilla.org
 foopy14.build.mozilla.org
 foopy15.build.mozilla.org
 foopy16.build.mozilla.org
 foopy17.build.mozilla.org

:)

Also, to save you a trip to inventory, these are mounted in 2.IDF in mountain view, at the top of the rack with all the tegras in it.

Comment 9

[Mike Taylor [:bear]]

Attachment: scripts to setup a foopy after it has been imaged

figure we should be storing these and tools/buildfarm/mobile/ looked to be a good spot.

these are what i've been working on the new foopies with and i've updated https://wiki.mozilla.org/ReferencePlatforms/Test/Foopy to reference them

Comment 10

[Mike Taylor [:bear]]

Attachment: add new tegras to buildbot-configs

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 562043 [details] [diff] [review]
add new tegras to buildbot-configs

Hm.
It's good keeping things contiguous, but this means any try jobs running at the time of the reconfig will die.

If we keep 65,95 as try and add the new tegras in either pool, then the reconfig will have no negative effect.

Comment 12

[Mike Taylor [:bear]]

Attachment: add new tegras to buildbot-configs

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 562132 [details] [diff] [review]
add new tegras to buildbot-configs

(Still not completely sold on this split, but that's not for this patch. r=me)

Comment 14

[Mike Taylor [:bear]]

Comment on attachment 562132 [details] [diff] [review]
add new tegras to buildbot-configs

committed changeset 4794:3961a546f29c

Comment 15

[Mike Taylor [:bear]]

Comment on attachment 561690 [details] [diff] [review]
scripts to setup a foopy after it has been imaged

committed changeset 1821:082274864dd6