order and configure suffient foopies/build masters to support first 60 new panda boards

RESOLVED FIXED

Status

Release Engineering
Buildduty
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: hwine, Assigned: hwine)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [reit-panda][buildduty][mobile][foopy])

60 new pandas in 5 chassis were ordered via bug 777359. The additional systems (foopies and build-masters) to operate these pandas have not been ordered.

Placeholder for now, will be updated with the quantity after some discussion.

Since these will be linux foopies (which have ganglia support), we are considering ordering fewer (supporting more devices per foopy), and monitoring ganglia to see if we're okay.
IIUC there is also foopy25 that can be used for this purpose.
We discussed that 12 panda boards per linux foopy might be a good start.
(In reply to Armen Zambrano G. [:armenzg] from comment #1)
> IIUC there is also foopy25 that can be used for this purpose.
> We discussed that 12 panda boards per linux foopy might be a good start.

foopy25 is already servicing the prototype chassis with 12 boards. We are going to experiment with 1 foopy to 2 panda chassis (24 boards total) as part of bug 789516.
plan is to pull 5 more hp boxes to use as foopies at the rate of one foopy per panda chassis. As soon as the 5 hp boxes are identified and taken out of service, I'll open the bug to get those reimaged.
Please decommission 5 hp boxes from service. See bug 776977 comment #4 for how this was done last time.
Whiteboard: [reit-panda] → [reit-panda][buildduty][mobile][foopy]
(In reply to Hal Wine [:hwine] from comment #4)
> Please decommission 5 hp boxes from service. See bug 776977 comment #4 for
> how this was done last time.

Just to be extra clear, these 5 boxes will remain in scl1.

Comment 6

6 years ago
I've decommissioned the following build and try slaves in slavealloc. 

bld-centos6-hp-011	
bld-centos6-hp-012
bld-centos6-hp-013
bld-centos6-hp-034
bld-centos6-hp-035

Comment 7

6 years ago
I was just going through the buildduty queue and discovered that some of these machines are broken even though there isn't a note in slavealloc to denote that.

So the new list is
bld-centos6-hp-010
bld-centos6-hp-011	
bld-centos6-hp-014
bld-centos6-hp-033
bld-centos6-hp-034
Depends on: 790339
No longer depends on: 790339
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
What foopies and masters got generated at the end of this bug?

I'm confused as why this got fixed.
Bah- I knew I should have opened a new bug for the "final configure" instead of morphing bug 790339 into that.

The foopies being used are listed in bug 790339 - no need for additional buildbot master was identified, so we'll be using existing masters
Hal, since we're ordering 400 panda boards, we'll need a lot more foopies, depending on if the testing shows a single foopy can support one or two chassis.  I was talking to coop about this and he said that we'll need to order some more HP machines to support this given that we don't have that many left to reimage for foopies.
kmoir: I've already suggested purchasing 10 ix-multinode systems (4 nodes each) which will give us one foopy per chassis and account for the imaging server that is needed per rack.
(In reply to Amy Rich [:arich] [:arr] from comment #11)
> kmoir: I've already suggested purchasing 10 ix-multinode systems (4 nodes
> each) which will give us one foopy per chassis and account for the imaging
> server that is needed per rack.

*IFF* we are to use ix-multinode, we will need to qualify it as a foopy. So if that is a wanted [all around] choice, please get a bug on file to do so, get a cent6 image setup based on the same image for current foopies, and the setup handed back to releng so we can stage it for a min of 2-3 days.

Questions I would want to know (answered in said qualifiation bug) if we do that, "how does the Disk I/O throughput per node compare against the HPs", "How does the CPU/RAM allocation per node compare against the HPs"

Until/unless qualification is done on another Hardware type, I don't think we can accept that as a solution
changing summary to reflect this bug was only for the first 60 new pandas. New bugs will be opened for each order.

To clarify, there is no action yet on any new foopy hardware qualification or order.
Summary: order and configure suffient foopies/build masters to support new panda boards → order and configure suffient foopies/build masters to support first 60 new panda boards

Updated

6 years ago
Blocks: 799698
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.