Power and rack 100 mac minis

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
8 years ago
3 years ago

People

(Reporter: joduinn, Assigned: jlaz)

Tracking

Details

(Whiteboard: Win7 x64 Mini)

2nd batch of minis were delivered.

Comment 1

8 years ago
jlaz is going to head this one up.
Assignee: server-ops → jlazaro
(Assignee)

Comment 2

8 years ago
Macs unboxed, tagged on a spreadsheet, sent to oremj and mrz
(Assignee)

Updated

8 years ago
Whiteboard: Waiting to be imaged/processed
any ETA on these?
(Assignee)

Comment 4

8 years ago
We've started imaging these machines today, with 20 Minis imaged for Leopard to be finished by EOD
(Assignee)

Updated

8 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 5

8 years ago
30 Minis imaged for Linux/Fedora today
(Assignee)

Comment 6

8 years ago
80% imaging complete

Completed
20 Leopard
20 Snow Leopard
20 Fedora 32-bit
20 XP

Processing
20 Windows 7
Thanks for the update jlazaro; still on track for today?
(Assignee)

Comment 8

8 years ago
Summary: 40 machines connected, but have random IP addresses until they are manually entered into DNS.  Another 40 should be connected Monday.

Details


Windows 7 machines have yet to be imaged until we figure out the licensing issues/new reference image.

The rev1 machines were taken offline and will be put into the QA lab, while the rev3 machines in the QA are in the process of moving to the rest of the rev3 machines in the server closet on the 2nd floor.

All machines excluding the fed64 (which are currently in the QA lab) and Windows 7 are racked, with 20 Fedora-32 and 20 XP currently connected to the network, but still need to be added to DNS/DHCP.  

We are planning to install another switch to accomodate the rest of the machines, since we have run out of space on the current switch.  We may also need 4 10-outlet powerstrips and 2 Mac Mini Racks to accomodate the rest of the machines.
Thanks for the info, jlazaro.

(In reply to comment #8)
> Summary: 40 machines connected, but have random IP addresses until they are
> manually entered into DNS.  
Any ETA to having DNS/DHCP fixed, and getting these machines into production? 



...
> Windows 7 machines have yet to be imaged until we figure out the licensing
> issues/new reference image.
This is news to me. Can you clarify what is the problem with the win7 ref image? We are currently using that ref image in production and it works for us. We need all the slaves in a pool to be identical, so if there is any change needed, can you elaborate what exactly needs to be changed?

Comment 10

8 years ago
(In reply to comment #9)
> Thanks for the info, jlazaro.
> 
> (In reply to comment #8)
> > Summary: 40 machines connected, but have random IP addresses until they are
> > manually entered into DNS.  
> Any ETA to having DNS/DHCP fixed, and getting these machines into production? 
> 
> 

These are all in DNS and DHCP.

> 
> ...
> > Windows 7 machines have yet to be imaged until we figure out the licensing
> > issues/new reference image.
> This is news to me. Can you clarify what is the problem with the win7 ref
> image? We are currently using that ref image in production and it works for us.
> We need all the slaves in a pool to be identical, so if there is any change
> needed, can you elaborate what exactly needs to be changed?

I couldn't get the current license key activated on the new batch of minis.  It said that the hardware has been changed and we need a new key.  I am going to try and apply a new license and create a new image for it.  This will save me the trouble of having to call Microsoft and activate each of them over the phone.
(In reply to comment #10)
> I couldn't get the current license key activated on the new batch of minis.  It
> said that the hardware has been changed and we need a new key.  I am going to
> try and apply a new license and create a new image for it.  This will save me
> the trouble of having to call Microsoft and activate each of them over the
> phone.

If this truly is a change in hardware then that could be an problem for the uniformity of the talos pool. Could you boot one of this batch to OS X and run the System Profiler, and attach the output here. Then we can repeat with one from the first batch and look for changes.
Depends on: 551010
Another 40 Mac Minis (20 Snow Leopard and 20 Leopard) are currently racked and online.  The 20 remaining Windows 7 Minis are being processed with phong's recent windows 7 ref image, which should be online once we have enough power allocated for them (powerstrips on order).
Slaves w7-031 thru w7-040 now up and available.  Hostname didn't get auto-set by opsi (will have to confer with bhearsum on this), have manually set.

Screen size on these boxes keeps resetting to 800x600 on reboot, unlike the first set of w7 machines.  Will diagnose with jlazaro.  Without correct 1280x1040 screen size these machines cannot create trustworthy perf results.
w7-040 was reserved to be a w7x64 slave, it will not come online with this set.
(In reply to comment #13)
> Slaves w7-031 thru w7-040 now up and available.  Hostname didn't get auto-set
> by opsi (will have to confer with bhearsum on this), have manually set.
typo: slaves w7-021...w7-040


> Screen size on these boxes keeps resetting to 800x600 on reboot, unlike the
> first set of w7 machines.  Will diagnose with jlazaro.  Without correct
> 1280x1040 screen size these machines cannot create trustworthy perf results.
With this incorrect resolution, we must remove these machines from production immediately before they give incorrect results. For now, we're disabling those new slaves, while RelEng and phong figure out whats wrong the screen resolution.
Removed win7 slaves 21-40 through a talos master reconfig until we get their screen sizes fixed (minor change on config.py).

(typo in comment #13, should be w7-021 thru 040)
Phong: can you get us access to the newer win7 refimage machine - we'd like to see if that somehow reverted to 800x600 before it was used to generate the image?

Comment 18

8 years ago
I used talos-r3-w7-021 as the ref machine.
Summary aiui:

All done except for:
* screen resolution problem with talos-r3-w7-021...040
* one windows 7 x64 need to be setup in bug#543625.
* leopard-034 needs to be serviced
* fed64-011 still needs to be moved from QA to the server room
* fed64-020 missing

Anything else to do here?

Comment 20

8 years ago
(In reply to comment #19)

> * fed64-020 missing
> 
> Anything else to do here?

fed64 only goes up to 19.
(In reply to comment #19)
> * fed64-011 still needs to be moved from QA to the server room

jlazaro just moved this, and its now back in production.
Leopard-034 is currently being serviced by Apple with a turnaround time of 5-7 business days.  

A working Win7 x64 Mini should be up and running by the end of the week, where Phong is finalizing the reference image.
Whiteboard: Waiting to be imaged/processed → Waiting on Leopard-034 / Win7 x64 Mini
Depends on: 553081
(In reply to comment #20)
> (In reply to comment #19)
> 
> > * fed64-020 missing
> > 
> > Anything else to do here?
> 
> fed64 only goes up to 19.

I've found another 2.26 mini upstairs beside Sean which he knows nothing about. Its still in its unopened shipping carton. Handed over to jlazaro to image up as fed64-020.
fed64-020 has been inventoried, added to DHCP/DNS, and should now be online

-jlazaro
Leopard-034 is back, 
W7x64 mini is all that is left
(Assignee)

Updated

8 years ago
Whiteboard: Waiting on Leopard-034 / Win7 x64 Mini → Win7 x64 Mini
All is set.

-jlazaro
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.