Closed Bug 521494 Opened 15 years ago Closed 13 years ago

[Tracking bug] Put existing hardware to better use

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: joduinn, Assigned: joduinn)

References

Details

(Whiteboard: [buildslaves][capacity])

Attachments

(1 file)

summary of what machines went where
2.18 KB, text/csv
Details 
Digging through inventory, I noted the following. 

1) We have 7 xserves on older branches:
bm-xserve01 (ppc)
bm-xserve02 (ppc)
bm-xserve03 (ppc)
bm-xserve04 (ppc)		
bm-xserve05 (ppc)
bm-xserve07 (ppc)
bm-xserve08 (ppc)
We need to leave some of PPC bm-xserve* machines on TB2, FF3.0. However, can we move the rest to moz2 coverage, and replace them with slower 1.6GHz non-PPC minis on the lower-traffic older branches?


2) Are any of these assorted machines being used? If not, I'd like to reimage them and add them to the moz2 production pools.
bm-stage-osx-01 (mini)
unused-2098 (mini)
unused-2099 (mini)
qm-image-master
qm-purify01
qm-xserve03  	 	 	 	 	 	 	 	 	qm-xserve04 									
qm-xserve05
try-pmac-unit-01
(In reply to comment #1)
> Digging through inventory, I noted the following. 
> 2) Are any of these assorted machines being used? If not, I'd like to reimage
> them and add them to the moz2 production pools.
> bm-stage-osx-01 (mini)
> unused-2098 (mini)
> unused-2099 (mini)
> qm-image-master
> qm-purify01
> qm-xserve03                                              qm-xserve04            
> qm-xserve05
> try-pmac-unit-01


I think its fine to turn off these machines now. If there's no complaints in a week, we will reimage.
Turned off:
bm-stage-osx-01 (Wed Oct 21 ~1PM Pac.)

Reachable (wrong credentials):
qm-xserve03

Reachable (no ssh):
qm-purify01

Unreachable (had IP):
qm-image-master
qm-xserve04
qm-xserve05

Unreachable (no IP):
unused-2098
unused-2099
try-pmac-unit-01

Phong, do you know the status of the machines that I wasn't able to shut down?  Can they be shutdown on your next trip to the colo (unless you know they are being actively used)?
Assignee: nobody → server-ops
Component: Release Engineering → Server Operations
QA Contact: release → mrz
(In reply to comment #2)

> 
> Reachable (wrong credentials):
> qm-xserve03

This is being used by QA (bug 518883)
(In reply to comment #2)
> 
> Unreachable (no IP):
> unused-2098
> unused-2099
> try-pmac-unit-01

unused-2099 = talos-rev2-tiger14  	ym9030hmyl1  	2099
unused-2098 = talos-rev2-leopard15  	ym9032dzyl1  	2098

I've deleted them from inventory.
(In reply to comment #4)
> (In reply to comment #2)
> unused-2099 = talos-rev2-tiger14      ym9030hmyl1      2099
> unused-2098 = talos-rev2-leopard15      ym9032dzyl1      2098
> 
> I've deleted them from inventory.

Phong: 

1) thanks for removing the "unused-209*" entries from inventory. Can you also give me access to bug#518883 so I can investigate qm-xserve03? If its something we loaned to QA, I'll see if we can get it back.

2) who owns qm-purify01? Its on our inventory, but none standard name for us, and we are unable to login with our standard passwords.

3) when you are next in colo, can you investigate the remaining machines we could not connect to?

qm-image-master
qm-xserve04 
qm-xserve05
try-pmac-unit-01
Component: Server Operations → Release Engineering
Assignee: server-ops → dmoore
Assignee: dmoore → phong
(In reply to comment #0)
> Digging through inventory, I noted the following. 
> bm-xserve02 (ppc)

Used to be the 1.8 firefox builder. Not doing anything now.
   
> bm-xserve05 (ppc)

This is the tb 1.8 release machine

> bm-xserve07 (ppc)

This one is already in production.
Component: Release Engineering → Server Operations
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #2)
> > unused-2099 = talos-rev2-tiger14      ym9030hmyl1      2099
> > unused-2098 = talos-rev2-leopard15      ym9032dzyl1      2098
> > 
> > I've deleted them from inventory.
> 
> Phong: 
> 
> 1) thanks for removing the "unused-209*" entries from inventory. Can you also
> give me access to bug#518883 so I can investigate qm-xserve03? If its something
> we loaned to QA, I'll see if we can get it back.

CC'ed on the bug

> 
> 2) who owns qm-purify01? Its on our inventory, but none standard name for us,
> and we are unable to login with our standard passwords.
> 
This belongs to Rob Sayre

> 3) when you are next in colo, can you investigate the remaining machines we
> could not connect to?
> 
> qm-image-master
on my desk to be re-used.

> qm-xserve04 
> qm-xserve05
> try-pmac-unit-01
I will check on these tomorrow.
(In reply to comment #7)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > (In reply to comment #2)
> > > unused-2099 = talos-rev2-tiger14      ym9030hmyl1      2099
> > > unused-2098 = talos-rev2-leopard15      ym9032dzyl1      2098
> > > 
> > > I've deleted them from inventory.
> > 
> > Phong: 
> > 
> > 1) thanks for removing the "unused-209*" entries from inventory. Can you also
> > give me access to bug#518883 so I can investigate qm-xserve03? If its something
> > we loaned to QA, I'll see if we can get it back.
> CC'ed on the bug
Thanks Phong. 

From irc with bc, he's ok with us using this for build/unittest. I'll split out a separate bug to image that. 
> 
> > 
> > 2) who owns qm-purify01? Its on our inventory, but none standard name for us,
> > and we are unable to login with our standard passwords.
> > 
> This belongs to Rob Sayre
Was incorrectly listed in RelEng inventory, Phong has now fixed. 


> > 3) when you are next in colo, can you investigate the remaining machines we
> > could not connect to?
> > 
> > qm-image-master
> on my desk to be re-used.
This is another 1.83ghz mini.


> > qm-xserve04 
> > qm-xserve05
> > try-pmac-unit-01
> I will check on these tomorrow.
Cool.
(In reply to comment #6)
> (In reply to comment #0)
> > Digging through inventory, I noted the following. 
> > bm-xserve02 (ppc)
> 
> Used to be the 1.8 firefox builder. Not doing anything now.
oh, thats right, its the hot-standby in case bm-xserve01 (ppc) dies on the thunderbird folks and closes their tree.
(In reply to comment #8)
> (In reply to comment #7)
Phong, while in colo, can you also look for these two machines, missing in sequence in inventory?
> > (In reply to comment #5)
> > > (In reply to comment #4)
> > > > (In reply to comment #2)
> > > qm-xserve04 
> > > qm-xserve05
> > > try-pmac-unit-01
bm-xserve09
bm-xserve10

> > I will check on these tomorrow.
> Cool.
(In reply to comment #0)

> qm-xserve04
is on and not sure what it is doing

> qm-xserve05
was powered off.  I've turned it back on and see if anyone noticed.

> try-pmac-unit-01
Sitting on the rack with the other mac minis just like it said in inventory.
qm-xserve05 looks like it was wiped clean and it has a black OEM image on it.
Found in inventory:
qm-mini-centos01
qm-mini-centos02 	

What are these being used for? What spec of minis are these?
(In reply to comment #11)
> (In reply to comment #0)
> > qm-xserve04
> is on and not sure what it is doing
> > qm-xserve05
> was powered off.  I've turned it back on and see if anyone noticed.
> > try-pmac-unit-01
> Sitting on the rack with the other mac minis just like it said in inventory.

Inventory doesnt say much about these 3 machines. Can you verify asset#s for these machines to see if they are assigned to someone already? 

If that doesnt help, the only thing I can think of is to power these three machines down, wait some long time for complaints, and then *finally* discover if these machines are in use, or are available to help with wait times.
Not sure what is left to do here.
Assignee: phong → joduinn
Component: Server Operations → Release Engineering
QA Contact: mrz → release
(In reply to comment #15)
> Not sure what is left to do here.

Took some digging, but the summary is:

minis to reimage
================
qm-image-master (mini)
try-pmac-unit-01 (mini)
qm-mini-centos01
qm-mini-centos02


xserves to reimage and switch to new branch / geriatric
=======================================================
bm-xserve03 (ppc)
bm-xserve04 (ppc)        
qm-xserve03 (intel) - not being used by QA anymore; 


missing/lost?
=============
bm-stage-osx-01 (mini) - phong: was that used for something else (jhford?)
bm-xserve07 - MoMo using? joduinn emailed dasher/gozar 26feb
bm-xserve09 - phong: please verify if powered/racked in MPT
bm-xserve10 - phong: please verify if powered/racked in MPT
qm-xserve04 - QA using? joduinn emailed QA 26feb
qm-xserve05 - QA using? joduinn emailed QA 26feb


ok:
===
bm-xserve01 (ppc) TB2 production
bm-xserve02 (ppc) hot-standby for bm-xserve01
bm-xserve05 (ppc)
bm-xserve08 (ppc)
No-one loves the inventory ...

(In reply to comment #16)
> xserves to reimage and switch to new branch / geriatric
> =======================================================
> bm-xserve03 (ppc)
> bm-xserve04 (ppc)        

I don't we think we should put ppc machines into the moz2 pool. It's probably another ref image, and a slightly different cross compile from the Intel boxes. Much better fit to put them on geriatric tests.

> missing/lost?
> =============
> bm-xserve07 - MoMo using? joduinn emailed dasher/gozar 26feb
> bm-xserve09 - phong: please verify if powered/racked in MPT

Both of these are connected to pm01 already. Should be left alone.

> bm-xserve10 - phong: please verify if powered/racked in MPT

This is fx-mac-1.9-slave2, the production release machine Fx3.0 releases. Should be left alone for the moment.

> ok:
> ===
> bm-xserve01 (ppc) TB2 production

My memory was that this was the PPC spare. Used to be used for partner builds. Log into it and it thinks it's bm-xserve02, which implies it was reimaged from that. I've set the hostname to bm-xserve01.

> bm-xserve02 (ppc) hot-standby for bm-xserve01

idle ? 

> bm-xserve05 (ppc)

This is being used for Tb2 nightly and release builds (bug 491077).

> bm-xserve08 (ppc)

Intel box doing Fx3.0 nightly builds, dep and debug.
(In reply to comment #17)
> No-one loves the inventory ...
> 
> (In reply to comment #16)
> > xserves to reimage and switch to new branch / geriatric
> > =======================================================
> > bm-xserve03 (ppc)
> > bm-xserve04 (ppc)        
> 
> I don't we think we should put ppc machines into the moz2 pool. It's probably
> another ref image, and a slightly different cross compile from the Intel boxes.
> Much better fit to put them on geriatric tests.
Agreed. Filed bug#549559 to track moving them to geriatric-master.


> > missing/lost?
> > =============
> > bm-xserve07 - MoMo using? joduinn emailed dasher/gozar 26feb
gozer confirmed this is handed back. nthomas confirmed already reused in production.

> > bm-xserve09 - phong: please verify if powered/racked in MPT
> Both of these are connected to pm01 already. Should be left alone.
Inventory fixed for bm-xserve07,09.


> > bm-xserve10 - phong: please verify if powered/racked in MPT
> This is fx-mac-1.9-slave2, the production release machine Fx3.0 releases.
> Should be left alone for the moment.
ok.



> > ok:
> > ===
> > bm-xserve01 (ppc) TB2 production
> 
> My memory was that this was the PPC spare. Used to be used for partner builds.
> Log into it and it thinks it's bm-xserve02, which implies it was reimaged from
> that. I've set the hostname to bm-xserve01.
> > bm-xserve02 (ppc) hot-standby for bm-xserve01
> idle ? 
bm-xserve01/02 are the live TB2 production build machine, and the hot-swap-backup. Once we EOL TB2, we can kick these over to geriatric-master, but for now, they remain tier1 production machines. Inventory updated.


> > bm-xserve05 (ppc)
> This is being used for Tb2 nightly and release builds (bug 491077).
I went to fix inventory, and found it was already updated - was that magic-work by nthomas ?


> > bm-xserve08 (ppc)
> Intel box doing Fx3.0 nightly builds, dep and debug.
ok. (Again, I find inventory updated - thank you nthomas?)
(In reply to comment #16)
> (In reply to comment #15)
> > Not sure what is left to do here.
> Took some digging, but the summary is:
> 
> minis to reimage
> ================
> qm-image-master (mini)
> try-pmac-unit-01 (mini)
> qm-mini-centos01
> qm-mini-centos02
bug#549570 to track bringing these 4 minis up in talos staging.


> xserves to reimage and switch to new branch / geriatric
> =======================================================
> bm-xserve03 (ppc)
> bm-xserve04 (ppc)        
bug#549559 tracking moving these two machines to geriatric-master.

> qm-xserve03 (intel) - not being used by QA anymore; 
bug#523726 tracked qm-xserve03 being reimaged as bm-xserve09 and running it in production. Inventory fixed.


> missing/lost?
> =============
> bm-stage-osx-01 (mini) - phong: was that used for something else (jhford?)
Unknown; still investigating.

> bm-xserve07 - MoMo using? joduinn emailed dasher/gozar 26feb
Already used in production - inventory fixed.

> bm-xserve09 - phong: please verify if powered/racked in MPT
Already used in production - inventory fixed.

> bm-xserve10 - phong: please verify if powered/racked in MPT
Per nthomas, bm-xserve10 is now fx-mac-1.9-slave2. Inventory fixed.

> qm-xserve04 - QA using? joduinn emailed QA 26feb
> qm-xserve05 - QA using? joduinn emailed QA 26feb
Unknown; still investigating.

> 
> 
> ok:
> ===
> bm-xserve01 (ppc) TB2 production
> bm-xserve02 (ppc) hot-standby for bm-xserve01
> bm-xserve05 (ppc)
Already used in production - inventory fixed.
> bm-xserve08 (ppc)
Already used in production - inventory fixed.
(In reply to comment #16)
> (In reply to comment #15)
> > Not sure what is left to do here.
> Took some digging, but the summary is:
> 
> minis to reimage
> ================
> qm-image-master (mini)
> try-pmac-unit-01 (mini)
> qm-mini-centos01
> qm-mini-centos02
bug#549570 to track bringing these 4 minis up in talos staging.


> xserves to reimage and switch to new branch / geriatric
> =======================================================
> bm-xserve03 (ppc)
> bm-xserve04 (ppc)        
bug#549559 tracking moving these two machines to geriatric-master.

> qm-xserve03 (intel) - not being used by QA anymore; 
bug#523726 tracked qm-xserve03 being reimaged as bm-xserve09 and running it in production. Inventory fixed.


> missing/lost?
> =============
> bm-stage-osx-01 (mini) - phong: was that used for something else (jhford?)
Unknown; still investigating.

> bm-xserve07 - MoMo using? joduinn emailed dasher/gozar 26feb
Already used in production - inventory fixed.

> bm-xserve09 - phong: please verify if powered/racked in MPT
Already used in production - inventory fixed.

> bm-xserve10 - phong: please verify if powered/racked in MPT
Per nthomas, bm-xserve10 is now fx-mac-1.9-slave2. Inventory fixed.

> qm-xserve04 - QA using? joduinn emailed QA 26feb
> qm-xserve05 - QA using? joduinn emailed QA 26feb
Unknown; still investigating.

> 
> 
> ok:
> ===
> bm-xserve01 (ppc) TB2 production
> bm-xserve02 (ppc) hot-standby for bm-xserve01
> bm-xserve05 (ppc)
Already used in production - inventory fixed.
> bm-xserve08 (ppc)
Already used in production - inventory fixed.
bm-xserve08,10,20 are being reimaged in bug#580346.

bm-xserve01,02,05 are being tracked in bug#581210, bug#578234
Depends on: 581210, 578234, 580346
Summary: Put existing hardware to better use → [Tracking bug] Put existing hardware to better use
(In reply to comment #20)
> > missing/lost?
> > =============
...
> > qm-xserve04 - QA using? joduinn emailed QA 26feb
> > qm-xserve05 - QA using? joduinn emailed QA 26feb
> Unknown; still investigating.

From email with QA in feb/mar 2009, these are not known, not being used. According to inventory, these are intel based xserves. Now bug#580346 tracks have them reimaged, renamed to bm-xserve23,24 and added to Tryserver.
Depends on: 609742
Whiteboard: [buildslaves][capacity]
What's left to do here?  It looks like this just entails tossing three slaves (one of which is down atm) onto geriatric master - bug 578234?
(In reply to comment #24)
> What's left to do here?  It looks like this just entails tossing three slaves
> (one of which is down atm) onto geriatric master - bug 578234?

Yep, thats all that is left to do here - I've already verified that all the other issues described here, and other dep-bugs here are all handled. 

Out of good-bug-manners, I didnt want to close this tracking bug, until all the tracked-dep-bugs were closed. I'll nudge in the one remaining depbug.
All dependent bugs are fixed. Resolving.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: