876514 - t-xp32-ix-099 and t-w732-ix-100 seem to not come back up

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Armen [:armenzg]]

I tried to power cycle them or turn them on but it did not work.

Would you please have a look at them when you have a chance?
Thanks!

Comment 1

[Q]

We will take a look.

Comment 2

[Q]

I was able to get the ACPI power state to change and a boot to start after a cold bmc reset on both machines. However it looks like they both hung back last night and did the same again this morning. It does indeed look like a colo trip for DCops

Comment 3

[Van Le [:van]]

I was unable to get video output from either servers using the onboard and external video adapters. Opened RMA cases with iX systems.

Comment 4

[Hubert Eulalia [:Hubear]]

T-w732-ix-100 has been installed and back online, would you like us to re-image this host?

Unfortunately, t-xp32-ix-099 wouldn't turn on. We will open a RMA case to repair this host.

Comment 5

[Amy Rich [:arr] [:arich]]

Q is taking care of reimaging t-w732-ix-100

Comment 6

[Van Le [:van]]

received update today: Apologies for the delays, Van but we had to replace the blade connector arm.
Everything looks well again and the node is consistently powering on for us. 

Server should arrive today or early tomorrow.

Comment 7

[Van Le [:van]]

blade installed but still doesn't power on. Reported issue to iX again.

Comment 8

[Salvador Espinoza [:sal]]

IX tech is taking node t-xp32-ix-099 back for repairs, not sure what the issue is.

Comment 9

[Armen [:armenzg]]

Can you please have a look to t-w732-ix-100?

I can reboot it but I can't see anything through the console redirection?
I also see this message:
"2012/07/01 22:37:42	Chassis Intru 	Physical Security (Chassis Intrusion)	General Chassis Intrusion - Asserted"

Is there something wrong with the date/time as well?

Comment 10

[Van Le [:van]]

No POST. Reopened ticket LVV-546247

Comment 11

[Ashlee Nguyen [:ashlee]]

This node was picked up by iX. 
They will order another node today for replacement and I will update the bug with ETA.

Comment 12

[Hubert Eulalia [:Hubear]]

2 IX systems node has arrived and is back online for t-xp32-ix-099 and t-w732-ix-100, will configure IPMI. Will update when completed.

Comment 13

[Van Le [:van]]

we have to send t-w732-ix-100 back again. I'm reimaging t-xp32-ix-099 and will let you guys know when it's finished.

Comment 14

[Van Le [:van]]

t-xp32-ix-099 has been updated in inventory with new nic[01] MAC addy and reimaged. please let me know if there are any persisting issues. I'm going to drop off t-w732-ix-100 for them so we can get this resolved ASAP as this bug has been opened for over 2 months.

[[e-mail]3 ~]$ fping !$
fping t-xp32-ix-099.wintest.releng.scl3.mozilla.com
t-xp32-ix-099.wintest.releng.scl3.mozilla.com is alive
[[e-mail]3 ~]$ ssh !$
ssh t-xp32-ix-099.wintest.releng.scl3.mozilla.com
The authenticity of host 't-xp32-ix-099.wintest.releng.scl3.mozilla.com (10.26.41.119)' can't be established.
RSA key fingerprint is db:f1:c0:08:dc:38:26:b5:14:96:93:3f:a2:68:0e:c6.
Are you sure you want to continue connecting (yes/no)?

Comment 15

[Van Le [:van]]

t-w732-ix-100 had a bad video 3rd party card. ETA is Wednesday for the RMA.

Comment 16

[Van Le [:van]]

blah, 3rd party video card*

Comment 17

[Amy Rich [:arr] [:arich]]

We should have some of those in stock at scl3 which you can swap in (we ordered some just for this purpose).  :}

Comment 18

[Van Le [:van]]

:arr, the host didn't power on again, no POST, so I wasn't sure what the issue was until they sent me an update.

t-w732-ix-100 has been updated in inventory, reimaged and back online. video output redirected to 3rd party video card.

[[e-mail]3 ~]$ fping !$
fping t-w732-ix-100.wintest.releng.scl3.mozilla.com
t-w732-ix-100.wintest.releng.scl3.mozilla.com is alive
[[e-mail]3 ~]$ ssh !$
ssh t-w732-ix-100.wintest.releng.scl3.mozilla.com
The authenticity of host 't-w732-ix-100.wintest.releng.scl3.mozilla.com (10.26.40.250)' can't be established.
RSA key fingerprint is 61:72:ab:fb:45:4d:dd:b3:f9:8b:0a:dd:4e:19:19:a0.
Are you sure you want to continue connecting (yes/no)?