890317 - (t-w732-ix-064) t-w732-ix-064 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[[e-mail] (:bhearsum)]

trying pdu reboot

Comment 1

[[e-mail] (:bhearsum)]

didn't work, needs it help

Comment 2

[Justin Wood (:Callek)]

Per 890333 the nic is still being "weird" but it is taking jobs right now.

I gave the OK to it to yank it as needed, I've disabled in slavealloc so future boots don't start jobs.

Comment 3

[Carsten Book [:Tomcat]]

cc'ing the sheriffs in case the issue mentioned in comment 2 gets on our radars

Comment 4

[Justin Wood (:Callek)]

re-enabled in slavealloc

Comment 5

[[e-mail] (:bhearsum)]

Slave is ready for production...but didn't come back from a reboot.

Comment 6

[Phil Ringnalda (:philor)]

Slave is in production, but not ready for it - failing webgl tests, timing out xperf tests, timing out cloning talos, generally acting like even after a reimage to catch it up with all the things it missed over the last two months, it'll still be in pretty bad shape.

Disabled in slavealloc.

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Phil Ringnalda (:philor) from comment #6)
> Slave is in production, but not ready for it - failing webgl tests, timing
> out xperf tests, timing out cloning talos, generally acting like even after
> a reimage to catch it up with all the things it missed over the last two
> months, it'll still be in pretty bad shape.
> 
> Disabled in slavealloc.

Cheers. Back to diagnostics.

Comment 8

[[e-mail] (:bhearsum)]

Diagnostics showed no errors. Not sure what to do now.

Comment 9

[Justin Wood (:Callek)]

Since diags did a reimage and saw no errors I tried a reboot, but that failed too, so we'll need human touch again anyway.

[[e-mail]3 ~]$ for i in 064; do curl http://slaveapi-dev1.srv.releng.scl3.mozilla.com:8000/slave/t-w732-ix-$i/action/reboot; done
{
  "reboots": {
    "55975568": {
      "state": 3,
      "text": "Attempting SSH reboot...Failed.\nAttempting IPMI reboot...Failed.\nCan't do
 anything else, human intervention needed."
    }
  }
}

Comment 10

[Armen [:armenzg]]

I rebooted it again.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Still having problems. Disabled.
https://tbpl.mozilla.org/php/getParsedLog.php?id=29988866&tree=Mozilla-Inbound

Comment 12

[Armen [:armenzg]]

Attachment: Screen Shot 2013-11-01 at 4.59.51 PM.png

2 monitors.
dxdiag shows acceleration disabled.

Need IT's intervention.

Comment 13

[[e-mail] (:bhearsum)]

The graphics setup should be correct now - I put the machine back in production.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Sure about that?
https://tbpl.mozilla.org/php/getParsedLog.php?id=30437374&tree=Mozilla-Central

Disabled.

Comment 15

[[e-mail] (:bhearsum)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #14)
> Sure about that?
> https://tbpl.mozilla.org/php/getParsedLog.php?id=30437374&tree=Mozilla-
> Central
> 
> Disabled.

Of course I'm not.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Definitely not a one-off fluke.
https://tbpl.mozilla.org/php/getParsedLog.php?id=30438553&tree=Mozilla-Aurora

Comment 17

[Armen [:armenzg]]

Hard drive has been replaced and the machine has been re-imaged.
Back into production.

Comment 18

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1188656)

Comment 19

[Phil Ringnalda (:philor)]

Reenabled after another disk replacement/reimage.

Comment 20

[Phil Ringnalda (:philor)]

But after multiple reboots, it still isn't taking jobs.

Comment 21

[Alin Selagea [:aselagea]]

Re-imaged the slave enabled it in slavealloc. At the moment, it has already successfully completed two jobs.