933901 - (t-w732-ix-117) t-w732-ix-117 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

Needs human intervention!

Armens-MacBook-Air ~ $ curl http://slaveapi-dev1.srv.releng.scl3.mozilla.com:8080/slave/talos-r3-w7-117/action/reboot;
{
  "reboots": {
    "58205200": {
      "state": 3, 
      "text": "Attempting SSH reboot...Failed.\nCan't do anything else, human intervention needed."
    }
  }
}

Comment 1

[Armen [:armenzg]]

It's back!

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Burning everything it touches, even after a reboot.
https://treeherder.mozilla.org/logviewer.html#?job_id=2637423&repo=b2g-inbound

Comment 3

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1200531)

Comment 4

[Phil Ringnalda (:philor)]

Not reenabling, since it's not clear why it was rebooted.

Comment 5

[Vlad Ciobancai [:vladC] [:vciobancai]]

Re-imaged and enabled in slavealloc

Comment 6

[Phil Ringnalda (:philor)]

Graphics problems, https://treeherder.mozilla.org/logviewer.html#?job_id=14131551&repo=mozilla-inbound, redisabled.

Comment 7

[Phil Ringnalda (:philor)]

One of only five slaves to have ever hit bug 1199709. The graphics on this slave are busted. Disabled.

Comment 8

[Alin Selagea [:aselagea]]

Enabled in slavealloc after having its video card replaced and been re-imaged. Will keep an eye on it.

Comment 9

[Phil Ringnalda (:philor)]

The bug 1207160 thing finally got to be too much for me to take, disabled.

Comment 10

[Phil Ringnalda (:philor)]

Quite probably working now (and at any rate, enabled which means FIXED whether or not it is).

Comment 11

[Phil Ringnalda (:philor)]

And disabled again, it's in the middle of the second spell it has had recently of being unable to find c:\mozilla-build\python27\python27.dll to copy it to the venv for several jobs in a row, like http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/1444075836/mozilla-central_win7-ix_test-mochitest-e10s-browser-chrome-4-bm119-tests1-windows-build38.txt.gz

Comment 12

[Alin Selagea [:aselagea]]

Tried another re-image and enabled the machine in slavealloc after that..will keep an eye on it during the next days to see how it goes.

Comment 13

[Alin Selagea [:aselagea]]

Looking good at the moment, as it completes most of the jobs it takes without issues.

Comment 14

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1299771)

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs.

Comment 16

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1304690)

Comment 17

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1305687)

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs.

Comment 19

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1446994)

Comment 20

[Adrian Pop]

Checked on slave alloc and slave health > the machine seems to be disabled.
Is this machine still needed ? can we close the ticket ?

Comment 21

[Phil Ringnalda (:philor)]

Never much liked this one.