1004815 - (t-w864-ix-026) t-w864-ix-026 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Comment 1

[Phil Ringnalda (:philor)]

Seems to be enabled and working, for some time.

Comment 2

[Phil Ringnalda (:philor)]

Got stuck trying and failing to remove "C:\slave\test\scripts\.hg\store\data\mozharness\mozilla" and thus infinitely retrying.

Disabled in slavealloc.

Comment 3

[Carsten Book [:Tomcat]]

(In reply to Phil Ringnalda (:philor) from comment #2)
> Got stuck trying and failing to remove
> "C:\slave\test\scripts\.hg\store\data\mozharness\mozilla" and thus
> infinitely retrying.
> 
> Disabled in slavealloc.

somehow this slave was still enabled - disabled again

Comment 4

[Q]

re-imaging host

Comment 5

[Q]

reimage went well with no manual intervention

Comment 6

[Chris Cooper [:coop] (he/him)]

(In reply to Q from comment #5)
> reimage went well with no manual intervention

Back in production.

Comment 7

[Armen [:armenzg]]

It can't reboot and it has been burning every job.
I stopped buildbot.
I also disabled slavealloc.

'c:\\mozilla-build\\hg\\hg' 'clone' 'https://hg.mozilla.org/build/ash-mozharness' 'scripts'
 in dir C:\slave\test\. (timeout 1320 secs)
...
program finished with exit code -1073741800

c:/mozilla-build/python27/python: can't open file 'scripts/external_tools/count_and_reboot.py': [Errno 2] No such file or directory

Comment 8

[Phil Ringnalda (:philor)]

That (common, unexamined) state just needs a reboot to fix it - it can still reboot just fine, it just can't count_and_reboot since it failed to check out that file which would allow it to fix itself.

Rebooted and reenabled.

Comment 9

[Phil Ringnalda (:philor)]

Try push left it with a file it can't remove in C:\slave\test\build, disabled.

Comment 10

[Vlad Ciobancai [:vladC] [:vciobancai]]

The slave was re-imaged by Dcops and enabled in slavealloc

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/logviewer.html#?job_id=1204324&repo=mozilla-beta#L495

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Re-imaged and re-enabled in slvealloc,I will keep monitoring this slave to see if now will take jobs.

Comment 13

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1305609)

Comment 14

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1305610)

Comment 15

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs.