940092 - (t-xp32-ix-092) t-xp32-ix-092 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

Has something stuck in its craw that it can't rm, e.g.

https://tbpl.mozilla.org/php/getParsedLog.php?id=30726201&tree=Mozilla-Aurora

15:17:25     INFO - rmtree: C:\slave\test\build
15:17:25     INFO - Using _rmtree_windows ...
15:17:25     INFO - retry: Calling <bound method DesktopUnittest._rmtree_windows of <__main__.DesktopUnittest object at 0x01071690>> with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #1
15:17:25     INFO - retry: Failed, sleeping 60 seconds before retrying
15:18:24     INFO - retry: Calling <bound method DesktopUnittest._rmtree_windows of <__main__.DesktopUnittest object at 0x01071690>> with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #2
15:18:24     INFO - retry: Failed, sleeping 120 seconds before retrying
15:20:24     INFO - retry: Calling <bound method DesktopUnittest._rmtree_windows of <__main__.DesktopUnittest object at 0x01071690>> with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #3
15:20:24     INFO - retry: Failed, sleeping 240 seconds before retrying
15:24:23     INFO - retry: Calling <bound method DesktopUnittest._rmtree_windows of <__main__.DesktopUnittest object at 0x01071690>> with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #4
15:24:23     INFO - retry: Failed, sleeping 300 seconds before retrying
15:29:23     INFO - retry: Calling <bound method DesktopUnittest._rmtree_windows of <__main__.DesktopUnittest object at 0x01071690>> with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #5
15:29:23    FATAL - Unable to remove C:\slave\test\build!
15:29:23    FATAL - Caught exception: (5, 'SetFileAttributesW', 'Access is denied.')
15:29:23    FATAL - Caught exception: (5, 'SetFileAttributesW', 'Access is denied.')
15:29:23    FATAL - Caught exception: (5, 'SetFileAttributesW', 'Access is denied.')
15:29:23    FATAL - Caught exception: (5, 'SetFileAttributesW', 'Access is denied.')
15:29:23    FATAL - Caught exception: (5, 'SetFileAttributesW', 'Access is denied.')
15:29:23    FATAL - Running post_fatal callback...
15:29:23    FATAL - Exiting -1


Disabled in slavealloc.

Comment 1

[Chris Cooper [:coop] (he/him)]

I've rebooted this slave and it's currently running chkdsk on boot.

Comment 2

[Chris Cooper [:coop] (he/him)]

Back in production. Jobs are running green.

Comment 3

[Phil Ringnalda (:philor)]

Those were good times; now it's https://tbpl.mozilla.org/php/getParsedLog.php?id=31019596&tree=Mozilla-Inbound, "Caught exception: (1392, 'FindFirstFile', 'The file or directory is corrupted and unreadable.')"

Disabled in slavealloc.

Comment 4

[Armen [:armenzg]]

Re-image requested.

Comment 5

[Armen [:armenzg]]

Rebooted into production.

Comment 6

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1167580)