886195 - (t-w864-ix-032) t-w864-ix-032 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Justin Wood (:Callek)]

Currently failing to clone tools repo and burning through jobs...

No output but:
program finished with exit code -1073741800

Gracefulled in the buildmaster and disabled in slavealloc. In armens absence needinfo to coop because I don't know what to do next here.

Comment 1

[Chris Cooper [:coop] (he/him)]

I'll send it for diagnostics.

Comment 2

[[e-mail] (:bhearsum)]

Got a reimage, let's try again.

Comment 3

[Phil Ringnalda (:philor)]

Managed to stay busted after the bug 1180877 event today.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Re-enabled after switch fix in bug 1181615, and rebooted back into production.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Burning jobs left and right since being re-enabled.
https://treeherder.mozilla.org/logviewer.html#?job_id=11698425&repo=mozilla-inbound

Comment 6

[Justin Wood (:Callek)]

Host being reimaged

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Has burned multiple runs with the same error.
https://treeherder.mozilla.org/logviewer.html#?job_id=438877&repo=mozilla-beta

Comment 8

[Phil Ringnalda (:philor)]

Reenabled after disk utilities claims to have fixed some bad sectors.

Comment 9

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1212008)

Comment 10

[Phil Ringnalda (:philor)]

It stayed enabled through the reimage it got in bug 1212008, and the second job it took was http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/b2g-inbound-win64-pgo/1444145469/b2g-inbound_win8_64_test_pgo-mochitest-other-bm109-tests1-windows-build255.txt.gz completely failing to manage to do webgl. Disabled.

Comment 11

[Phil Ringnalda (:philor)]

Gambling that this will be the monitor-attached problem, which should hypothetically be fixed by the several times I've rebooted it since.

Comment 12

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1246102)

Comment 13

[Alin Selagea [:aselagea]]

Rebooted the slave via KVM and re-imaged it afterwards as it was getting stuck on attempting to repair screen. Already started taking jobs.

Comment 14

[Phil Ringnalda (:philor)]

Try push left it with a file it can't remove in C:\slave\test\build, disabled.

Comment 15

[Vlad Ciobancai [:vladC] [:vciobancai]]

The slave was re-imaged by Dcops and enabled in slavealloc

Comment 16

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1291052)

Comment 17

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1309001)

Comment 18

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs.

Comment 19

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1337107)

Comment 20

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online and taking jobs after been re-imaged by DCOps team.