1095059 - (b-2008-ix-0078) b-2008-ix-0078 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ryan VanderMeulen [:RyanVM]]

Corrupted builds and an awful-looking recent health. Disabled for reimaging.
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=379008&repo=mozilla-aurora

Comment 1

[Phil Ringnalda (:philor)]

Diagnostics saw nothing, maybe the reimage helped. Reenabled.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Still seems pretty burntastic to me. I've seen a couple like the one below since it was put back into production:
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=31332&repo=mozilla-b2g34_v2_1

Comment 3

[Chris Cooper [:coop] (he/him)]

Re-imaged. If we still see failures, we'll try another pass of diagnostics.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=59297&repo=mozilla-b2g34_v2_1

 LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt

Comment 5

[Chris Cooper [:coop] (he/him)]

Re-imaged and returned to production.

Comment 6

[Chris Cooper [:coop] (he/him)]

Re-imaged and re-enabled.

Comment 7

[Amy Rich [:arr] [:arich]]

This host has been showing as down for 33 days. I'm not sure why slaveapi hasn't filed a bug for it?

Comment 8

[Phil Ringnalda (:philor)]

It's now a staging slave, in the dev/pp environment and attached to dev-bhearsum-builds, so slaveapi has no reason to touch it.

Comment 9

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting IPMI reboot...Failed.
Filed IT bug for reboot (bug 1139927)

Comment 10

[Chris Cooper [:coop] (he/him)]

Re-imaged and returned to the production environment.

Comment 11

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=9886626
repository: mozilla-inbound
start_time: 2015-05-15T12:49:49
who: rvandermeulen[at]mozilla[dot]com
machine: b-2008-ix-0078
buildname: WINNT 5.2 mozilla-inbound leak test build
revision: 55bbf00eb10a

exceptions.IOError: [Errno 28] No space left on device

Comment 12

[Ryan VanderMeulen [:RyanVM]]

This is burning Windows nightly jobs:
https://treeherder.mozilla.org/logviewer.html#?job_id=877823&repo=mozilla-aurora

I've disabled it in slavealloc.

Comment 13

[Mark Cornmesser [:markco] OOO 2024/04/15]

I am looking into this. I am not seeing a clear cut reason for the failure. There is the "03:48:52 INFO - IOError: [Errno 13] Permission denied: 'c:/builds/crash-stats-api.token'" error, but the permissions are consistent with other machines. 

RyanVM: any thoughts on why it burned?

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Nope. I asked Ted about it and he said it was a slave issue.

Comment 15

[Phil Ringnalda (:philor)]

Are the running processes consistent with other machines? "I downloaded a file, and when I went to use it I got a permission denied error" on a Windows machine, my first thought is AV or search indexing grabbing onto the newly-created file, including maybe something that is running on other machines too, but on other machines has been told to keep its hands off c:/builds/.

Comment 16

[Mark Cornmesser [:markco] OOO 2024/04/15]

This machine, due to troubleshooting, had puppet ran by 2 different local accounts, system and cltbld. I suspect that may have caused an issue with permissions.  I am going to reimage the machine and reenable it.

Comment 17

[Alin Selagea [:aselagea]]

Re-imaged slave, did not enable it in slavealloc.