963197 - (b-2008-ix-0001) b-2008-ix-0001 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[[e-mail] (:bhearsum)]

This shouldn't have been opened because SlaveAPI managed to reboot through IPMI. I filed bug 96204 to fix this.

Comment 2

[Carsten Book [:Tomcat]]

seems slave is still taking jobs while disabled

Comment 3

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=193348&repo=mozilla-beta

 LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or corrupt

Comment 4

[Chris Cooper [:coop] (he/him)]

Re-imaged and returned to production.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Disabled for the various ways it falls under bug 1122975.

Comment 6

[Phil Ringnalda (:philor)]

Reenabled with 8GB.

Comment 7

[Phil Ringnalda (:philor)]

Nice work, slave, timed out in the "set props: purge_target" step.

Disabled.

Comment 8

[Phil Ringnalda (:philor)]

Interesting, its partner in crime, b-2008-ix-0005, also failed purging on its first job, but managed to show some output so I could see it was stuck having to remove dozens of objdirs, so I let it try again, and after one more failure it has managed several successful builds.

Entirely possible that's saying we have some which not only had too little RAM but also have too slow disk, but it might just be saying that we cannot leave a Windows build slave disabled for two weeks and then put it back in production without either reimaging or manually removing all the expired objdirs, because they just aren't fast enough at removing them to get through several dozen during a production job.

Comment 9

[Jordan Lund (:jlund)]

I think it's the latter in that all our builddirs got 'old' and purge is trying to collect them all. I believe there is an effort to change purge behaviour but for now, I'll re-image this slave

Comment 10

[Chris Cooper [:coop] (he/him)]

Re-imaged and returned to production.

Comment 11

[Alin Selagea [:aselagea]]

Re-imaged slave, did not enable it in slavealloc.