730545 - (talos-r4-lion-058) talos-r4-lion-058 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

+++ This bug was initially created as a clone of Bug #728535 +++

Same symptom as bug 728535, when talos-r4-snow-007 took to saying that every file in rm -rf tools was an invalid argument, and then dying download builds, saying Cannot write to `firefox-13.0a1.en-US.mac.dmg' (Invalid argument).

Anyway, it's chewing up jobs (what jobs there are to chew right now) like crazy because it only takes a couple of seconds to fail to rm and then fail to save a downloaded build.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Disabled in slavealloc, and did a graceful shutdown on the master. Should in the naughty corner now.

Comment 2

[[e-mail] (:bhearsum)]

Back in the production pool.

Comment 3

[Jake Watkins [:dividehex]]

This mini has been repaired, reimaged and placed back in scl1.  It is ready to be placed in production.

Comment 4

[Chris AtLee [:catlee]]

Back in production

Comment 5

[Phil Ringnalda (:philor)]

Last job was 8 days ago.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Power off and on again as it was refusing an ssh connection.

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14658404&tree=Ionmonkey

"inflating: reftest/tests/layout/reftests/fonts/dejavu-sans/DejaVuSans-Oblique.ttf   bad CRC 21b8d878  (should be 0b256820)"

Expect me to be coming around claiming that the disk is bad before too much longer.

Comment 8

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14758821&tree=Mozilla-Inbound

Error 1000 (image data corrupted).
calculated CRC32 $C8359846, expected   CRC32 $269CA257

Comment 9

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=14760663&tree=Ionmonkey

  inflating: reftest/tests/layout/reftests/fonts/dejavu-sans/DejaVuSans.ttf   bad CRC 21e182a9  (should be d8a4d667)

Needs hardware diagnostics, I fear.

Comment 10

[Armen [:armenzg]]

comment 3 suggested that this slave was repaired 4 months ago and no one reported any issues. Perhaps it just broke again and that is why it was down for several days?

Comment 11

[Armen [:armenzg]]

Added a note to slavealloc and disabled the slave.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Back into production now the RAM has been replaced.

Comment 13

[Phil Ringnalda (:philor)]

Made it 6 jobs before https://tbpl.mozilla.org/php/getParsedLog.php?id=15309593&tree=Mozilla-Inbound where it apparently went into a coma that was forcibly ended by a reconfig.

Comment 14

[Van Le [:van]]

host had memory issues and ram was replaced but looks like issues persisting. will need to bring to apple certified tech in desktop to rerun diagnostics.

Bug 794184 opened to track.

Comment 15

[Phil Ringnalda (:philor)]

I don't actually want to ever see this slave back again, but I know I will anyway, and I feel duty-bound to comment before acking the nagios alert that's been going off forever now: bug 794184 comment 2 claimed there was nothing wrong with this clearly broken slave, and that it was going back into production, but hasn't been heard from in the 6 weeks since, so somebody probably ought to do something so we can get back to reimaging it every few days.

Comment 16

[Amy Rich [:arr] [:arich]]

Did anyone ever actually put this back into production or test it out after the RAM was replaced?

Comment 17

[Phil Ringnalda (:philor)]

The RAM replacement was 2012-09-14, went back into production 2012-09-17, only did six jobs and then died, no?

Comment 18

[Amy Rich [:arr] [:arich]]

philor: Ah, okay, I was misreading the timeline and thought that the RAM had been replaced after comment 14.  I've asked dcops to run some more diags on it to see if they can find something in addition to Apple's own hardware diags (which come up clean).

Comment 19

[Chris Cooper [:coop] (he/him)]

Diagnostics were run and came up clean. Returning to production. 

If it fails again, it will get a sharper hook.

Comment 20

[Amy Rich [:arr] [:arich]]

coop: We've run extensive diagnostics and they've repeatedly come up clean.  Despite that, philor says it burns jobs almost immediately when put back into production (see comment 15).  I'm not sure where to go from here.

Comment 21

[Chris Cooper [:coop] (he/him)]

(In reply to Amy Rich [:arich] [:arr] from comment #20)
> coop: We've run extensive diagnostics and they've repeatedly come up clean. 
> Despite that, philor says it burns jobs almost immediately when put back
> into production (see comment 15).  I'm not sure where to go from here.

Filed bug 818467 to get this mini re-imaged. One last try before decommissioning.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Down again. IRC conversation concluded that we should decommission ?

Comment 23

[Amy Rich [:arr] [:arich]]

nthomas: yeah, hardware diags show nothing wrong, we've reimaged it, and all it does is burn jobs and go down.

Comment 24

[[e-mail] (:bhearsum)]

Attachment: remove from buildbot-configs

Comment 25

[[e-mail] (:bhearsum)]

Attachment: remove from puppet

Comment 26

[Justin Wood (:Callek)]

Hey ben, can we get these patches landed please :-)

--> removing from buildduty queue

Comment 27

[[e-mail] (:bhearsum)]

(In reply to Justin Wood (:Callek) from comment #26)
> Hey ben, can we get these patches landed please :-)
> 
> --> removing from buildduty queue

You could've just landed these yourself, but OK...

Comment 28

[Rail Aliiev [:rail]]

in production

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Decommissioned in bug 820115.