797245 - (talos-r4-lion-063) talos-r4-lion-063 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

Please disable this slave and have hardware diagnostics run on it, particularly RAM tests - we've had this bug 795215 "pink pixel of death" problem for a while, where we'll get intermittent failures in a ton of reftests and the difference is that one canvas has a single pixel which is pink rather than white (a single bit of difference), but once I filed it so we could see that it was actually talos-r4-lion-063 twice in a row, hello bad RAM.

Comment 1

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15793995&tree=Mozilla-Inbound

Comment 2

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15789651&tree=Mozilla-Aurora

Comment 3

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15780634&tree=Profiling

Comment 4

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15806775&tree=Mozilla-Inbound

Comment 5

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=15809478&tree=Mozilla-Inbound

Comment 6

[Armen [:armenzg]]

Putting back into the pool.
This slave might have issues even if hardware diagnostics did not catch anything.
If it gives us trouble we should dig into the problem and try to find a test case.
If such test case is found we should put the slave in bug 712206.
At that point we can ask re-imaging and try again.
If that fails again we should decommission the slave.

Comment 7

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17848078&tree=Mozilla-Aurora is the same old pink pixel of death. So is https://tbpl.mozilla.org/php/getParsedLog.php?id=17878263&tree=Ionmonkey.

Test case? Write the same large number of values to memory twice, read them, make sure they are still the same when you read them. Although there's something in the reftest harness about "reusing canvases" that may mean it's partially more like writing a big block of zeros, overwriting a small part with non-zero data, then reading it over and over again and making sure none of the zeros become non-zero.

Comment 8

[Phil Ringnalda (:philor)]

Nice, now it has burned 12 jobs in a row, "hdiutil: attach failed - Device not configured"

Comment 9

[Aki Sasaki (not active)]

[16:15]	<philor>	who's on sledgehammerduty?
[16:17]	<philor>	that damn talos-r4-lion-063, which has hardware problems no matter how many inadequate runs of inadequate hardware diagnostics it gets, has burned its last dozen runs

Disabled in slavealloc

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17960391&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17971048&tree=Mozilla-Aurora

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17972053&tree=Mozilla-Beta

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17973046&tree=Mozilla-Beta

Comment 13

[Jim Mathies [:jimm]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=17974516&tree=Elm&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=17975033&tree=Elm&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=17974855&tree=Elm&full=1

Comment 14

[Justin Wood (:Callek)]

(In reply to Aki Sasaki [:aki] from comment #9)
> Disabled in slavealloc

And it managed to dodge the sledgehammer:

talos-r4-lion-063:~ cltbld$ uptime
11:26  up 19:51, 3 users, load averages: 0.66 0.58 0.53

I manually restarted teh slave now, so it should properly come back disabled.

Comment 15

[[e-mail] (:bhearsum)]

We should decomm this slave. Hardware diagnostics show no problems with it, but it keeps burning jobs.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: remove from puppet

Comment 17

[[e-mail] (:bhearsum)]

Attachment: remove from buildbot

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 699981 [details] [diff] [review]
remove from buildbot

I think it'll conflict with https://bugzilla.mozilla.org/attachment.cgi?id=699975&action=edit

Comment 19

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #18)
> Comment on attachment 699981 [details] [diff] [review]
> remove from buildbot
> 
> I think it'll conflict with
> https://bugzilla.mozilla.org/attachment.cgi?id=699975&action=edit

I was planning to resolve that when I land.

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 699981 [details] [diff] [review]
remove from buildbot

r+ in this case.

BTW, maybe it'll be easier to read the cofigs if we have something likes this:

'lion': dict([("talos-r4-lion-%03i" % x, {}) for x in \
      set(range(4,85)) - set([10, 63, 83]) ]),

Comment 21

[Justin Wood (:Callek)]

Not a buildduty need for tracking anymore

Ben --> can you please make sure all the patches are properly landed.