925462 - (talos-r4-lion-027) talos-r4-lion-027 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

We saw one instance of _Black_ Pixel of Death
https://tbpl.mozilla.org/php/getParsedLog.php?id=28937841&tree=Mozilla-Inbound#error0

Comment 1

[Armen [:armenzg]]

Nothing to do though.

RyanVM: should the text get a little bit more of fuzzing?

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Why would I add fuzz to a test for a one-off failure? Looking at the screenshot and the location of the wrong pixel, I find it hard to believe that the failure had anything to do with the specific test being run.

Comment 3

[Armen [:armenzg]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #2)
> Why would I add fuzz to a test for a one-off failure? Looking at the
> screenshot and the location of the wrong pixel, I find it hard to believe
> that the failure had anything to do with the specific test being run.

I was hoping that it would make it hide this type of one-off failures and see one less intermittent orange.

Comment 4

[Phil Ringnalda (:philor)]

This is one weird slave. PPoD is perfectly understandable, that shade of pink is one bit off from white, so one bit of memory flipped to the wrong state gives you pink instead of white, but... https://tbpl.mozilla.org/php/getParsedLog.php?id=29473226&tree=Fx-Team is another black pixel of death. How is it doing that?

And no, fuzzing every single reftest in the tree to paper over bad slaves isn't something we're going to do. Somewhere we have another bug where you suggested that, and I ranted for 500 words about how reftests cleanly show bad memory, but other suites show it by crashing because a bit in a pointer was flipped so it accesses memory where it shouldn't be, or by failing because the actual result of something was one number, but it passed through the bad memory and came back as a different number.

Comment 5

[Armen [:armenzg]]

It seems that our only option is to replace the memory.
We could send the memory to dolske to see if he finds anything interesting.

Comment 6

[Armen [:armenzg]]

Is comment 5 accurate?

Comment 7

[Chris Cooper [:coop] (he/him)]

Back in production.