873728 - (t-w732-ix-044) t-w732-ix-044 problem tracking

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=23106002&tree=Fx-Team was a Pink Pixel of Death (actually two literally pink pixels) reftest failure. In just one test image, rather than the reference image we read over and over from memory, so it might have been two cosmic rays passing through the slave, but I'm going to start being suspicious of the RAM on this slave from now on.

Comment 1

[Armen [:armenzg]]

Why did you incomplete it? Is this so you can re-open if you see another issue on this specific machine?

Comment 2

[Phil Ringnalda (:philor)]

Right, I'm not willing to say it's got bad RAM and we should disable it for a couple of weeks before running diagnostics (that always come back clean anyway) based on a single failure, but the existence of the bug will remind me of my suspicions when I'm puzzling over, say, an unfamiliar GC crash on the same slave.

Comment 3

[Phil Ringnalda (:philor)]

And another PPoD in https://tbpl.mozilla.org/php/getParsedLog.php?id=23275374&tree=Mozilla-Inbound, RAM diagnostics time I'm afraid.

Comment 4

[Phil Ringnalda (:philor)]

PPoD, https://tbpl.mozilla.org/php/getParsedLog.php?id=23506454&tree=UX

Comment 5

[Justin Wood (:Callek)]

So this was sent for diags, but never taken out of production... its also taking green jobs, so I marked it as wontfix for diags, and we'll revisit if it hurts again

Comment 6

[Ryan VanderMeulen [:RyanVM]]

PPoD. Disabled.
https://tbpl.mozilla.org/php/getParsedLog.php?id=32810855&tree=Mozilla-Inbound

Comment 7

[Phil Ringnalda (:philor)]

And that's 97 days sitting idle because the bug wasn't reopened to prompt diagnostics.

Comment 8

[Phil Ringnalda (:philor)]

Reenabled and rebooted, only a couple of weeks of sitting idle with a new graphics card.

Comment 9

[Alin Selagea [:aselagea]]

Lots of jobs ending up with "Connection to the other side was lost in a non-clean fashion". 
Disabled.

Comment 10

[Alin Selagea [:aselagea]]

Did a re-image and returned it to prod.

Comment 11

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 1370555)

Comment 12

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online

Comment 13

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 1373693)

Comment 14

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online.

Comment 15

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Filed IT bug for reboot (bug 1375127)

Comment 16

[Andrei Obreja [:aobreja NOT AVAILABLE][:buildduty]]

Back online.

Comment 17

[Radu Iman[:riman]]

Windows testing machine t-w732-ix-044 failing to work properly even after re-image.

Comment 18

[Roland Mutter Michael (:rmutter)]

Looks like there are a lot of "retry" status jobs.

Comment 19

[Adrian Pop]

Checked on slave alloc and slave health > the machine seems to be disabled.