990210 - (panda-0188) panda-0188 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Ryan VanderMeulen [:RyanVM]]

Been seeing ongoing oddball crashes with this panda, and slave health is painting a pretty poor picture of its performance of late. I've disabled it until diagnostics can be run on it.

https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=panda-0188
https://tbpl.mozilla.org/php/getParsedLog.php?id=37014015&tree=Mozilla-Aurora

Comment 1

[Release Engineering SlaveAPI Service]

Attempting SSH reboot...Failed.
Attempting reboot via Mozpool...Failed.
Filed IT bug for reboot (bug 996215)

Comment 2

[Chris Cooper [:coop] (he/him)]

Failing and retrying every job.

Comment 3

[Chris Cooper [:coop] (he/him)]

Recovery bug is resolved. Re-enabling this panda in production.

Comment 4

[Phil Ringnalda (:philor)]

Since then, it's been barely managing to stay above a .500 success rate (normal is more like .800), but the PPoD in https://tbpl.mozilla.org/php/getParsedLog.php?id=47191298&tree=Mozilla-Inbound tipped me over the edge. The boy ain't right, disabled in slavealloc.

Comment 5

[Chris Cooper [:coop] (he/him)]

Re-enabled.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Random reftest off by ones and a disconnect history that paints a pretty terrible picture says this needs re-disabling.

Comment 7

[Van Le [:van]]

replaced SD card, panda passed self test.

Comment 8

[Phil Ringnalda (:philor)]

Reenabled to build up to strike FOUR.

Comment 9

[Phil Ringnalda (:philor)]

PPoD and an atrocious failure rate. Strike four, please decomm.

Comment 10

[Chris Cooper [:coop] (he/him)]

Decommissioned.