770513 - (tegra-218) tegra-218 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Comment 1

[Justin Wood (:Callek)]

Got this back up just now after a few PDU reboots and a cp stop/start cycle

Comment 2

[Phil Ringnalda (:philor)]

Last 49 jobs have been red or purple.

Comment 3

[[e-mail] (:bhearsum)]

Back in production.

Comment 4

[Phil Ringnalda (:philor)]

And post-recovery, it managed 11 green builds, 1 orange, and has now been red for the last 11 in a row.

Comment 5

[[e-mail] (:bhearsum)]

ran stop_cp.sh on it.

Comment 6

[[e-mail] (:bhearsum)]

Not sure what to do next. Callek?

Comment 7

[Phil Ringnalda (:philor)]

Run stop_cp on it again? This time --with-fire?

Comment 8

[Justin Wood (:Callek)]

(In reply to Phil Ringnalda (:philor) from comment #7)
> Run stop_cp on it again? This time --with-fire?

done

Comment 9

[Justin Wood (:Callek)]

IT handled this, start_cp run now

Comment 10

[Phil Ringnalda (:philor)]

7 red in a row.

Comment 11

[Phil Ringnalda (:philor)]

haHa, https://secure.pub.build.mozilla.org/buildapi/recent/tegra-218

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Ran stop_cp on it.

Comment 13

[Justin Wood (:Callek)]

[TODO Hand to recovery]

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Justin Wood (:Callek) from comment #13)
> [TODO Hand to recovery]

Is there a reason you didn't block the tegra-recovery bug when you made this comment? This Tegra has now sat idle for weeks for no good reason.

Comment 15

[[e-mail] (:bhearsum)]

Back in production.

Comment 16

[Phil Ringnalda (:philor)]

36% green.

Comment 17

[Justin Wood (:Callek)]

Ran ./stop_cp.sh

Comment 18

[Justin Wood (:Callek)]

This is back to taking job(s) and is up now.

Comment 19

[Phil Ringnalda (:philor)]

Pretty much worthless - today it took two jobs, RETRY on one; yesterday it took two jobs, RETRY on one; the day before zero jobs; the day before that two jobs, RETRY on one. The 22nd was a big day for it, seven jobs, RETRY on six.

Comment 20

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 21

[Justin Wood (:Callek)]

Successfully recovered.

Comment 22

[[e-mail] (:bhearsum)]

agent check failing; pdu reboot didn't help

Comment 23

[[e-mail] (:bhearsum)]

2014-01-16 10:22:49 tegra-218 p    online   active  OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 

pdu reboot didn't help

Comment 24

[John Pech [:jpech-lbc]]

SD card have been wiped and tegra have been flashed/re-imaged.

Comment 25

[Justin Wood (:Callek)]

taking production jobs