771560 - (tegra-073) tegra-073 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Justin Wood (:Callek)]

Tegra-073 is having a problem

* error.flg [Unable to properly uninstall org.mozilla.fennec] 

We need to reimage it

Comment 1

[Justin Wood (:Callek)]

Last Job 4 days, 3:03:56 ago

Comment 2

[Justin Wood (:Callek)]

This is still having issues. Current symptom is lack of an sdcard drive (relegating to a failed SUTAgent upgrade).

Can you please replace the sdcard entirely, and re-image the whole tegra.

Comment 3

[[e-mail] (:bhearsum)]

Back in production.

Comment 4

[Phil Ringnalda (:philor)]

It took a little nap between 2012-09-11 and 2012-09-20 (and good riddance), but since it has been back, it has done 45 jobs, succeeding at fewer than half of them.

Near as I can tell, it was reimaged at least three times and got a new sd card twice in August. I say scrap it.

Comment 5

[Phil Ringnalda (:philor)]

No words can adequately describe my hatred of the utter randomness of tegras.

Comment 6

[Chris Cooper [:coop] (he/him)]

Back in production.

Comment 7

[Justin Wood (:Callek)]

Thursday, November 07, 2013 7:52:11 AM

Comment 8

[Eric Ramirez [:Eric]]

SD card has been replaced and reimaged/flashed.

Comment 9

[Eric Ramirez [:Eric]]

sdcard replaced and reimaged/flashed.

Comment 10

[Justin Wood (:Callek)]

handled in last recovery on Dec 16.

Comment 11

[[e-mail] (:bhearsum)]

2014-01-16 10:20:19 tegra-073 p    online   active  OFFLINE :: error.flg [Automation Error: Unable to connect to device after 5 attempts] 

pdu reboot didn't help

Comment 12

[Eric Ramirez [:Eric]]

SD card reformatted and flashed.

Comment 13

[[e-mail] (:bhearsum)]

(In reply to Eric Ramirez [:Eric] from comment #12)
> SD card reformatted and flashed.

can we try again?

Comment 14

[Van Le [:van]]

SD card formatted, tegra reimaged and flashed.

[[e-mail]3 ~]$ fping tegra-073.tegra.releng.scl3.mozilla.com
tegra-073.tegra.releng.scl3.mozilla.com is alive

Comment 15

[Pete Moore [:pmoore][:pete] [PTO until 13 April 2026]]

We have a green test result from 27 Feb, since fix - but the latest test is a FAIL - so waiting for more results before closing.

Comment 16

[John Hopkins (:jhopkins)]

Power cycled, waited a day.

Last status: 2014-04-02 11:40:19 tegra-073 p    online   active  OFFLINE :: 

SD card reformat was successful:

$>exec newfs_msdos -F 32 /dev/block/vold/179:9
newfs_msdos: warning, /dev/block/vold/179:9 is not a character device
newfs_msdos: Skipping mount checks
/dev/block/vold/179:9: 31100416 sectors in 485944 FAT32 clusters (32768 bytes/cluster)
bps=512 spc=64 res=32 nft=2 mid=0xf0 spt=16 hds=4 hid=0 bsec=31108096 bspf=3797 rdcl=2 infs=1 bkbs=2
return code [0]
$>exec rebt
$>^]

Comment 17

[Jordan Lund (:jlund)]

lockfile issues. hasn't run a job since reformat

watcher.log snippet:
850 04/07/2014 10:25:06: DEBUG: calling [ping -c 5 tegra-073]
851 04/07/2014 10:25:10: INFO: Connecting to: tegra-073
852 2014-04-07 10:30:01 -- *** ERROR *** failed to aquire lockfile
853 04/07/2014 10:30:11: INFO: INFO: Unable to connect to device after 1 try
854 04/07/2014 10:30:11: INFO: We're going to sleep for 90 seconds
855 04/07/2014 10:31:41: INFO: Connecting to: tegra-073
856 2014-04-07 10:35:02 -- *** ERROR *** failed to aquire lockfile
857 04/07/2014 10:36:43: INFO: INFO: Unable to connect to device after 2 try
858 04/07/2014 10:36:43: INFO: We're going to sleep for 90 seconds
859 04/07/2014 10:38:13: INFO: Connecting to: tegra-073
860 2014-04-07 10:40:01 -- *** ERROR *** failed to aquire lockfile

Comment 18

[Chris Cooper [:coop] (he/him)]

Automation Error: Unable to connect to device after 5 attempts

Multiple PDU reboots haven't brought this device back. Same as reported in comment #17.

Comment 19

[Van Le [:van]]

formatted sd card, flashed and reimaged tegra

[[e-mail]3 ~]$  telnet tegra-073.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.53...
Connected to tegra-073.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]q

telnet> q

Comment 20

[Phil Ringnalda (:philor)]

Still hasn't taken a job, apparently there's something more wrong with it.

Comment 21

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 22

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-073.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.53...
Connected to tegra-073.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 23

[Phil Ringnalda (:philor)]

Reenabled, but I'm not going to bother closing the bug unless it happens to take its first job since March 3rd, which there's no reason to believe it will.

Comment 24

[Phil Ringnalda (:philor)]

And after 24 hours and multiple reboots, it hasn't taken a job.

Comment 25

[Justin Wood (:Callek)]

but it has now!