728257 - (tegra-268) tegra-268 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Armen [:armenzg]]

I couldn't manage to recover tegra-268.

Could you please recover it when you can get to it?
Thanks!

foopy23:~ cltbld$ /builds/check.sh -t tegra-268 -c
2012-02-17 08:33:17,989  Tegra ID M    Tegra       CP    Slave :: Msg
2012-02-17 08:33:33,368 tegra-268 p   OFFLINE   active  OFFLINE :: PING tegra-268.build.mtv1.mozilla.com (10.250.51.108): 56 data bytes Request timeout for icmp_seq 0;
2012-02-17 08:33:33,369 power cycling tegra
SNMPv2-SMI::enterprises.1718.3.2.3.1.11.1.1.10 = INTEGER: 

foopy23:~ cltbld$ /builds/check.sh -t tegra-268 
2012-02-17 08:48:21,806  Tegra ID M    Tegra       CP    Slave :: Msg
2012-02-17 08:48:36,837 tegra-268 p   OFFLINE   active  OFFLINE :: PING tegra-268.build.mtv1.mozilla.com (10.250.51.108): 56 data bytes Request timeout for icmp_seq 0;

Comment 1

[Armen [:armenzg]]

bear, here is a paste of what was the last step it ever run in case we need to file a bug to find out if there is something we can do:

python /builds/sut_tools/cleanup.py 10.250.51.108
 in dir /builds/tegra-268/test/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['python', '/builds/sut_tools/cleanup.py', '10.250.51.108']
 environment:
  PATH=/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/tegra-268/test
  SUT_IP=10.250.51.108
  SUT_NAME=tegra-268
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 closing stdin
 using PTY: False
Warning proxy.flg found during cleanup
Connecting to: 10.250.51.108
reconnecting socket
unable to connect socket
reconnecting socket
unable to connect socket
reconnecting socket
unable to connect socket
reconnecting socket
unable to connect socket
reconnecting socket
unable to connect socket
reconnecting socket
unable to connect socket
devroot None
/builds/tegra-268/test/../error.flg
Remote Device Error: devRoot from devicemanager [None] is not correct

Comment 2

[Matthew Larrain[:MaRu]]

The sdcard might be dead trying a second reimage

Comment 3

[Matthew Larrain[:MaRu]]

imaged and put back into the rack

Comment 4

[Armen [:armenzg]]

This seems to have gone down again.

03-02-2012 13:37:01 PING down since then.

What should we do?

Comment 5

[[e-mail] (:bhearsum)]

ping?

Comment 6

[Matthew Larrain[:MaRu]]

I will try a new SD card on it

Comment 7

[Matthew Larrain[:MaRu]]

new SD card imaged and plugged back in. Please test again. I will copy and zip the contents of the possible bad SD card. Who would like it for review?

Comment 8

[[e-mail] (:bhearsum)]

I'm still having trouble getting this Tegra up. Our scripts are saying:
2012-03-08 12:31:13,032 tegra-268 p   OFFLINE   active  OFFLINE :: ping: sendto: No route to host PING tegra-268.build.mtv1.mozilla.com (10.250.51.108): 56 data bytes;


And I've already tried rebooting it via the PDU.

Comment 9

[[e-mail] (:bhearsum)]

17:03 < bear-afk> bhearsum - tegra-268 should be put back for some lovin'  - most likely the power switch is 
                  wrong or something simple - that keeps it from starting the OS 
17:03 < bhearsum|buildduty> alrighty
17:03 < bhearsum|buildduty> thanks bear
17:03 < bear-afk> the power switch has 3 positions, two of them prevent it from working :)
17:03 < bhearsum|buildduty> of course!
17:03 < bear-afk> one is for "simulate battery" or some such

Comment 10

[Matthew Larrain[:MaRu]]

it only has two modes battery mode and power supply I always make sure it is on power supply mode which it should be. I think i used the wrong SD card (grabbed one of the ones for panda boards) when I replaced it using a new one meant for tegras.

Comment 11

[Mike Taylor [:bear]]

moved over to staging by manually editing the tac file - clientproxy restarted

Comment 12

[[e-mail] (:bhearsum)]

Is this tegra ready for production again?

Comment 13

[[e-mail] (:bhearsum)]

Looks like it hasn't done any jobs in staging yet.

Comment 14

[Mike Taylor [:bear]]

It appears to be hard offline, i've given the PDU a kick and will check on it in an hour

Comment 15

[Mike Taylor [:bear]]

268 is now offline for use in rackmount enclosure testing - closing bug