734321 - (tegra-110) tegra-110 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

https://build.mozilla.org/buildapi/recent/tegra-110?numbuilds=100 - it was in fairly crappy shape, failing every two to five builds, until the morning of March 6th, when it went to pure red with slight accents of purple when it managed to get itself killed before it got itself killed for timing out.

Comment 1

[Mike Taylor [:bear]]

I ran ./stop_cp on it but nothing else

Comment 2

[Aki Sasaki (not active)]

Started after reimage.

Comment 3

[Armen [:armenzg]]

This board has not taken any jobs for 110 days.
Adjusting the slave alloc note and re-opening this bug.

tegra-110	108 days, 11:49:52	broken	bm19-tests1-tegra	2012-03-08 22:02:55	0:07:16	4	bug 734321

Comment 4

[Mike Taylor [:bear]]

it's a staging tegra and dashboard activity shows it has been online and available - if it's idle it's because of staging tegra master

Comment 5

[Armen [:armenzg]]

I will then remove it from production_config.py.

Comment 6

[Justin Wood (:Callek)]

(In reply to Armen Zambrano G. [:armenzg] from comment #5)
> I will then remove it from production_config.py.

Never happened.

...its also actually sitting on a production foopy, (even though its pointing at staging buildbot) I'm touching up buildbot now and will start it targetted for production. If we need to decom/move it we should pull it off the production foopy.

I did a PDU cycle, and it didn't seem to come back properly, lets reimage+sdcard swap and see where we end up

Comment 7

[Ashlee Nguyen [:ashlee]]

Tegra-110 re imaged and pingable + sd card swapped

--- tegra-110.build.mtv1.mozilla.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss

Comment 8

[Ryan VanderMeulen [:RyanVM]]

The last 3 times this tegra has attempted to run mochitest-7 on beta, it has hit bug 872076. Haven't seen that failure on any other machines.

Comment 9

[Justin Wood (:Callek)]

Sending this slave to recovery
-->Automated message.

Comment 10

[Justin Wood (:Callek)]

Successfully recovered.

Comment 11

[Phil Ringnalda (:philor)]

Hasn't taken a job for 29 days.

Comment 12

[Chris Cooper [:coop] (he/him)]

PDU reboot works, but can't even telnet into this tegra to see what's still wrong.

Comment 13

[Vinh Hua [:vinh]]

SD card replaced, reflashed, and reimaged.

telnet tegra-110.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.86...
Connected to tegra-110.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 14

[Phil Ringnalda (:philor)]

It apparently was enabled through that, and it's been rebooted several times since, but it hasn't taken a job.

Comment 15

[Phil Ringnalda (:philor)]

Disabled in slavealloc to stop the pointless stream of reboots.

Comment 16

[Van Le [:van]]

SD card formatted, tegra flashed and reimaged.

vle@vle-10516 ~ $ telnet tegra-110.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.86...
Connected to tegra-110.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 17

[Phil Ringnalda (:philor)]

Reenabled.

Comment 18

[Phil Ringnalda (:philor)]

Not actually taking jobs, though.

Comment 19

[Van Le [:van]]

replaced SD card, flashed and reimaged tegra.

vle@vle-10516 ~ $ telnet tegra-110.tegra.releng.scl3.mozilla.com 20701
Trying 10.26.85.86...
Connected to tegra-110.tegra.releng.scl3.mozilla.com.
Escape character is '^]'.
$>^]
telnet> q

Comment 20

[Phil Ringnalda (:philor)]

Let's pretend it'll work this time.

Comment 21

[Justin Wood (:Callek)]

foopy117 is going away, and these tegras are attached to it.

They are being decomissioned in Bug 1043938, and I just disabled all of them in slavealloc and marked them as decom with a comment pointing at Bug 1043938