618384 - figure out why tegras are falling over so often

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Aki Sasaki (not active)]

This may involve per-device logging.

Is it when they run a particular dm command, or when they run a certain test?
Do a bunch of them fall over at the same time?

Comment 1

[Aki Sasaki (not active)]

I asked Bob about possibly adding a Watcher check that does an "if uptime > CONFIGURABLE_LENGTH_OF_TIME and SUT_IS_NOT_RUNNING then rebt".

This may not catch everything, but it'll catch a lot of what we're seeing.

Preferably that configurable length of time would be set in watcher.ini.

I'm still curious to see if we can figure out why.

Comment 2

[Aki Sasaki (not active)]

Bob and I looked at a down tegra and Watcher wasn't running (or SUT either), so the above watcher check wouldn't help at all.

Per-device logging with symptoms and/or suspected causes it is.

Comment 3

[Mike Taylor [:bear]]

Aki - I want to close this as changes to sutAgent, clientproxy and the sut_tools helper script have reduced the number of failed tegras to a small occurance.

Comment 4

[Mike Taylor [:bear]]

closing - we had a full night of green runs with the odd exception and sut_tools caught them all