808468 - tegra recovery

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Justin Wood (:Callek)]

+++ This bug was initially created as a clone of Bug #807965 +++

We have another batch of tegras to recover.

Unfortunately we should figure out a theory/solution to Bug 808437 before we do this again though.

Comment 1

[Van Le [:van]]

:Callek, do you think we should move on with reimaging/recovering these tegras? Were you able to check the image?

Comment 2

[Justin Wood (:Callek)]

(In reply to Van Le [:van] from comment #1)
> :Callek, do you think we should move on with reimaging/recovering these
> tegras? Were you able to check the image?

Ok, from my understanding the image is fine. I would for this batch request we do a 1-pass "full" sdcard check for bad sectors. (I am told this is doable, just time consuming)

The check should be done on (a) a full batch of sdcards to swap, for each tegra here [all get a new card]
(b) all sdcards we remove from these tegras [so we can try and identify if there was an sdcard issue]

if possible, but not a hard req, I would love a "this sdcard was bad" mapping to what tegra it was in, after the fact.

We won't immediately bring up all these devices into production, but aprox 1/3'rd to try and see if they come back fine.

And in the near-future I may have a new image/script for you guys to use, which updates apk's we install as well as [slightly] the method we use to install them. It won't however update the base image.

Comment 3

[Amy Rich [:arr] [:arich]]

callek: so this would take a dedicated day of one of the dcops folks time, and they doesn't have the man power for this right now because they are split between locations.  Is there a step that can be cut out, or would you like us to defer getting all of these done until more people are around?

Comment 4

[Van Le [:van]]

We tested and confirmed the media of the brand new SD cards with "THE DUPLICATOR!!!" before we began the flash/reimage process for the tegras in this bug. 

The tegras should all be online now after a new SD card swap, flash, and reimage.

Prior to removing the SD card from the tegra, we wrote down the name of it in permanent marker so we can test it as requested.

2 of the tegras came back with bad media when we tested their original cards: tegra[194, 218].

Comment 5

[Justin Wood (:Callek)]

I'll note that none of these turned back on are succeeding to take jobs at the moment. We are failing to update the watcher.ini, with an issue that is fixed in current (non-deployed) code, I'm going to get a bug on file to track enabling/verifying these before we put new ones through the DC Reimage process.

Comment 6

[Justin Wood (:Callek)]

...this bug shouldn't throw e-mails up if the tegras are reopened/resolved. Fixing deps.