Closed Bug 832424 Opened 7 years ago Closed 7 years ago

Recuperate tegras as best as possible

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

x86
macOS

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: armenzg, Unassigned)

References

Details

Attachments

(2 files)

Currently, we do not recuperate tegras all the way with kittenherder.
I will try to determine why and fix it.
Priority: -- → P2
Blocks: 821883
Priority: P2 → P3
Callek, jmaher: this code is a rough pass. I included jhopkin's reboot-pdu code and just added one function "pdu_reboot". It grabs the PDU info from the inventory.

I assume that after the second attempt to write we should be able to. If we're not I assume that we should file a bug to replace the SD card.
What do you think?

The code attempts to do this:
exec mount
exec newfs_msdos -F 32 /dev/block/vold/179:9
exec rebt
^]
quit
as specified in here:
https://wiki.mozilla.org/ReleaseEngineering:How_To:Android_Tegras#Disable_a_tegra

This is what I run:
Armens-MacBook-Air sut_tools hg:[default!] $ python format_sd_card.py 
02/22/2013 16:14:38: INFO: The device needs to be rebooted through the PDU
02/22/2013 16:14:38: INFO: Starting new HTTPS connection (1): inventory.mozilla.org
02/22/2013 16:14:38: DEBUG: "GET /en-US/tasty/v3/system/?limit=100&hostname__startswith=tegra-335 HTTP/1.1" 200 None
Running command: '/usr/bin/snmpset -c private pdu2.dcops.build.mtv1.mozilla.com 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.1 i 3'
SNMPv2-SMI::enterprises.1718.3.2.3.1.11.1.1.1 = INTEGER: 3
02/22/2013 16:14:39: INFO: Waiting for device to come back...
02/22/2013 16:15:09: INFO: Try 1
reconnecting socket
Could not connect; sleeping for 5 seconds.
reconnecting socket
Could not connect; sleeping for 10 seconds.
^@reconnecting socket
02/22/2013 16:16:02: INFO: devroot /data/data/com.mozilla.SUTAgentAndroid/files/tests
reconnecting socket
Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/writetest 11724'; err='Push failed!'
02/22/2013 16:16:03: INFO: We have to assume that we cannot write to the SD card since we don't raise an exception.
Let's try to recover.
Rebooting through soft reboot...
reconnecting socket
reconnecting socket
reconnecting socket
reconnecting socket
02/22/2013 16:16:05: INFO: Waiting for device to come back...
02/22/2013 16:16:35: INFO: Try 1
reconnecting socket
02/22/2013 16:16:58: INFO: devroot /data/data/com.mozilla.SUTAgentAndroid/files/tests
reconnecting socket
Automation Error: error pushing file: Automation Error: Agent Error processing command 'push /mnt/sdcard/writetest 11724'; err='Push failed!'
02/22/2013 16:17:00: CRITICAL: We should have been able to write after formating remotely the SD card.File a bug for investigation
In fact, running the step on the board as writing on the instructions does not fix the board.
This is my wip to fix the case were we can't telnet:
"Automation Error: Unable to connect to device after 5 attempts"

It reboots through the PDU (without SUT agent) and then it hopes that the next time that verify is called we will be fine.
I'm looking at my queue and I don't have time for this for a while.
Assignee: armenzg → nobody
Comment on attachment 717293 [details] [diff] [review] [diff] [review]
format sd card script

---------AUTOMATIC COMMENT---------
--  filter pep8-callek-june-2013 --
---------AUTOMATIC COMMENT---------

$pep8 <dir> --diff --max-line-length=159 --show-source < attachment.diff
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:9:1: E302 expected 2 blank lines, found 1
def _waitForDevice(device, waitTime=30, silent=False):
^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:15:17: E261 at least two spaces before inline comment
    maxTries = 6 # defaults should be 5 mins
                ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:20:15: E711 comparison to None should be 'if cond is not None:'
        if dm != None:
              ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:35:1: E302 expected 2 blank lines, found 1
def main():
^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:49:11: E711 comparison to None should be 'if cond is None:'
    if dm == None:
          ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:52:19: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
        if result == False:
                  ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:57:15: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
    if result == False:
              ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:66:19: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
        if result == False:
                  ^
/tmp/tmp.cdkvLI145A/sut_tools/format_sd_card.py:71:23: E712 comparison to False should be 'if cond is False:' or 'if not cond:'
            if result == False:
                      ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:28:1: E302 expected 2 blank lines, found 1
class DeviceNotFoundException(Exception):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:30:1: E302 expected 2 blank lines, found 0
class TooManyMatchesException(Exception):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:33:1: E302 expected 2 blank lines, found 1
class PDUInfo(object):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:41:48: E261 at least two spaces before inline comment
        self.vendor_string = '1.3.6.1.4.1.1718' # serverTech enterprises
                                               ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:60:13: E128 continuation line under-indented for visual indent
            self.sentry,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:61:13: E128 continuation line under-indented for visual indent
            self.system_tables,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:62:13: E128 continuation line under-indented for visual indent
            self.outlet_tables,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:63:13: E128 continuation line under-indented for visual indent
            self.outlet_entry,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:64:13: E128 continuation line under-indented for visual indent
            self.control_action,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:65:13: E128 continuation line under-indented for visual indent
            self.sentry_enclosure_id,
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:66:13: E128 continuation line under-indented for visual indent
            str(self.input_power_feed),
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:67:13: E128 continuation line under-indented for visual indent
            str(self.outlet_id)]
            ^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:94:1: E302 expected 2 blank lines, found 1
class HostInfo(object):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:113:1: E302 expected 2 blank lines, found 1
def run_command(cmd, verbose=True, dry_run=False):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:125:1: E302 expected 2 blank lines, found 1
def load_config(filename):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:130:1: E302 expected 2 blank lines, found 1
def get_device(url, hostname, username, password, verbose=False):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:142:1: E302 expected 2 blank lines, found 1
def get_devices(url, filter, username, password, verbose=False):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:163:1: E302 expected 2 blank lines, found 1
def pdu_reboot_host(host_info, dry_run=True):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:168:1: E302 expected 2 blank lines, found 1
def pdu_reboot(host, config_file):
^
/tmp/tmp.cdkvLI145A/sut_tools/reboot_pdu.py:191:160: E501 line too long (163 > 159 characters)
    device_details = get_device(url=configd['url'], hostname=options.hostname, username=configd['username'], password=configd['password'], verbose=options.verbose)
                                                                                                                                                               ^
Product: mozilla.org → Release Engineering
Callek and Pete deployed a bunch of changes over the summer. We've also had a couple of rounds of tegra-recovery since the last bug update here.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.