809012 - panda boards do not reboot when the sutagent is in a hung state

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

the last step we run is reboot.py, but I found that it is hanging on trying to print the logcat information, then we timeout.  We must always reboot at the end, otherwise the device stays up for hours and we get stuck.

here is an example:
http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%20Panda%20mozilla-central%20opt%20test%20mochitest-4/builds/1376/steps/reboot%20device/logs/stdio

inside of reboot.py, we have this:
    try:
        dm.getInfo('process')
        log.info(dm._runCmds([{'cmd': 'exec su -c "logcat -d -v time *:W"'}]))

        log.info('forcing device reboot')
        status = soft_reboot(dm=dm, device=deviceName, ipAddr=proxyIP, port=proxyPort)
        log.info(status)
    finally:
        try:
            waitForDevice(dm, waitTime=300)
        except SystemExit:
            setFlag(errorFile, "Remote Device Error: call for device reboot failed")
            return 1


we really should change it to be more like this:
    try:
        dm.getInfo('process')
        log.info(dm._runCmds([{'cmd': 'exec su -c "logcat -d -v time *:W"'}]))
    except:
        log.info('Error getting logcat information')

    try:
        log.info('forcing device reboot')
        status = soft_reboot(dm=dm, device=deviceName, ipAddr=proxyIP, port=proxyPort)
        log.info(status)
    except:
        log.info('Error running soft_reboot')

    # missing code to verify the device was rebooted, maybe try to connect and verify it fails

    try:
        waitForDevice(dm, waitTime=300)
    except SystemExit:
        setFlag(errorFile, "Remote Device Error: unable to connect to device after reboot")
        return 1

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Add logic to reboot when logcat fails (1.0)

tested this on a hung panda in the pool and it works.

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 678717 [details] [diff] [review]
Add logic to reboot when logcat fails (1.0)

Review of attachment 678717 [details] [diff] [review]:
-----------------------------------------------------------------

r+ ...And as joked about on IRC yes I do think this is a symptom of an underlying problem. But doesn't mean we should ignore the symptom when we don't yet know the cause.

::: sut_tools/reboot.py
@@ +19,5 @@
>      try:
>          dm.getInfo('process')
> +        log.info(dm._runCmds([{'cmd': 'exec su -c "logcat -d -v time *:W"'}], timeout=10))
> +    except:
> +        log.info("unable to run logcat on device")

nit: Failure to trying to run logcat on device

@@ +26,5 @@
>          log.info('forcing device reboot')
>          status = soft_reboot(dm=dm, device=deviceName, ipAddr=proxyIP, port=proxyPort)
>          log.info(status)
> +    except:
> +        log.info("Unable to reboot device successfully")

Nit: Failure while rebooting device

@@ +32,5 @@
> +
> +    try:
> +        waitForDevice(dm, waitTime=300)
> +    except SystemExit:
> +        setFlag(errorFile, "Remote Device Error: call for device reboot failed")

Nit: given logic change, perhaps "Remote Device Error: Device failed to recover after reboot"

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/build/tools/rev/0d98286d34f6