1175733 - Retry tests jobs that fail with DMError

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Since bug 681948, we have been retrying test jobs containing "devicemanager.DMError":

http://hg.mozilla.org/build/buildbotcustom/annotate/825dd26b2063/status/errors.py#l12

But current Android 4.3 jobs that throw a devicemanager.DMError report it like this:

11:40:40     INFO -  Automation Error: Exception caught while running tests
11:40:40     INFO -  Traceback (most recent call last):
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/remotereftest.py", line 551, in main
11:40:40     INFO -      retVal = reftest.runTests(manifest, options, cmdlineArgs)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/runreftest.py", line 334, in runTests
11:40:40     INFO -      return self.runSerialTests(testPath, options, cmdlineArgs)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/runreftest.py", line 632, in runSerialTests
11:40:40     INFO -      debuggerInfo=debuggerInfo)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/remotereftest.py", line 449, in runApp
11:40:40     INFO -      timeout=timeout)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/automation.py", line 844, in runApp
11:40:40     INFO -      crashed = self.checkForCrashes(os.path.join(profileDir, "minidumps"), symbolsPath)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/remoteautomation.py", line 200, in checkForCrashes
11:40:40     INFO -      logcat = self._devicemanager.getLogcat(filterOutRegexps=fennecLogcatFilters)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/devicemanager.py", line 160, in getLogcat
11:40:40     INFO -      root=self._logcatNeedsRoot)
11:40:40     INFO -    File "/builds/slave/test/build/tests/reftest/devicemanager.py", line 399, in shellCheckOutput
11:40:40     INFO -      raise DMError("Non-zero return code for command: %s (output: '%s', retval: '%s')" % (cmd, output, retval))
11:40:40     INFO -  DMError: Non-zero return code for command: ['/system/bin/logcat', '-v', 'time', '-d', 'dalvikvm:I', 'ConnectivityService:S', 'WifiMonitor:S', 'WifiStateTracker:S', 'wpa_supplicant:S', 'NetworkStateTracker:S'] (output: '', retval: 'None')

"devicemanager.DMError" is not in the output -- just "DMError" -- so the job does not retry.

Bug 1157948 is a frequent DMError bug; it produces the output shown above. I cannot think of any other current failures involving DMError.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: retry on DMError

DMError generally indicates that the test harness cannot communicate with the device -- an "infrastructure" issue that is likely not associated with a particular test: Retrying seems appropriate.

The only immediate practical change I am expecting is that bug 1157948 failures will become retries. I'll keep trying to address root cause of 1157948, but in the meantime, this should provide some relief from starring.

Comment 2

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/buildbotcustom/rev/7038aa783115

Comment 3

[Geoff Brown [:gbrown]]

Verified: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11-debug/1434648883/mozilla-inbound_ubuntu64_vm_armv7_large-debug_test-mochitest-4-bm114-tests1-linux64-build66.txt.gz caused a retry.