1121501 - Devices on the CI are lost from adb and fastboot during test runs

Status: RESOLVED DUPLICATE of bug 1121374

(Firefox OS Graveyard :: Infrastructure, defect)

Description

[Florin Strugariu [:Bebe]]

Please check that b2g-20.1  is powered on, connected via USB, and can be seen by ADB

The node can't find the device


I tried to ssh in to the node and reboot the device but it's not found in adb or fastboot mode

http://jenkins1.qa.scl3.mozilla.com/computer/b2g-20.1/

Comment 1

[Florin Strugariu [:Bebe]]

Last two build that ran on the device had issues:

http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.sanity/6177/console
  IOError: Connection to Marionette server is lost. Check gecko.log (desktop firefox) or logcat (b2g) for errors.


http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.smoke/2070/console 
  mozdevice.devicemanager.DMError: bad status for device e472d8d7: offline


In the last build looks like we lost connection with the device during the test run. After this point the testrun could not find the device anymore

Comment 2

[Florin Strugariu [:Bebe]]

Geo 
Can you take a look over the device and see in what state is it?
Also try to get a logcat from the device as this might be a bigger issue

Comment 3

[Florin Strugariu [:Bebe]]

same issue we have with b2g-23.1 

See http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.smoke/2042/console

I will disable the device

Comment 4

[Florin Strugariu [:Bebe]]

Looks like it's the same behaviour as the previous device.

While running email tests the device stops responding and disappears from adb and fastboot. 
After this the device can't be found or restarted. 
Manual debugging is required.


Staktrace of the issue:

Traceback (most recent call last):
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/bin/gaiatest", line 9, in <module>
    load_entry_point('gaiatest==0.32', 'console_scripts', 'gaiatest')()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/runtests.py", line 106, in main
    cli(runner_class=GaiaTestRunner, parser_class=GaiaTestOptions)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runtests.py", line 35, in cli
    runner = startTestRunner(runner_class, options, tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runtests.py", line 20, in startTestRunner
    runner.run_tests(tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 765, in run_tests
    self.run_test_sets()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 951, in run_test_sets
    self.run_test_set(self.tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 930, in run_test_set
    self.run_test(test['filepath'], test['expected'], test['test_container'])
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 906, in run_test
    results = runner.run(suite)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 235, in run
    result = super(MarionetteTextTestRunner, self).run(test)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/moztest-0.7-py2.7.egg/moztest/adapters/unit.py", line 165, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/marionette_test.py", line 258, in run
    result.addError(self, sys.exc_info())
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 134, in addError
    self.add_test_result(test, output=self._exc_info_to_string(err, test), result_actual='ERROR')
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 126, in add_test_result
    result_expected, result_actual, output, context = modifier(t, result_expected, result_actual, output, context)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 113, in b2g_output_modifier
    extra_output = self._diagnose_socket()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 74, in _diagnose_socket
    device_manager = get_dm(self.marionette)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 19, in get_dm
    **kwargs)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 72, in __init__
    self.connect()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 81, in connect
    self._verifyDevice()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 653, in _verifyDevice
    raise DMError("bad status for device %s: %s" % (self._deviceSerial, deviceStatus))
mozdevice.devicemanager.DMError: bad status for device e472d8d7: offline

Comment 5

[Florin Strugariu [:Bebe]]

b2g-09.1 is in the same state:

http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.mozilla-central.ui.functional.smoke/259/console

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like this often happens in the e-mail or SMS tests? Perhaps this is a genuine crasher. Can we replicate it locally, or try with an older build? Is it possible that we need the v18D-1 base build?

Comment 9

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Today's carnage:

7
9
13
17.1
17.2
18.2
19.1
21.1
21.2
26.1
28.2

I brought up all of them, and put any offline back online except 7, 9, and 21.1. I'm keeping those temporarily offline to make sure we can run functional tests on Monday while nobody's in MTV (bring them back online for those runs only please).

We do have progress on a theory. Turns out the USB ports only deliver 500mA, which may not be enough to fully recharge the phone under load. If there's a higher-than-normal power drain, that would cause the behavior we see. Right now, that's the most plausible explanation.