Devices on the CI are lost from adb and fastboot during test runs

RESOLVED DUPLICATE of bug 1121374

Status

--
critical
RESOLVED DUPLICATE of bug 1121374
4 years ago
4 years ago

People

(Reporter: Bebe, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Please check that b2g-20.1  is powered on, connected via USB, and can be seen by ADB

The node can't find the device


I tried to ssh in to the node and reboot the device but it's not found in adb or fastboot mode

http://jenkins1.qa.scl3.mozilla.com/computer/b2g-20.1/
(Reporter)

Comment 1

4 years ago
Last two build that ran on the device had issues:

http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.sanity/6177/console
  IOError: Connection to Marionette server is lost. Check gecko.log (desktop firefox) or logcat (b2g) for errors.


http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.smoke/2070/console 
  mozdevice.devicemanager.DMError: bad status for device e472d8d7: offline


In the last build looks like we lost connection with the device during the test run. After this point the testrun could not find the device anymore
(Reporter)

Comment 2

4 years ago
Geo 
Can you take a look over the device and see in what state is it?
Also try to get a logcat from the device as this might be a bigger issue
Flags: needinfo?(gmealer)
(Reporter)

Comment 3

4 years ago
same issue we have with b2g-23.1 

See http://jenkins1.qa.scl3.mozilla.com/job/flame-kk-319.b2g-inbound.ui.functional.smoke/2042/console

I will disable the device
(Reporter)

Updated

4 years ago
Summary: b2g-20.1 cannot be found via ADB → Devices on the CI are lost from adb and fastboot during test runs
(Reporter)

Comment 4

4 years ago
Looks like it's the same behaviour as the previous device.

While running email tests the device stops responding and disappears from adb and fastboot. 
After this the device can't be found or restarted. 
Manual debugging is required.


Staktrace of the issue:

Traceback (most recent call last):
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/bin/gaiatest", line 9, in <module>
    load_entry_point('gaiatest==0.32', 'console_scripts', 'gaiatest')()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/runtests.py", line 106, in main
    cli(runner_class=GaiaTestRunner, parser_class=GaiaTestOptions)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runtests.py", line 35, in cli
    runner = startTestRunner(runner_class, options, tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runtests.py", line 20, in startTestRunner
    runner.run_tests(tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 765, in run_tests
    self.run_test_sets()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 951, in run_test_sets
    self.run_test_set(self.tests)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 930, in run_test_set
    self.run_test(test['filepath'], test['expected'], test['test_container'])
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 906, in run_test
    results = runner.run(suite)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 235, in run
    result = super(MarionetteTextTestRunner, self).run(test)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/moztest-0.7-py2.7.egg/moztest/adapters/unit.py", line 165, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 395, in __call__
    return self.run(*args, **kwds)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/marionette_test.py", line 258, in run
    result.addError(self, sys.exc_info())
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 134, in addError
    self.add_test_result(test, output=self._exc_info_to_string(err, test), result_actual='ERROR')
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/base.py", line 126, in add_test_result
    result_expected, result_actual, output, context = modifier(t, result_expected, result_actual, output, context)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 113, in b2g_output_modifier
    extra_output = self._diagnose_socket()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 74, in _diagnose_socket
    device_manager = get_dm(self.marionette)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.6-py2.7.egg/marionette/runner/mixins/b2g.py", line 19, in get_dm
    **kwargs)
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 72, in __init__
    self.connect()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 81, in connect
    self._verifyDevice()
  File "/var/jenkins/1/workspace/flame-kk-319.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/mozdevice-0.44-py2.7.egg/mozdevice/devicemanagerADB.py", line 653, in _verifyDevice
    raise DMError("bad status for device %s: %s" % (self._deviceSerial, deviceStatus))
mozdevice.devicemanager.DMError: bad status for device e472d8d7: offline
(Reporter)

Updated

4 years ago
Severity: normal → critical
No longer depends on: 1121439
(Reporter)

Updated

4 years ago
Depends on: 1121439
Duplicate of this bug: 1121439
(Reporter)

Updated

4 years ago
No longer depends on: 1121439
It looks like this often happens in the e-mail or SMS tests? Perhaps this is a genuine crasher. Can we replicate it locally, or try with an older build? Is it possible that we need the v18D-1 base build?
(Reporter)

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1121374
(Reporter)

Updated

4 years ago
Flags: needinfo?(gmealer)
Today's carnage:

7
9
13
17.1
17.2
18.2
19.1
21.1
21.2
26.1
28.2

I brought up all of them, and put any offline back online except 7, 9, and 21.1. I'm keeping those temporarily offline to make sure we can run functional tests on Monday while nobody's in MTV (bring them back online for those runs only please).

We do have progress on a theory. Turns out the USB ports only deliver 500mA, which may not be enough to fully recharge the phone under load. If there's a higher-than-normal power drain, that would cause the behavior we see. Right now, that's the most plausible explanation.
You need to log in before you can comment on or make changes to this bug.