1112697 - Android Talos tspaint frequently fails: mozdevice.devicemanager.DMError: Remote Device Error: unable to connect to x.x.x.x after 5 attempts

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: Testing, defect)

Description

[Geoff Brown [:gbrown]]

All Android Talos jobs are currently hidden (bug 1088019) but we hope to un-hide them soon -- except for ts-paint, which has a very high retry rate. We should determine what's wrong with ts-paint.

This looks typical:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-11/1418817215/mozilla-central_panda_android_test-remote-tspaint-bm102-tests1-panda-build1143.txt.gz

05:14:53     INFO - #### Running talos suites
05:14:53     INFO - Running command: ['unzip', '-q', '-o', '/builds/panda-0143/test/build/fennec-37.0a1.en-US.android-arm.apk'] in /builds/panda-0143/talos-data/fennec
05:14:53     INFO - Copy/paste: unzip -q -o /builds/panda-0143/test/build/fennec-37.0a1.en-US.android-arm.apk
05:14:54     INFO - Return code: 0
05:14:54     INFO - Reading from file /builds/panda-0143/talos-data/fennec/package-name.txt
05:14:54     INFO - Contents:
05:14:54     INFO -  org.mozilla.fennec
05:14:54     INFO - mkdir: /builds/panda-0143/test/build/blobber_upload_dir
05:14:54     INFO - ENV: SHELL is now /bin/sh
05:14:54     INFO - ENV: SHLVL is now 4
05:14:54     INFO - ENV: PYTHONPATH is now /builds/sut_tools
05:14:54     INFO - ENV: OLDPWD is now /home/cltbld
05:14:54     INFO - ENV: SUT_NAME is now panda-0143
05:14:54     INFO - ENV: TEST_DEVICE is now panda-0143
05:14:54     INFO - ENV: PWD is now /builds/panda-0143/test
05:14:54     INFO - ENV: LOGNAME is now cltbld
05:14:54     INFO - ENV: USER is now cltbld
05:14:54     INFO - ENV: PROPERTIES_FILE is now /builds/panda-0143/test/buildprops.json
05:14:54     INFO - ENV: MOZ_UPLOAD_DIR is now /builds/panda-0143/test/build/blobber_upload_dir
05:14:54     INFO - ENV: PATH is now /builds/panda-0143/test/build/venv/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin
05:14:54     INFO - ENV: SUT_IP is now 10.26.128.126
05:14:54     INFO - ENV: HOME is now /home/cltbld
05:14:54     INFO - ENV: DM_TRANS is now sut
05:14:54     INFO - ENV: _ is now /tools/buildbot/bin/python2.7
05:14:54     INFO - ENV: MINIDUMP_SAVE_PATH is now /builds/panda-0143/test/build/blobber_upload_dir
05:14:54     INFO - Running command: ['python', '-u', '/builds/panda-0143/talos-data/talos/run_tests.py', u'--apkPath=/builds/panda-0143/test/build/fennec-37.0a1.en-US.android-arm.apk', '--noisy', 'local.yml', '--datazilla-url', 'https://datazilla.mozilla.org/talos', '--authfile', '/builds/panda-0143/test/oauth.txt'] in /builds/panda-0143/talos-data/talos
05:14:54     INFO - Copy/paste: python -u /builds/panda-0143/talos-data/talos/run_tests.py --apkPath=/builds/panda-0143/test/build/fennec-37.0a1.en-US.android-arm.apk --noisy local.yml --datazilla-url https://datazilla.mozilla.org/talos --authfile /builds/panda-0143/test/oauth.txt
05:14:54     INFO - Using env: {'DM_TRANS': 'sut',
05:14:54     INFO -  'HOME': '/home/cltbld',
05:14:54     INFO -  'LOGNAME': 'cltbld',
05:14:54     INFO -  'MINIDUMP_SAVE_PATH': '/builds/panda-0143/test/build/blobber_upload_dir',
05:14:54     INFO -  'MOZ_UPLOAD_DIR': '/builds/panda-0143/test/build/blobber_upload_dir',
05:14:54     INFO -  'OLDPWD': '/home/cltbld',
05:14:54     INFO -  'PATH': '/builds/panda-0143/test/build/venv/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cltbld/bin',
05:14:54     INFO -  'PROPERTIES_FILE': '/builds/panda-0143/test/buildprops.json',
05:14:54     INFO -  'PWD': '/builds/panda-0143/test',
05:14:54     INFO -  'PYTHONPATH': '/builds/sut_tools',
05:14:54     INFO -  'SHELL': '/bin/sh',
05:14:54     INFO -  'SHLVL': '4',
05:14:54     INFO -  'SUT_IP': '10.26.128.126',
05:14:54     INFO -  'SUT_NAME': 'panda-0143',
05:14:54     INFO -  'TEST_DEVICE': u'panda-0143',
05:14:54     INFO -  'USER': 'cltbld',
05:14:54     INFO -  '_': '/tools/buildbot/bin/python2.7'}
05:14:54     INFO -  panda-0143:
05:14:54     INFO -  		Started Wed, 17 Dec 2014 05:14:54
05:14:54     INFO -  Running test ts_paint:
05:14:54     INFO -  		Started Wed, 17 Dec 2014 05:14:54
05:16:09     INFO -  mozversion INFO | application_buildid: 20141217035335
05:16:09     INFO -  mozversion INFO | application_changeset: 8549d93bb248
05:16:09     INFO -  mozversion INFO | application_display_name: Nightly
05:16:09     INFO -  mozversion INFO | application_name: Fennec
05:16:09     INFO -  mozversion INFO | application_repository: https://hg.mozilla.org/mozilla-central
05:16:09     INFO -  mozversion INFO | application_version: 37.0a1
05:16:09     INFO -  mozversion INFO | platform_buildid: 20141217035335
05:16:09     INFO -  mozversion INFO | platform_changeset: 8549d93bb248
05:16:09     INFO -  mozversion INFO | platform_repository: https://hg.mozilla.org/mozilla-central
05:40:11     INFO -  Remote Device Error: Unable to remove file '<built-in function dir>' on remote device
05:40:44    ERROR -  Traceback (most recent call last):
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/run_tests.py", line 296, in <module>
05:40:44     INFO -      main()
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/run_tests.py", line 293, in main
05:40:44     INFO -      sys.exit(run_tests(parser))
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/run_tests.py", line 233, in run_tests
05:40:44     INFO -      talos_results.add(mytest.runTest(browser_config, test))
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/ttest.py", line 480, in runTest
05:40:44     INFO -      self._ffprocess.runProgram(browser_config, command_args, timeout=timeout)
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/ffprocess_remote.py", line 195, in runProgram
05:40:44     INFO -      self.removeFile(remoteLog)
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/ffprocess_remote.py", line 167, in removeFile
05:40:44     INFO -      self.testAgent.removeFile(filename)
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/mozdevice/devicemanagerSUT.py", line 433, in removeFile
05:40:44     INFO -      if self.fileExists(filename):
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/mozdevice/devicemanagerSUT.py", line 416, in fileExists
05:40:44     INFO -      return filename in self.listFiles(containingpath)
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/mozdevice/devicemanagerSUT.py", line 422, in listFiles
05:40:44     INFO -      data = self._runCmds([{ 'cmd': 'cd ' + rootdir }, { 'cmd': 'ls' }])
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/mozdevice/devicemanagerSUT.py", line 151, in _runCmds
05:40:44     INFO -      self._sendCmds(cmdlist, outputfile, timeout, retryLimit=retryLimit)
05:40:44     INFO -    File "/builds/panda-0143/talos-data/talos/mozdevice/devicemanagerSUT.py", line 142, in _sendCmds
05:40:44     INFO -      raise DMError("Remote Device Error: unable to connect to %s after %s attempts" % (self.host, retryLimit))
05:40:44     INFO -  mozdevice.devicemanager.DMError: Remote Device Error: unable to connect to 10.26.128.126 after 5 attempts
05:40:44    ERROR - Return code: 1
05:40:44    FATAL - Failed talos ['python', '-u', '/builds/panda-0143/talos-data/talos/run_tests.py', u'--apkPath=/builds/panda-0143/test/build/fennec-37.0a1.en-US.android-arm.apk', '--noisy', 'local.yml', '--datazilla-url', 'https://datazilla.mozilla.org/talos', '--authfile', '/builds/panda-0143/test/oauth.txt'] command run.
05:40:44    FATAL - Running post_fatal callback...
05:40:44     INFO - Request 'http://mobile-imaging-001.p1.releng.scl3.mozilla.com/api/request/2642961/' deleted on cleanup
05:40:44    FATAL - Exiting -1

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Once we have autophone data reporting to treeherder, we can turn off remote-tspaint.  This is the only android talos test that is problematic and we have a reliable and more useful replacement, just some details to sort out.

Comment 2

[Geoff Brown [:gbrown]]

Can we stop running Android ts-paint? It has been failing (retrying) and hidden for more than 2 months now.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I believe in the last year all tspaint regressions have been caught by autophone!

I would also like to recommend turning off:
robopan: http://graphs.mozilla.org/graph.html#tests=[[174,63,29]]&sel=1403426740494.671,1424956381001,0,319672.1311475411&displayrange=365&datatype=running
* this is so noisy, we should just shut this off (3 alert instances in the last year)

tprovider: http://graphs.mozilla.org/graph.html#tests=[[200,63,29]]&sel=1393420508460,1424956508460,360.655737704918,786.8852459016393&displayrange=365&datatype=running
* this is so noisy, we should just shut this off 91 alert instance in the last year)

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Joel Maher (:jmaher) from comment #3)
Yeah, I'm fine with turning those off.

Comment 5

[Phil Ringnalda (:philor)]

And then they were, in bug 1141656