1057597 - WebRTC tests do not work on Aurora

Status: RESOLVED INVALID

(Core :: WebRTC, defect)

Description

[Syd Polk :sydpolk]

Using the steeplechase infrastructure, the WebRTC tests fail on Aurora:

python /home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py --binary /home/mozilla/firefoxes/aurora/linux64/firefox/firefox --save-logs-to /home/mozilla/jenkins/workspace/aurora-linux64-linux64/logs --specialpowers-path /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/specialpowers --prefs-file /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/prefs_general.js --signalling-server http://172.16.141.51:8080 --html-manifest /home/mozilla/firefoxes/nightly/linux64/tests/steeplechase/tests/steeplechase.ini --host1 172.16.141.52:20701 --host2 172.16.141.53:20701
steeplechase INFO | Pushing app to Client 1...

steeplechase INFO | Pushing app to Client 2...

steeplechase INFO | Waiting for results...

Writing profile for Client 1...
Pushing profile to Client 1...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', 'http://172.16.141.51:46262/index.html']
Writing profile for Client 2...
Pushing profile to Client 2...
cmd: ['/tmp/tests/steeplechase/app/firefox', '-no-remote', '-profile', '/tmp/tests/steeplechase/profile', 'http://172.16.141.51:46262/index.html']
Traceback (most recent call last):
  File "/home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py", line 311, in <module>
    sys.exit(0 if main(sys.argv[1:]) else 1)
  File "/home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py", line 301, in main
    html_pass_count, html_fail_count = test.run()
  File "/home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py", line 187, in run
    passes, failures = result
TypeError: 'NoneType' object is not iterable
Exception in thread Client 1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py", line 100, in run
    output = dm.shellCheckOutput(cmd, env=env)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanager.py", line 391, in shellCheckOutput
    retval = self.shell(cmd, buf, env=env, cwd=cwd, timeout=timeout, root=root)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanagerSUT.py", line 326, in shell
    self._sendCmds([{ 'cmd': '%s %s' % (cmd, cmdline) }], outputfile, timeout)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanagerSUT.py", line 136, in _sendCmds
    raise err
DMError: Automation Error: Timeout in command exec "MOZ_CRASHREPORTER_NO_REPORT=1,XPCOM_DEBUG_BREAK=warn,DISPLAY=:0" /tmp/tests/steeplechase/app/firefox -no-remote -profile /tmp/tests/steeplechase/profile http://172.16.141.51:46262/index.html

Exception in thread Client 2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/mozilla/jenkins/workspace/aurora-linux64-linux64/steeplechase/steeplechase/runsteeplechase.py", line 100, in run
    output = dm.shellCheckOutput(cmd, env=env)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanager.py", line 391, in shellCheckOutput
    retval = self.shell(cmd, buf, env=env, cwd=cwd, timeout=timeout, root=root)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanagerSUT.py", line 326, in shell
    self._sendCmds([{ 'cmd': '%s %s' % (cmd, cmdline) }], outputfile, timeout)
  File "/usr/local/lib/python2.7/dist-packages/mozdevice-0.37-py2.7.egg/mozdevice/devicemanagerSUT.py", line 136, in _sendCmds
    raise err
DMError: Automation Error: Timeout in command exec "MOZ_CRASHREPORTER_NO_REPORT=1,XPCOM_DEBUG_BREAK=warn,DISPLAY=:0" /tmp/tests/steeplechase/app/firefox -no-remote -profile /tmp/tests/steeplechase/profile http://172.16.141.51:46262/index.html

Comment 1

[Syd Polk :sydpolk]

Need to use aurora test assets.

Comment 2

[Nils Ohlmeier [:drno]]

Unfortunately the output in these Negatus timeout cases is completely useless.
To have at least the possibility to debug such issues we should either:
a) increase the Negatus timeout to be higher then the mochitest timeout, to allow mochitest to timeout and then Negatus should get some log output why mochitest ran into issues
b) execute Firefox on the test clients always with a wrapper scripts which on the client always writes stdout + stderr into a file and if steeplechase ran into a Negatus timeout it would try to fetch that stdout/stderr log from the client. We use such a script in the sunny day env already. It is firefox-tee in the mozwebrtc repository.
(The error handling of timeouts on the steeplechase side might actually in general be a good idea, because I think in this case it should be save to run request via Negatus to run a cleanup script on the client (to kill all Firefox), before that clients get assigned to another test run by Jenkins.)