Closed Bug 829551 Opened 11 years ago Closed 10 years ago

Intermittent B2G "marionette.errors.TimeoutException: socket.timeout" (apparent emulator hang on startup)

Categories

(Firefox OS Graveyard :: General, defect)

ARM
Gonk (Firefox OS)
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: emorley, Unassigned)

References

Details

(Keywords: intermittent-failure)

Probably just a dupe of some of the other bugs, but I find these failure modes fairly unhelpful to grok (ideally we'd have all socket.timeouts caught and a constant message given, if they re all due to the same emulator issue)

b2g_ics_armv7a_gecko_emulator mozilla-inbound opt test reftest-6 on 2013-01-11 00:43:14 PST for push 11879a83e5e7

slave: talos-r3-fed-079

https://tbpl.mozilla.org/php/getParsedLog.php?id=18702752&tree=Mozilla-Inbound

{
00:48:40     INFO -  REFTEST INFO | runreftest.py | Running tests: start.
00:48:45     INFO -  MOZPROCESS WARNING: ProcessHandler.waitForFinish() is deprecated, use ProcessHandler.wait() instead
00:52:04     INFO -  Traceback (most recent call last):
00:52:04     INFO -    File "runreftestb2g.py", line 564, in main
00:52:04     INFO -      retVal = reftest.runTests(manifest, options, cmdlineArgs)
00:52:04     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/runreftest.py", line 140, in runTests
00:52:04     INFO -      timeout=options.timeout + 30.0)
00:52:04     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/automation.py", line 1072, in runApp
00:52:04     INFO -      stderr = subprocess.STDOUT)
00:52:04     INFO -    File "/home/cltbld/talos-slave/test/build/tests/reftest/b2gautomation.py", line 253, in Process
00:52:04     INFO -      session = self.marionette.start_session()
00:52:04     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/marionette/marionette.py", line 306, in start_session
00:52:04     INFO -      self.session = self._send_message('newSession', 'value')
00:52:04     INFO -    File "/home/cltbld/talos-slave/test/build/venv/lib/python2.6/site-packages/marionette/marionette.py", line 212, in _send_message
00:52:04    ERROR -      raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
00:52:04    ERROR -  TimeoutException: socket.timeout
00:52:04     INFO -  Automation Error: Exception caught while running tests
}
https://tbpl.mozilla.org/php/getParsedLog.php?id=24163871&tree=Mozilla-Inbound

I'd be remiss if I didn't point out that basically every single one of these happens at the same point. Jonathan, any ideas?

13:17:58     INFO -  copying /data/local/b2g/dictionaries/en-US.aff to /system/b2g/dictionaries/en-US.aff
13:17:58     INFO -  copying /data/local/b2g/dictionaries/en-US.dic to /system/b2g/dictionaries/en-US.dic
13:17:58     INFO -  restarting B2G
13:17:58     INFO -  Traceback (most recent call last):
13:17:58     INFO -    File "runreftestb2g.py", line 573, in <module>
13:17:58     INFO -      sys.exit(main())
13:17:58     INFO -    File "runreftestb2g.py", line 493, in main
13:17:58     INFO -      marionette = Marionette.getMarionetteOrExit(**kwargs)
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 291, in getMarionetteOrExit
13:17:58     INFO -      m = cls(*args, **kwargs)
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 266, in __init__
13:17:58     INFO -      busybox=busybox)
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/emulator.py", line 353, in setup
13:17:58     INFO -      self.wait_for_system_message(marionette)
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/emulator.py", line 256, in wait_for_system_message
13:17:58     INFO -      marionette.start_session()
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 444, in start_session
13:17:58     INFO -      self.session = self._send_message('newSession', 'value')
13:17:58     INFO -    File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette/marionette.py", line 340, in _send_message
13:17:58    ERROR -      raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
13:17:58    ERROR -  marionette.errors.TimeoutException: socket.timeout
Flags: needinfo?(jgriffin)
This has something to do with the way we're updating gecko in these static emulator snapshots.  The good news is, as soon as we get tests running against the full-stack emulator builds (which will be soon), this error will go away since we'll no longer be updating gecko.
Flags: needinfo?(jgriffin)
And another probably 30 per day - I can't be bothered to paste links into a "we're going to redo everything to get new and different bugs instead!" bug.
It was our hope that this error had something to do with updating gecko in the emulator at the beginning of a run, but apparently that was not the case.  :(

From the log, it looks like the emulator is hanging on startup.

I expect to see this in the logcat, indicating that Gaia has loaded:

I/Gecko   (   43): ###################################### forms.js loaded
I/Gecko   (   43): ############################### browserElementPanning.js loaded
I/Gecko   (   43): ######################## BrowserElementChildPreload.js loaded

But in these failing logs, this output never appears, which would seem to indicate that Gaia is never loaded.

I'll try to escalate this.
Flags: needinfo?(jgriffin)
Summary: Intermittent B2G "TimeoutException: socket.timeout" → Intermittent B2G "TimeoutException: socket.timeout" (apparent emulator hang on startup)
Depends on: 892958
https://tbpl.mozilla.org/php/getParsedLog.php?id=25412461&tree=Mozilla-Inbound
Summary: Intermittent B2G "TimeoutException: socket.timeout" (apparent emulator hang on startup) → Intermittent B2G "marionette.errors.TimeoutException: socket.timeout" (apparent emulator hang on startup)
Closing bugs where TBPLbot has previously commented, but have now not been modified for >3 months & do not contain the whiteboard strings for disabled/annotated tests or use the keyword leave-open. Filter on: mass-intermittent-bug-closure-2014-07
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.