Closed Bug 1208725 Opened 9 years ago Closed 9 years ago

Intermittent B2G ICS Emulator opt No tests run or test summary not found

Categories

(Firefox OS Graveyard :: Emulator, defect)

Product:
Firefox OS Graveyard
Component:
Emulator
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox46 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox46 --- fixed

People

(Reporter: aryx, Assigned: nigelb)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file, 1 obsolete file)

MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal
58 bytes, text/x-review-board-request
ahal
: review+
Details
From my shallow and probably incorrect investigation, this is what rings out to me from the logs: 21:44:49 INFO - Traceback (most recent call last): 21:44:49 INFO - File "runtestsb2g.py", line 206, in run_tests 21:44:49 INFO - self.marionette.start_session() 21:44:49 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1180, in start_session 21:44:49 INFO - resp = self._send_message("newSession", body) 21:44:49 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette_driver/decorators.py", line 36, in _ 21:44:49 INFO - return func(*args, **kwargs) 21:44:49 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette_driver/marionette.py", line 708, in _send_message 21:44:49 INFO - msg = self.client.receive() 21:44:49 INFO - File "/builds/slave/test/build/venv/local/lib/python2.7/site-packages/marionette_transport/transport.py", line 208, in receive 21:44:49 INFO - raise socket.timeout("connection timed out after %ds" % self.socket_timeout) 21:44:49 INFO - timeout: connection timed out after 360s 21:44:49 INFO - 0 ERROR Automation Error: Received unexpected exception while running application 21:44:49 INFO - Stopping web server
Talked to AutomatedTester and his suspicion that it's the emulator being slow to start up. Which means bug 1160010 is also related.
Are you the right person to investigate the emulator slowness or can you point me in the direction of the right person?
Flags: needinfo?(gbrown)
I am not convinced this is emulator slowness. In recent failure logs I see lots of SettingsManager warnings: 15:48:05 WARNING - 11-24 23:46:03.385 E/GeckoConsole( 799): [JavaScript Error: "Settings queue head blocked at {838fa984-cf09-4422-a4d7-2e15f2c08afa} for 39.909 secs, Settings API may be soft lockup. Lock from: SettingsLock@jar:file:///system/b2g/omni.ja!/components/SettingsManager.js:75:17 15:48:05 INFO - 11-24 23:46:03.385 E/GeckoConsole( 799): SettingsManager.prototype.createLock@jar:file:///system/b2g/omni.ja!/components/SettingsManager.js:296:16 15:48:05 INFO - 11-24 23:46:03.385 E/GeckoConsole( 799): sl_observe@chrome://b2g/content/settings.js:69:15 15:48:05 INFO - 11-24 23:46:03.385 E/GeckoConsole( 799): @chrome://b2g/content/settings.js:99:1 15:48:05 INFO - 11-24 23:46:03.385 E/GeckoConsole( 799): " {file: "resource://gre/modules/SettingsRequestManager.jsm" line: 1041}] 15:48:05 INFO - 11-24 23:46:03.385 I/Gecko ( 799): -*- SettingsRequestManager: Settings queue head blocked at {838fa984-cf09-4422-a4d7-2e15f2c08afa} for 39.909 secs, Settings API may be soft lockup. Lock from: SettingsLock@jar:file:///system/b2g/omni.ja!/components/SettingsManager.js:75:17 15:48:05 INFO - 11-24 23:46:03.385 I/Gecko ( 799): SettingsManager.prototype.createLock@jar:file:///system/b2g/omni.ja!/components/SettingsManager.js:296:16 15:48:05 INFO - 11-24 23:46:03.385 I/Gecko ( 799): sl_observe@chrome://b2g/content/settings.js:69:15 15:48:05 INFO - 11-24 23:46:03.385 I/Gecko ( 799): @chrome://b2g/content/settings.js:99:1 I'm not very skilled at interpretting b2g logs, but I'd start there -- can you find a SettingsManager expert? (Over in bug 1160010, there are similar "No tests run" messages, for Android. I suspect some of those failures, for a certain period, were caused by emulator slowness, in that the emulator was accidentally being run on the wrong type of aws instance. But I am pretty sure that issue has been resolved. Recent failures in that bug are more often attributable to Gecko crashes on startup or test hangs.)
Flags: needinfo?(gbrown)
See Also: → 1119727
Hello, can either of you look at the SettingsManager slowness and check if it's what's causing a large number of test failures on b2g?
Flags: needinfo?(gsquelart)
Flags: needinfo?(cyu)
I don't know enough about b2g, sorry. Glancing at a few logs, it could be a test server issue. Maja, any idea please?
Flags: needinfo?(gsquelart) → needinfo?(mjzffr)
(In reply to Nigel Babu [:nigelb] from comment #63) > Hello, can either of you look at the SettingsManager slowness and check if > it's what's causing a large number of test failures on b2g? SettingsManager is not the right one to blame. See http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-inbound/sha512/d2579bc703c925d93c13f1bb4b21fa98415f8f11178f6af95472c8d1e7920be4ee2c87bea80f1656898678c1a8adcf5bf92e76f495a02b7096e12691c5f3e67c The tests did run (the 1st time b2g got started (pid=44) and the 2nd time b2g got started (pid=750)). Lots of log entries showed that tests were run in the mochitest process (pid=850) like > 09-26 02:46:38.051 I/GeckoDump( 852): ⰲ겿{"action":"test_end","time":1443235598053,"thread":"","js_source":"TestRunner","pid":null,"source":"mochitest","test":"/tests/dom/xslt/tests/mochitest/test_parameter.html","status":"OK","message":"Finished in 2341ms","extra":{"runtime":2341}}ⰲ겿 It's more likely test infrastructure issue.
Flags: needinfo?(cyu)
(In reply to Gerald Squelart [:gerald] from comment #64) > I don't know enough about b2g, sorry. > Glancing at a few logs, it could be a test server issue. Maja, any idea > please? Sorry, I don't know much about b2g either. The logs indicate that b2g emulator is slow to start. It's a releng/taskcluster issue according to :AutomatedTester.
Flags: needinfo?(mjzffr)
This looks test infra related to me. We're either not waiting long enough somewhere, or we're not waiting for the right thing. I don't think taskcluster would have anything to do with this, and I agree that settingsmanager is most likely unrelated (I bet those same settingsmanager errors show up in a passing job too). From comment 65, the tests are running, just not showing up in the normal log due to the python side falling over.
Andrew, I'm going to do a try run with an increased timeout that you recommended the last time we talked about this.
Comment on attachment 8701043 [details] MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28873/diff/1-2/
Attachment #8701043 - Flags: review?(jgriffin)
Attachment #8701043 - Flags: review?(ato)
Attachment #8701043 - Flags: review?(ahalberstadt)
Comment on attachment 8701043 [details] MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28873/diff/1-2/
Comment on attachment 8701043 [details] MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal Review request updated; see interdiff: https://reviewboard.mozilla.org/r/28873/diff/1-2/
Attachment #8701043 - Attachment description: MozReview Request: Bug 1208725 - Increase the timeout for b2g tests → MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal
Attachment #8701043 - Flags: review?(jgriffin)
Attachment #8701043 - Flags: review?(ato)
Attachment #8701054 - Attachment is obsolete: true
Attachment #8701043 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8701043 [details] MozReview Request: Bug 1208725 - Increase the timeout for b2g tests, r=ahal https://reviewboard.mozilla.org/r/28873/#review25641 Lgtm!
Assignee: nobody → nigelbabu
Status: NEW → ASSIGNED
https://hg.mozilla.org/mozilla-central/rev/7d2475c0d623
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
The mochitest failures have now morphed into this: https://treeherder.mozilla.org/logviewer.html#?job_id=18924348&repo=mozilla-inbound And we still have Cpp failures that look like this: https://treeherder.mozilla.org/logviewer.html#?job_id=3621710&repo=b2g-inbound Should the Cpp ones be split out into their own bug?
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: