The default bug view has changed. See this FAQ.

Intermittent B2G emulator "TEST-UNEXPECTED-FAIL | * | EOFError: telnet connection closed"

RESOLVED FIXED in Firefox 18

Status

Testing
Marionette
RESOLVED FIXED
5 years ago
3 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

({intermittent-failure})

unspecified
mozilla19
intermittent-failure
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox18 fixed, firefox19 fixed)

Details

Attachments

(1 attachment)

(Assignee)

Description

5 years ago
Sometimes WebAPI test runs are aborting during newSession in the first test with this stack trace:

Traceback (most recent call last):
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/marionette_test.py", line 78, in setUp
    self.marionette.start_session()
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/marionette.py", line 243, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/marionette.py", line 162, in _send_message
    port = self.emulator.restart(self.local_port)
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator.py", line 255, in restart
    self.start()
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator.py", line 309, in start
    self.geo.set_default_location()
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator_geo.py", line 13, in set_default_location
    self.set_location(self.lon, self.lat)
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator_geo.py", line 16, in set_location
    self.emulator._run_telnet('geo fix %0.5f %0.5f' % (self.lon, self.lat))
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator.py", line 215, in _run_telnet
    return self._get_telnet_response(command)
  File "/data/jenkins/workspace/webapi-marionette-test/testing/marionette/client/marionette/emulator.py", line 204, in _get_telnet_response
    line = self.telnet.read_until('\n')
  File "/usr/lib/python2.7/telnetlib.py", line 319, in read_until
    return self.read_very_lazy()
  File "/usr/lib/python2.7/telnetlib.py", line 395, in read_very_lazy
    raise EOFError, 'telnet connection closed'
TEST-UNEXPECTED-FAIL : EOFError: telnet connection closed

It appears that setting the default geolocation is sometimes causing the emulator to crash.  I'm going to try and add a delay before this to see if that clears this up; if not, I will disable it.
(Assignee)

Comment 1

5 years ago
Created attachment 672586 [details] [diff] [review]
Add a delay before setting default emulator geolocation
(Assignee)

Comment 2

5 years ago
https://hg.mozilla.org/mozilla-central/rev/f549d4510829.

Leaving open for further tweaks.
Assignee: nobody → jgriffin
Target Milestone: --- → mozilla19
One of the changesets here turned marionette orange:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=marionette&rev=67f8d99d80fb

In that range, I believe this bug is the most likely contender.
> One of the changesets here turned marionette orange:

Fixed link:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=marionette&rev=67f8d99d80fb&noignore=1
(Assignee)

Comment 5

5 years ago
I haven't seen this problem recur, I'm closing this.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Whiteboard: [automation-needed-in-aurora]
(Assignee)

Comment 6

5 years ago
https://hg.mozilla.org/projects/ash/rev/6b8114dced68
(Assignee)

Comment 7

4 years ago
https://hg.mozilla.org/releases/mozilla-aurora/rev/647847eb85ce
status-firefox18: --- → fixed
status-firefox19: --- → fixed
Whiteboard: [automation-needed-in-aurora]
Still seeing this occasionally on inbound (ignore the mozharness logcat exception):
https://tbpl.mozilla.org/php/getParsedLog.php?id=17217191&tree=Mozilla-Inbound
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 9

4 years ago
This is another weird emulator problem.  I'm thinking the best fix is to add it to the retry code, so if it happens, we just try everything again.
https://tbpl.mozilla.org/php/getParsedLog.php?id=17290290&tree=Mozilla-Aurora
Whiteboard: [orange]
https://tbpl.mozilla.org/php/getParsedLog.php?id=17293148&tree=Mozilla-Inbound
Keywords: intermittent-failure
Summary: TEST-UNEXPECTED-FAIL : EOFError: telnet connection closed → Intermittent emulator TEST-UNEXPECTED-FAIL : EOFError: telnet connection closed
https://tbpl.mozilla.org/php/getParsedLog.php?id=17298891&tree=Mozilla-Inbound
Looking at that last log I see:
00:57:25     INFO - F/libc    (   43): Fatal signal 11 (SIGSEGV) at 0x00000047 (code=1)

Maybe this bug is a side effect of bug 814695. Though the tests still seem to run even after that seg fault so maybe it is completely unrelated.
Whiteboard: [orange]
https://tbpl.mozilla.org/php/getParsedLog.php?id=17358883&tree=Mozilla-Beta
(Assignee)

Comment 15

4 years ago
(In reply to Ryan VanderMeulen from comment #14)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=17358883&tree=Mozilla-Beta

This is an emulator crash, but one that occurred during a non-WebAPI test.
https://tbpl.mozilla.org/php/getParsedLog.php?id=17374301&tree=Firefox
(Assignee)

Comment 17

4 years ago
(In reply to Ed Morley [:edmorley, UTC, email: emorley@moco] from comment #16)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=17374301&tree=Firefox

This is an emulator crash during the test test_execute_async_script.py:test_execute_permission.  We really need better crash detection and messaging.
https://tbpl.mozilla.org/php/getParsedLog.php?id=17362640&tree=Mozilla-Beta

(sorry, but gotta post this somewhere...)
(Assignee)

Comment 19

4 years ago
(In reply to Ryan VanderMeulen from comment #18)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=17362640&tree=Mozilla-Beta
> 
> (sorry, but gotta post this somewhere...)

This is a crash during the test test_execute_script.py:test_that_we_can_pass_in_floats.  I'm beginning to think that re-enabling the telephony tests has caused some weird problem in which the emulator crashes after some delay.
https://tbpl.mozilla.org/php/getParsedLog.php?id=17384033&tree=Mozilla-Beta
(Assignee)

Comment 21

4 years ago
(In reply to Ryan VanderMeulen from comment #20)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=17384033&tree=Mozilla-Beta

Another random emulator crash, the one during test_perf.py:test_perf_basic
https://tbpl.mozilla.org/php/getParsedLog.php?id=17391237&tree=Mozilla-Inbound
(Assignee)

Comment 23

4 years ago
(In reply to Ryan VanderMeulen from comment #22)
> https://tbpl.mozilla.org/php/getParsedLog.php?id=17391237&tree=Mozilla-
> Inbound

Another random emulator crash, this one during test_execute_async_script.py:test_execute_no_return
https://tbpl.mozilla.org/php/getParsedLog.php?id=17451796&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17466770&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17467246&tree=Mozilla-Inbound
(Assignee)

Comment 27

4 years ago
This is likely fallout from re-enabling the telephony tests in bug 790463, but it's hard to be sure because this is so infrequent.

One idea that ahal and I had was to cause this error to automatically retry the run, which would effectively hide it.
https://tbpl.mozilla.org/php/getParsedLog.php?id=17505830&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17516611&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17532816&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=17545168&tree=Mozilla-Inbound
(In reply to Jonathan Griffin (:jgriffin) from comment #27)
> This is likely fallout from re-enabling the telephony tests in bug 790463,
> but it's hard to be sure because this is so infrequent.
> 
> One idea that ahal and I had was to cause this error to automatically retry
> the run, which would effectively hide it.

Seems to be happening more frequently now. We already have the code to retry the job in mozharness, we'd just need to add a new magic string to it (boo)
(Assignee)

Updated

4 years ago
Duplicate of this bug: 817250
https://tbpl.mozilla.org/php/getParsedLog.php?id=17560310&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17559366&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17568863&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17644512&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17551792&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=17664577&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17741506&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=17743251&tree=Mozilla-Beta
https://tbpl.mozilla.org/php/getParsedLog.php?id=17866267&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=17853667&tree=Mozilla-Beta
Blocks: 823076
https://tbpl.mozilla.org/php/getParsedLog.php?id=18094409&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18101604&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=18232343&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18235680&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18234987&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=18270015&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=18338101&tree=Mozilla-B2g18
https://tbpl.mozilla.org/php/getParsedLog.php?id=18341267&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18359097&tree=Firefox
https://tbpl.mozilla.org/php/getParsedLog.php?id=18373720&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18383555&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=18387656&tree=Mozilla-Aurora
https://tbpl.mozilla.org/php/getParsedLog.php?id=18398398&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=18398951&tree=Mozilla-Inbound
(Assignee)

Updated

4 years ago
Duplicate of this bug: 824045
(Assignee)

Updated

4 years ago
Duplicate of this bug: 823067
(Assignee)

Updated

4 years ago
Duplicate of this bug: 824079
https://tbpl.mozilla.org/php/getParsedLog.php?id=18409170&tree=Mozilla-Aurora
Duplicate of this bug: 826131
Duplicate of this bug: 824059
Summary: Intermittent emulator TEST-UNEXPECTED-FAIL : EOFError: telnet connection closed → Intermittent B2G emulator "TEST-UNEXPECTED-FAIL | * | EOFError: telnet connection closed"
Depends on: 813292
Depends on: 826182
https://tbpl.mozilla.org/php/getParsedLog.php?id=18402409&tree=Fx-Team
(Assignee)

Comment 64

4 years ago
This particular problem has been fixed due to bug 826132.
Status: REOPENED → RESOLVED
Last Resolved: 5 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.