Closed Bug 830431 Opened 8 years ago Closed 8 years ago

B2G panda tests failing with TimeoutException: socket.timeout or error: [Errno 113] No route to host

Categories

(Testing :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(b2g18 fixed)

RESOLVED FIXED
mozilla21
Tracking Status
b2g18 --- fixed

People

(Reporter: armenzg, Assigned: jgriffin)

References

Details

Attachments

(1 file)

08:38:03     INFO - Copy/paste: /builds/panda-0127/test/build/venv/bin/gaiatest --address panda-0127:2828 --type b2g /builds/panda-0127/test/build/tests/gaiatest/gaiatest/tests/manifest.ini
08:38:03     INFO -  running webserver on http://10.12.128.20:51548/
08:38:03     INFO -  TEST-START test_settings.py
08:41:04     INFO -  test_get_all_settings (test_settings.TestSettings) ... ERROR
08:44:04     INFO -  test_set_named_setting (test_settings.TestSettings) ... ERROR
08:47:04     INFO -  test_set_volume (test_settings.TestSettings) ... ERROR
08:47:04     INFO -  ======================================================================
08:47:04     INFO -  ERROR: test_get_all_settings (test_settings.TestSettings)
08:47:04     INFO -  ----------------------------------------------------------------------
08:47:04     INFO -  Traceback (most recent call last):
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 203, in setUp
08:47:04     INFO -      self.marionette.setup_touch()
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette_touch.py", line 18, in setup_touch
08:47:04     INFO -      self.import_script(self.library)
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 515, in import_script
08:47:04     INFO -      return self._send_message('importScript', 'ok', script=js)
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 215, in _send_message
08:47:04     INFO -      raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
08:47:04     INFO -  TEST-UNEXPECTED-FAIL | test_settings.py TestSettings.test_get_all_settings | TimeoutException: socket.timeout


https://tbpl.mozilla.org/php/getParsedLog.php?id=18785788&tree=Cedar&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=18783222&tree=Cedar&full=1

It happened between
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_panda&rev=bacf122225e9
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_panda&rev=ae1822e4f9c4
Also to note that when the job takes 2 hours it will be go red and if it finishes before 2 hours it will turn orange.
The Ash gaia tests also timeout but not all of them (unlike on Cedar):
https://tbpl.mozilla.org/?tree=Ash&jobname=b2g_panda
I ran a local test.  The problem is another socket issue.  The tests work fine when using adb (so you're not connecting to the Marionette socket remotely); they fail when connecting via a remote host.
(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I ran a local test.  The problem is another socket issue.  The tests work
> fine when using adb (so you're not connecting to the Marionette socket
> remotely); they fail when connecting via a remote host.

No, it's not quiet the simple.  The test starts, but dies in setup_touch, as indicated by the stack trace in the original description of this bug.  But, this only happens when not using adb port forwarding!
Due to the recent mouse/touch event changes, gaia-ui-tests was altered to import synthetic_gestures.js (a 21k JS file).  For some reason, when the client sends this to the panda, a little of the data at the end of the file is missing upon its reception, and this is causing the socket.timeout errors.

When we use adb port forwarding, adb only sends 4096 bytes at a time, and we don't see this error.

This patch changes client.py to send data in 4kb chunks, which resolves this problem.
Attachment #703677 - Flags: review?(mdas)
Comment on attachment 703677 [details] [diff] [review]
Send data from client in 4kb chunks,

Review of attachment 703677 [details] [diff] [review]:
-----------------------------------------------------------------

r+, surprising problem, but a good fix!
Attachment #703677 - Flags: review?(mdas) → review+
Armen, I've landed this directly on cedar (https://hg.mozilla.org/projects/cedar/rev/7d66b54e068b), so we don't have to wait for the m-i -> m-c -> cedar merges.
Assignee: nobody → jgriffin
This have cleared out the timeouts (thanks!) and we're now passing 6 tests.
gaia-ui-tests: 6/32/1
https://tbpl.mozilla.org/php/getParsedLog.php?id=18933827&tree=Cedar&full=1

We can now see a bunch of JavascriptException: TypeError: window.wrappedJSObject.SettingsListener is undefined plus few new oranges.

Would you like to deal with them on this bug or a new one?
Some of the current problems will be solved by another merge of gaia-ui-tests to gaia.  Let's do that and then see where we are.
Depends on: 832492
https://hg.mozilla.org/mozilla-central/rev/c635420087c8
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla21
The only major recent change is that I updated the snapshot on bug 833096.

I will re-trigger the previous changesets which should use the older snapshot IIUC and see if it is related.

It is happening again:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19015010&tree=Cedar&full=1
08:12:15     INFO -  TEST-UNEXPECTED-FAIL | test_clock_turn_on_off_alarm.py TestClockTurnOnOffAlarm.test_clock_turn_on_off_alarm | TimeoutException: socket.timeout
08:12:21     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_check_today_date | error: [Errno 113] No route to host
08:12:21     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_cards_view | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_kill_app_from_cards_view | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_that_app_can_be_launched_from_cards_view | error: [Errno 113] No route to host
08:12:33     INFO -  TEST-UNEXPECTED-FAIL | test_delete_app.py TestDeleteApp.test_delete_app | error: [Errno 113] No route to host
08:12:36     INFO -  TEST-UNEXPECTED-FAIL | test_keyboard.py TestKeyboard.test_keyboard_basic | error: [Errno 113] No route to host
08:12:39     INFO -  TEST-UNEXPECTED-FAIL | test_launch_app.py TestLaunchApp.test_launch_app | error: [Errno 113] No route to host
08:12:42     INFO -  TEST-UNEXPECTED-FAIL | test_wallpaper.py TestWallpaper.test_change_wallpaper | error: [Errno 113] No route to host
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
With gaia-central I get this instead:
08:38:37 INFO - TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | clearElement@chrome://marionette/content/marionette-listener.js:855

https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_&rev=d66b666e21c6

I have also re-triggered the 2 jobs for d66b666e21c6.
hrmm... it seems that it was intermittent :S
(In reply to Armen Zambrano G. [:armenzg] from comment #15)
> hrmm... it seems that it was intermittent :S

The specific error is:

InvalidElementStateException: Element must be user-editable in order to clear it.

might be some timing problem in the test case; I doubt it's a harness problem
No longer blocks: 802317
The original cause of this bug has been fixed.  If we see this problem again, let's open a new bug.
Status: REOPENED → RESOLVED
Closed: 8 years ago8 years ago
Resolution: --- → FIXED
Blocks: 834266
You need to log in before you can comment on or make changes to this bug.