830431 - B2G panda tests failing with TimeoutException: socket.timeout or error: [Errno 113] No route to host

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Armen [:armenzg]]

08:38:03     INFO - Copy/paste: /builds/panda-0127/test/build/venv/bin/gaiatest --address panda-0127:2828 --type b2g /builds/panda-0127/test/build/tests/gaiatest/gaiatest/tests/manifest.ini
08:38:03     INFO -  running webserver on http://10.12.128.20:51548/
08:38:03     INFO -  TEST-START test_settings.py
08:41:04     INFO -  test_get_all_settings (test_settings.TestSettings) ... ERROR
08:44:04     INFO -  test_set_named_setting (test_settings.TestSettings) ... ERROR
08:47:04     INFO -  test_set_volume (test_settings.TestSettings) ... ERROR
08:47:04     INFO -  ======================================================================
08:47:04     INFO -  ERROR: test_get_all_settings (test_settings.TestSettings)
08:47:04     INFO -  ----------------------------------------------------------------------
08:47:04     INFO -  Traceback (most recent call last):
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 203, in setUp
08:47:04     INFO -      self.marionette.setup_touch()
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette_touch.py", line 18, in setup_touch
08:47:04     INFO -      self.import_script(self.library)
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 515, in import_script
08:47:04     INFO -      return self._send_message('importScript', 'ok', script=js)
08:47:04     INFO -    File "/builds/panda-0127/test/build/venv/lib/python2.7/site-packages/marionette/marionette.py", line 215, in _send_message
08:47:04     INFO -      raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
08:47:04     INFO -  TEST-UNEXPECTED-FAIL | test_settings.py TestSettings.test_get_all_settings | TimeoutException: socket.timeout


https://tbpl.mozilla.org/php/getParsedLog.php?id=18785788&tree=Cedar&full=1
https://tbpl.mozilla.org/php/getParsedLog.php?id=18783222&tree=Cedar&full=1

It happened between
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_panda&rev=bacf122225e9
https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_panda&rev=ae1822e4f9c4

Comment 1

[Armen [:armenzg]]

Also to note that when the job takes 2 hours it will be go red and if it finishes before 2 hours it will turn orange.

Comment 2

[Armen [:armenzg]]

The Ash gaia tests also timeout but not all of them (unlike on Cedar):
https://tbpl.mozilla.org/?tree=Ash&jobname=b2g_panda

Comment 3

[Jonathan Griffin (:jgriffin)]

I ran a local test.  The problem is another socket issue.  The tests work fine when using adb (so you're not connecting to the Marionette socket remotely); they fail when connecting via a remote host.

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I ran a local test.  The problem is another socket issue.  The tests work
> fine when using adb (so you're not connecting to the Marionette socket
> remotely); they fail when connecting via a remote host.

No, it's not quiet the simple.  The test starts, but dies in setup_touch, as indicated by the stack trace in the original description of this bug.  But, this only happens when not using adb port forwarding!

Comment 5

[Jonathan Griffin (:jgriffin)]

Attachment: Send data from client in 4kb chunks,

Due to the recent mouse/touch event changes, gaia-ui-tests was altered to import synthetic_gestures.js (a 21k JS file).  For some reason, when the client sends this to the panda, a little of the data at the end of the file is missing upon its reception, and this is causing the socket.timeout errors.

When we use adb port forwarding, adb only sends 4096 bytes at a time, and we don't see this error.

This patch changes client.py to send data in 4kb chunks, which resolves this problem.

Comment 6

[Jonathan Griffin (:jgriffin)]

pushed to try:
B2G: https://tbpl.mozilla.org/?tree=Try&rev=89abed2930f5
desktop: https://tbpl.mozilla.org/?tree=Try&rev=8838ce9b95eb

Comment 7

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 703677 [details] [diff] [review]
Send data from client in 4kb chunks,

Review of attachment 703677 [details] [diff] [review]:
-----------------------------------------------------------------

r+, surprising problem, but a good fix!

Comment 8

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c635420087c8
https://hg.mozilla.org/releases/mozilla-b2g18/rev/e427bf1cfd2d

Comment 9

[Jonathan Griffin (:jgriffin)]

Armen, I've landed this directly on cedar (https://hg.mozilla.org/projects/cedar/rev/7d66b54e068b), so we don't have to wait for the m-i -> m-c -> cedar merges.

Comment 10

[Armen [:armenzg]]

This have cleared out the timeouts (thanks!) and we're now passing 6 tests.
gaia-ui-tests: 6/32/1
https://tbpl.mozilla.org/php/getParsedLog.php?id=18933827&tree=Cedar&full=1

We can now see a bunch of JavascriptException: TypeError: window.wrappedJSObject.SettingsListener is undefined plus few new oranges.

Would you like to deal with them on this bug or a new one?

Comment 11

[Jonathan Griffin (:jgriffin)]

Some of the current problems will be solved by another merge of gaia-ui-tests to gaia.  Let's do that and then see where we are.

Comment 12

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/mozilla-central/rev/c635420087c8

Comment 13

[Armen [:armenzg]]

The only major recent change is that I updated the snapshot on bug 833096.

I will re-trigger the previous changesets which should use the older snapshot IIUC and see if it is related.

It is happening again:
https://tbpl.mozilla.org/php/getParsedLog.php?id=19015010&tree=Cedar&full=1
08:12:15     INFO -  TEST-UNEXPECTED-FAIL | test_clock_turn_on_off_alarm.py TestClockTurnOnOffAlarm.test_clock_turn_on_off_alarm | TimeoutException: socket.timeout
08:12:21     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_check_today_date | error: [Errno 113] No route to host
08:12:21     INFO -  TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_cards_view | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_kill_app_from_cards_view | error: [Errno 113] No route to host
08:12:30     INFO -  TEST-UNEXPECTED-FAIL | test_cards_view.py TestCardsView.test_that_app_can_be_launched_from_cards_view | error: [Errno 113] No route to host
08:12:33     INFO -  TEST-UNEXPECTED-FAIL | test_delete_app.py TestDeleteApp.test_delete_app | error: [Errno 113] No route to host
08:12:36     INFO -  TEST-UNEXPECTED-FAIL | test_keyboard.py TestKeyboard.test_keyboard_basic | error: [Errno 113] No route to host
08:12:39     INFO -  TEST-UNEXPECTED-FAIL | test_launch_app.py TestLaunchApp.test_launch_app | error: [Errno 113] No route to host
08:12:42     INFO -  TEST-UNEXPECTED-FAIL | test_wallpaper.py TestWallpaper.test_change_wallpaper | error: [Errno 113] No route to host

Comment 14

[Armen [:armenzg]]

With gaia-central I get this instead:
08:38:37 INFO - TEST-UNEXPECTED-FAIL | test_calendar.py TestCalendar.test_that_new_event_appears_on_all_calendar_views | clearElement@chrome://marionette/content/marionette-listener.js:855

https://tbpl.mozilla.org/?tree=Cedar&jobname=b2g_&rev=d66b666e21c6

I have also re-triggered the 2 jobs for d66b666e21c6.

Comment 15

[Armen [:armenzg]]

hrmm... it seems that it was intermittent :S

Comment 16

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano G. [:armenzg] from comment #15)
> hrmm... it seems that it was intermittent :S

The specific error is:

InvalidElementStateException: Element must be user-editable in order to clear it.

might be some timing problem in the test case; I doubt it's a harness problem

Comment 17

[Jonathan Griffin (:jgriffin)]

The original cause of this bug has been fixed.  If we see this problem again, let's open a new bug.