Closed Bug 1246283 Opened 7 years ago Closed 7 years ago

many failures while running marionette jobs on taskcluster/docker instead of buildbot

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jmaher, Assigned: armenzg)

References

Details

TEST-UNEXPECTED-FAIL | test_typing.py TestTyping.testCutAndPasteShortcuts | AssertionError: u'http://127.0.0.1:51119/javascriptPage.html' != ''
TEST-UNEXPECTED-ERROR | test_about_pages.py TestAboutPages.test_navigate_shortcut_key | TimeoutException: TimeoutException: wait_for_condition timed out
TEST-UNEXPECTED-ERROR | test_modal_dialogs.py TestGlobalModals.test_alert_accept | TimeoutException: TimeoutException: Timed out after 5.1 seconds
TEST-UNEXPECTED-ERROR | test_modal_dialogs.py TestTabModals.test_alert_accept | TimeoutException: TimeoutException: Timed out after 5.1 seconds
TEST-UNEXPECTED-ERROR | test_key_actions.py TestKeyActions.test_open_in_new_window_shortcut | TimeoutException: TimeoutException: wait_for_condition timed out
TEST-UNEXPECTED-ERROR | test_shared_all.py TestSharedUnits.test_units | timeout: connection timed out after 360s 


you can see this is a common trend:
https://treeherder.mozilla.org/#/jobs?repo=try&author=armenzg@mozilla.com&selectedJob=16412573&filter-searchStr=Mn
:ato, can you help us figure out what is going on and how to work towards getting this green?  I am unfamiliar with the mn job/tests/harness- so this might be a simple thing to figure out.
Flags: needinfo?(ato)
Are those tests running headless? Do we have a window manager? This is actually something we have problems with when running our tests on Travis. So Andreas gave us this nice advice: bug 1240179 comment 5.
there is a window manager- the same one that runs reftest, mochitest, and web-platform-test.

the steps taken in the comment are similar to what we do to setup the docker image.
Aside from whimboo’s advice above, I can’t think of anything.  Do you have reproduction steps?
Flags: needinfo?(ato)
Here just my 2c while quickly observing the failures and checking the test content:

1. The failure in TestTyping.testCutAndPasteShortcuts we expect the urlbar to be empty after Meta+A + Meta+Backspace:

https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_typing.py?from=test_typing.py#82

As it looks like the backspace key was not sent correctly. Maybe even together with 'a' but that's hard to tell here. Keep in mind that we are in chrome context here. I could assume that the urlbar does not have focus yet. So a click() on it before may fix it.

2. test_about_pages.py
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_about_pages.py?from=test_about_pages.py+#

Those lines show me that here we are trying to open the add-on manager via meta+shift+a. Keep in mind that this shortcut is known to be broken on Linux if any text field has the focus. I'm not sure why this is part of this test actually. It has nothing to do with about pages.

3. TestGlobalModals.test_alert_accept

Not sure whats up with that. As long as the loaded HTML page is valid I can only assume that the click was not propagated correctly. That should not require a window manager. I'm not sure if Marionette server actually checks if actions were executed correctly.

4. test_key_actions.py TestKeyActions.test_open_in_new_window_shortcut

I do not understand what this test is trying to do.

https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_key_actions.py?from=test_key_actions.py#71

It clicks a link on the test page which has an onclick handler and replaces the content of a div. I do not see how this should open a new tab. 

5. test_shared_all.py TestSharedUnits.test_units

That's a disconnect after 6min. So something in this test seem to hang Marionette. In detail it looks to be in marionette.navigate(). We send the URL but do not receive a reply from the server. I wonder if bug 1207125 could play into here.
Maybe we need loopbackVideo like mochitests?
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml#15

loopbackVideo sets up a /dev/video0 for use testing v4l (video4linux)
It is not needed to set up a window manager.

I will give it a shot.
<dustin> I think only the tests that use video require it
<dustin> unlike pulseaudio, which it seems almost all tests require
<armenzg> dustin, what does "require video" mean in this context?
I think it means stuff that tests video playback
<dustin> but not including regular graphics rendering
I thought looopbackVideo was enabled for all firefox tests.  It can't hurt to try.
Adding loopbackVideo did not change Marionette's results.

Would loopbackAudio be important for it?
There's an easy way to find out :)
I've written some notes on how to run this locally:
https://pastebin.mozilla.org/8859086

I see the Ubuntu extension installation getting on the way:
http://people.mozilla.org/~armenzg/sattap/c93ff1c9.png

The tests run faster on my local host.
I'm testing on try with a bigger instance.

Locally, I can *not* reproduce:
* testing/marionette/client/marionette/tests/unit/test_typing.py

I can reproduce:
* testing/marionette/client/marionette/tests/unit/test_about_pages.py
* testing/marionette/client/marionette/tests/unit/test_key_actions.py
* testing/marionette/client/marionette/tests/unit/test_modal_dialogs.py
* tests/browser/extensions/loop/chrome/content/shared/test/test_shared_all.py
Does Marionette not prevent the installation of system add-ons? Maybe this preference has only been put into the Desktop class but not into that one which unit tests are using?
Flags: needinfo?(mjzffr)
It does not. I tried using ---pref extensions.installDistroAddons:false without success.

Removing the add-on from disk fixes it [1], however, I assume we want to fix this properly.

[1]
> rm -rf /usr/share/mozilla/extensions/\{ec8030f7-c20a-464f-9b0e-13a3a9e97384\}/
Depends on: 1247076
This patch [1] seems to prevent the prompt.
To be fixed under bug 1247076.

[1]
diff --git a/testing/marionette/driver/marionette_driver/geckoinstance.py b/testing/marionette/driver/marionette_driver/geckoinstance.py
--- a/testing/marionette/driver/marionette_driver/geckoinstance.py
+++ b/testing/marionette/driver/marionette_driver/geckoinstance.py
@@ -36,4 +36,6 @@ class GeckoInstance(object):
         "datareporting.policy.dataSubmissionPolicyAccepted": False,
         "dom.ipc.reportProcessHangs": False,
+        "extensions.enabledScopes": 5,
+        "extensions.autoDisableScopes": 10,
         "focusmanager.testmode": True,
         "marionette.defaultPrefs.enabled": True,
The same prefs are in DesktopInstance. If you're adding them to GeckoInstance, they should be removed from DesktopInstance. https://dxr.mozilla.org/mozilla-central/source/testing/marionette/driver/marionette_driver/geckoinstance.py#207-208
Flags: needinfo?(mjzffr)
Fixed in dep bug 1247076.
Assignee: nobody → armenzg
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.