1246283 - many failures while running marionette jobs on taskcluster/docker instead of buildbot

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

TEST-UNEXPECTED-FAIL | test_typing.py TestTyping.testCutAndPasteShortcuts | AssertionError: u'http://127.0.0.1:51119/javascriptPage.html' != ''
TEST-UNEXPECTED-ERROR | test_about_pages.py TestAboutPages.test_navigate_shortcut_key | TimeoutException: TimeoutException: wait_for_condition timed out
TEST-UNEXPECTED-ERROR | test_modal_dialogs.py TestGlobalModals.test_alert_accept | TimeoutException: TimeoutException: Timed out after 5.1 seconds
TEST-UNEXPECTED-ERROR | test_modal_dialogs.py TestTabModals.test_alert_accept | TimeoutException: TimeoutException: Timed out after 5.1 seconds
TEST-UNEXPECTED-ERROR | test_key_actions.py TestKeyActions.test_open_in_new_window_shortcut | TimeoutException: TimeoutException: wait_for_condition timed out
TEST-UNEXPECTED-ERROR | test_shared_all.py TestSharedUnits.test_units | timeout: connection timed out after 360s 


you can see this is a common trend:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&selectedJob=16412573&filter-searchStr=Mn

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:ato, can you help us figure out what is going on and how to work towards getting this green?  I am unfamiliar with the mn job/tests/harness- so this might be a simple thing to figure out.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Are those tests running headless? Do we have a window manager? This is actually something we have problems with when running our tests on Travis. So Andreas gave us this nice advice: bug 1240179 comment 5.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

there is a window manager- the same one that runs reftest, mochitest, and web-platform-test.

the steps taken in the comment are similar to what we do to setup the docker image.

Comment 4

[Andreas Tolfsen ❲:ato❳]

Aside from whimboo’s advice above, I can’t think of anything.  Do you have reproduction steps?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Here just my 2c while quickly observing the failures and checking the test content:

1. The failure in TestTyping.testCutAndPasteShortcuts we expect the urlbar to be empty after Meta+A + Meta+Backspace:

https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_typing.py?from=test_typing.py#82

As it looks like the backspace key was not sent correctly. Maybe even together with 'a' but that's hard to tell here. Keep in mind that we are in chrome context here. I could assume that the urlbar does not have focus yet. So a click() on it before may fix it.

2. test_about_pages.py
https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_about_pages.py?from=test_about_pages.py+#

Those lines show me that here we are trying to open the add-on manager via meta+shift+a. Keep in mind that this shortcut is known to be broken on Linux if any text field has the focus. I'm not sure why this is part of this test actually. It has nothing to do with about pages.

3. TestGlobalModals.test_alert_accept

Not sure whats up with that. As long as the loaded HTML page is valid I can only assume that the click was not propagated correctly. That should not require a window manager. I'm not sure if Marionette server actually checks if actions were executed correctly.

4. test_key_actions.py TestKeyActions.test_open_in_new_window_shortcut

I do not understand what this test is trying to do.

https://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_key_actions.py?from=test_key_actions.py#71

It clicks a link on the test page which has an onclick handler and replaces the content of a div. I do not see how this should open a new tab. 

5. test_shared_all.py TestSharedUnits.test_units

That's a disconnect after 6min. So something in this test seem to hang Marionette. In detail it looks to be in marionette.navigate(). We send the URL but do not receive a reply from the server. I wonder if bug 1207125 could play into here.

Comment 6

[Armen [:armenzg]]

Maybe we need loopbackVideo like mochitests?
https://dxr.mozilla.org/mozilla-central/source/testing/taskcluster/tasks/tests/fx_linux64_mochitest_plain.yml#15

loopbackVideo sets up a /dev/video0 for use testing v4l (video4linux)
It is not needed to set up a window manager.

I will give it a shot.

Comment 7

[Armen [:armenzg]]

<dustin> I think only the tests that use video require it
<dustin> unlike pulseaudio, which it seems almost all tests require
<armenzg> dustin, what does "require video" mean in this context?
I think it means stuff that tests video playback
<dustin> but not including regular graphics rendering

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I thought looopbackVideo was enabled for all firefox tests.  It can't hurt to try.

Comment 9

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bdaf24d820c8

Comment 10

[Armen [:armenzg]]

Adding loopbackVideo did not change Marionette's results.

Would loopbackAudio be important for it?

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

There's an easy way to find out :)

Comment 12

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=046770dd9ed7

Comment 13

[Armen [:armenzg]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=4093c1b05664

Comment 14

[Armen [:armenzg]]

I've written some notes on how to run this locally:
https://pastebin.mozilla.org/8859086

I see the Ubuntu extension installation getting on the way:
http://people.mozilla.org/~armenzg/sattap/c93ff1c9.png

The tests run faster on my local host.
I'm testing on try with a bigger instance.

Locally, I can *not* reproduce:
* testing/marionette/client/marionette/tests/unit/test_typing.py

I can reproduce:
* testing/marionette/client/marionette/tests/unit/test_about_pages.py
* testing/marionette/client/marionette/tests/unit/test_key_actions.py
* testing/marionette/client/marionette/tests/unit/test_modal_dialogs.py
* tests/browser/extensions/loop/chrome/content/shared/test/test_shared_all.py

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Does Marionette not prevent the installation of system add-ons? Maybe this preference has only been put into the Desktop class but not into that one which unit tests are using?

Comment 16

[Armen [:armenzg]]

It does not. I tried using ---pref extensions.installDistroAddons:false without success.

Removing the add-on from disk fixes it [1], however, I assume we want to fix this properly.

[1]
> rm -rf /usr/share/mozilla/extensions/\{ec8030f7-c20a-464f-9b0e-13a3a9e97384\}/

Comment 17

[Armen [:armenzg]]

This patch [1] seems to prevent the prompt.
To be fixed under bug 1247076.

[1]
diff --git a/testing/marionette/driver/marionette_driver/geckoinstance.py b/testing/marionette/driver/marionette_driver/geckoinstance.py
--- a/testing/marionette/driver/marionette_driver/geckoinstance.py
+++ b/testing/marionette/driver/marionette_driver/geckoinstance.py
@@ -36,4 +36,6 @@ class GeckoInstance(object):
         "datareporting.policy.dataSubmissionPolicyAccepted": False,
         "dom.ipc.reportProcessHangs": False,
+        "extensions.enabledScopes": 5,
+        "extensions.autoDisableScopes": 10,
         "focusmanager.testmode": True,
         "marionette.defaultPrefs.enabled": True,

Comment 18

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

The same prefs are in DesktopInstance. If you're adding them to GeckoInstance, they should be removed from DesktopInstance. https://dxr.mozilla.org/mozilla-central/source/testing/marionette/driver/marionette_driver/geckoinstance.py#207-208

Comment 19

[Armen [:armenzg]]

Fixed in dep bug 1247076.