558743 - Mozmill tests should be able to run in the background

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Right now we still suffer from a lot of problems when running Mozmill tests in the background. It would be very helpful if we could get Mozmill to also run tests in the background. In such a case we could execute multiple tes-runs in parallel.

Neil, do we have any dependency with synthesizing events (EventUtils.js) here or do those functions always work if the browser is in the background?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #0)
> Neil, do we have any dependency with synthesizing events (EventUtils.js) here
> or do those functions always work if the browser is in the background?

Neil, it would be better if you could give an answer on bug 558743 instead. Thanks.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #1)
> Neil, it would be better if you could give an answer on bug 558743 instead.
> Thanks.

Which is this one. Sorry, but by adding the dependencies I mixed up the tabs.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For Thunderbird they already using the VNC server on Linux to run tests in the background:

http://mxr.mozilla.org/comm-central/source/mail/test/mozmill/runtest.py#205

That would be a good start.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

On Windows we should check if the tool from Sysinternal would allow us to run the test in a virtual desktop.

http://technet.microsoft.com/en-us/sysinternals/cc817881.aspx

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We are still not able to run Mozmill tests in the background even with the focusmanager testmode in place. Some notifications still don't come up when the browser is in the background or if mozmill gets run via xvfb.

Neil, any idea what we would further need here? Not sure which kind of bug to file for that. :/

Comment 6

[Neil Deakin]

If you provide some steps that fail, I can take a look.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so here the steps to do:

1. Setup xvfb on a Linux system (I have Ubuntu 13.10. 64bit) to use e.g. display 99
2. Get the latest mozmill-env for your platform and unzip: https://mozqa.com/mozmill-env/
3. Clone the mozmill-test repository: http://hg.mozilla.org/qa/mozmill-tests/
4. Run: DISPLAY=0:99 mozmill -b %path_to_firefox% -m mozmill-tests/nightly/firefox/tests/functional/testAwesomeBar/manifest.ini

As failures you will see all of the popup related actions to fail:

TEST-START | testAccessLocationBar.js | testAccessLocationBarHistory
ERROR | Test Failure | {
  "exception": {
    "message": "Autocomplete results should be visible", 
    "lineNumber": 27, 
    "name": "AssertionError", 
    "fileName": "resource://mozmill/modules/errors.js"
  }
}
TEST-UNEXPECTED-FAIL | testAccessLocationBar.js | testAccessLocationBarHistory
TEST-START | testAccessLocationBar.js | teardownModule
TEST-END | testAccessLocationBar.js | finished in 10048ms
TEST-START | testCheckItemHighlight.js | setupModule
TEST-START | testCheckItemHighlight.js | testCheckItemHighlight
ERROR | Test Failure | {
  "exception": {
    "message": "Autocomplete popup has been opened - got 'false', expected 'true'", 
    "lineNumber": 27, 
    "name": "AssertionError", 
    "fileName": "resource://mozmill/modules/errors.js"
  }
}

So as it looks like none of the actions are really triggering the popups to get opened? Is the event lost or not handled correctly? Thanks!

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Neil, have you had a change to look at this?

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Neil, we would kinda appreciate your feedback here. So resetting needinfo flag if it got lost.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.