1262158 - Problems running Firefox UI tests according to documentation

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

From: https://developer.mozilla.org/en-US/docs/Mozilla/QA/firefox-ui-tests

> To run the tests, first build Mozilla with your changes; then run
> ./mach firefox-ui-test
> This will launch your build, execute all existing tests, and report the results to stdout.

However, the firefox-ui-tests command is not found, and instead the firefox-ui-update command is executed.

Running ./mach help shows that there's firefox-ui-functional and firefox-ui-update commands. Running either of these does not run all existing tests, but instead reports:

> must specify one or more test files, manifests, or directories

Running with a directory specified is failing with the following:

> $ ./mach firefox-ui-functional testing/firefox-ui/tests/firefox_ui_tests/functional/security/
>  0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/dhunt/workspace/firefox"
>  1:00.04 LOG: MainThread ERROR Failure during test execution.
> Traceback (most recent call last):
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
>     runner.run_tests(tests)
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 841, in run_tests
>     self.start_marionette()
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 786, in start_marionette
>     self.marionette = self.driverclass(**self._build_kwargs())
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 602, in __init__
>     self.raise_for_port(self.wait_for_port(timeout=startup_timeout))
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/decorators.py", line 36, in _
>     return func(*args, **kwargs)
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 689, in raise_for_port
>     raise IOError("Timed out waiting for port!")
> IOError: Timed out waiting for port!
>  1:00.04 LOG: MainThread ERROR Failure during harness setup
> Traceback (most recent call last):
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 89, in cli
>     failed = harness_class(runner_class, parser_class, args=args).run()
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
>     runner.run_tests(tests)
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 841, in run_tests
>     self.start_marionette()
>   File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 786, in start_marionette
>     self.marionette = self.driverclass(**self._build_kwargs())
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 602, in __init__
>     self.raise_for_port(self.wait_for_port(timeout=startup_timeout))
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/decorators.py", line 36, in _
>     return func(*args, **kwargs)
>   File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 689, in raise_for_port
>     raise IOError("Timed out waiting for port!")
> IOError: Timed out waiting for port!
> Could not kill process, could not find pid: 71512, assuming it's already dead

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

I'm also failing to pass in a directory of tests. In my case Firefox starts up, but I get:

> ./mach firefox-ui-functional testing/firefox-ui/tests/firefox_ui_tests/functional/security/
> 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/mozilla/dev/firefox"
> 0:02.82 LOG: MainThread INFO Initial Profile Destination is "/var/folders/5k/xmn_fndx0qs2jcpcwhzl86wm0000gn/T/tmph8o8T3.mozrunner"
> 0:02.82 LOG: MainThread INFO starting httpd
> 0:02.84 LOG: MainThread INFO running httpd on http://127.0.0.1:63917/
> 0:02.84 LOG: MainThread ERROR Failure during test execution.
>Traceback (most recent call last):
>  File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
>    runner.run_tests(tests)
>  File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runner/base.py", line 881, in run_tests
>    % '\n  '.join(invalid_tests))
>Exception: Tests file names must starts with 'test_'. Invalid test names:
>  /Users/mozilla/dev/firefox/testing/firefox-ui/tests/firefox_ui_tests/functional/security
> 0:02.84 LOG: MainThread ERROR Failure during harness setup
>Traceback (most recent call last):
>  File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 89, in cli
>    failed = harness_class(runner_class, parser_class, args=args).run()
>  File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
>    runner.run_tests(tests)
>  File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runner/base.py", line 881, in run_tests
>    % '\n  '.join(invalid_tests))
> Exception: Tests file names must starts with 'test_'. Invalid test names:
>  /Users/mozilla/dev/firefox/testing/firefox-ui/tests/firefox_ui_tests/functional/security

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

You can ignore the last part of comment 0 concerning "IOError: Timed out waiting for port!" as this was due to a bad local build of Firefox.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Took another quick look at this. 
The path mentioned in comment 0 and comment 1 (testing/firefox-ui/tests/firefox_ui_tests/*) doesn't exist; it's now testing/firefox-ui/tests/functional/security and running firefox-ui-functional with that works fine. Nevertheless it would be nice for Marionette runner to give a clearer error message when a directory doesn't exist.

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r?sydpolk

Review commit: https://reviewboard.mozilla.org/r/46553/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/46553/

Comment 5

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: MozReview Request: Bug 1262158 - Fail early if test path does not exist; r?AutomatedTester

Review commit: https://reviewboard.mozilla.org/r/46555/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/46555/

Comment 6

[Syd Polk :sydpolk]

Comment on attachment 8741536 [details]
MozReview Request: Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r?sydpolk

https://reviewboard.mozilla.org/r/46553/#review43267

I hate the way args handling is done in mozharness. But this looks right.

Comment 7

[David Burns :automatedtester]

Comment on attachment 8741537 [details]
MozReview Request: Bug 1262158 - Fail early if test path does not exist; r?AutomatedTester

https://reviewboard.mozilla.org/r/46555/#review43653

Comment 8

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/865e124224cd
https://hg.mozilla.org/integration/mozilla-inbound/rev/f5f9a967030d

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/865e124224cd
https://hg.mozilla.org/mozilla-central/rev/f5f9a967030d

Comment 10

[David Burns :automatedtester]

https://hg.mozilla.org/releases/mozilla-aurora/rev/0ae4de45a952

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Maja, would the fx ui tests part also need to be uplifted? David only did this for the Marionette code changes.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Btw. I have updated the MDN documentation for our tests so that the correct mach command and folders are listed. Thanks for noticing Dave!

Comment 13

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

If you want the mach command to use the default test manifest correctly in aurora, then yes, that needs to be uplifted, too.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Only one patch got backported to 47.0. I would also like to see attachment 8741536 [details] landed on beta. Thanks.

Comment 15

[Carsten Book [:Tomcat]]

whimboo, this has conflicts in grafting 340390:865e124224cd "Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r=sydpolk"
merging testing/firefox-ui/mach_commands.py
warning: conflicts while merging testing/firefox-ui/mach_commands.py! 

when uplifting the other patch to beta, could you take a look, thanks!

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

In that case I would say we don't spend time in getting the patch backported. At least not as long no-one complains about broken functionality. Lets decide again if necessary. Thanks for pointing it out.