Closed
Bug 1262158
Opened 8 years ago
Closed 8 years ago
Problems running Firefox UI tests according to documentation
Categories
(Testing :: Firefox UI Tests, defect)
Tracking
(firefox47 fixed, firefox48 fixed)
RESOLVED
FIXED
mozilla48
People
(Reporter: davehunt, Assigned: impossibus)
Details
Attachments
(2 files)
MozReview Request: Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r?sydpolk
58 bytes,
text/x-review-board-request
|
sydpolk
:
review+
|
Details |
58 bytes,
text/x-review-board-request
|
automatedtester
:
review+
|
Details |
From: https://developer.mozilla.org/en-US/docs/Mozilla/QA/firefox-ui-tests > To run the tests, first build Mozilla with your changes; then run > ./mach firefox-ui-test > This will launch your build, execute all existing tests, and report the results to stdout. However, the firefox-ui-tests command is not found, and instead the firefox-ui-update command is executed. Running ./mach help shows that there's firefox-ui-functional and firefox-ui-update commands. Running either of these does not run all existing tests, but instead reports: > must specify one or more test files, manifests, or directories Running with a directory specified is failing with the following: > $ ./mach firefox-ui-functional testing/firefox-ui/tests/firefox_ui_tests/functional/security/ > 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/dhunt/workspace/firefox" > 1:00.04 LOG: MainThread ERROR Failure during test execution. > Traceback (most recent call last): > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run > runner.run_tests(tests) > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 841, in run_tests > self.start_marionette() > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 786, in start_marionette > self.marionette = self.driverclass(**self._build_kwargs()) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 602, in __init__ > self.raise_for_port(self.wait_for_port(timeout=startup_timeout)) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/decorators.py", line 36, in _ > return func(*args, **kwargs) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 689, in raise_for_port > raise IOError("Timed out waiting for port!") > IOError: Timed out waiting for port! > 1:00.04 LOG: MainThread ERROR Failure during harness setup > Traceback (most recent call last): > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 89, in cli > failed = harness_class(runner_class, parser_class, args=args).run() > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run > runner.run_tests(tests) > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 841, in run_tests > self.start_marionette() > File "/Users/dhunt/workspace/firefox/testing/marionette/harness/marionette/runner/base.py", line 786, in start_marionette > self.marionette = self.driverclass(**self._build_kwargs()) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 602, in __init__ > self.raise_for_port(self.wait_for_port(timeout=startup_timeout)) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/decorators.py", line 36, in _ > return func(*args, **kwargs) > File "/Users/dhunt/workspace/firefox/testing/marionette/client/marionette_driver/marionette.py", line 689, in raise_for_port > raise IOError("Timed out waiting for port!") > IOError: Timed out waiting for port! > Could not kill process, could not find pid: 71512, assuming it's already dead
Assignee | ||
Comment 1•8 years ago
|
||
I'm also failing to pass in a directory of tests. In my case Firefox starts up, but I get:
> ./mach firefox-ui-functional testing/firefox-ui/tests/firefox_ui_tests/functional/security/
> 0:00.00 LOG: MainThread INFO Using workspace for temporary data: "/Users/mozilla/dev/firefox"
> 0:02.82 LOG: MainThread INFO Initial Profile Destination is "/var/folders/5k/xmn_fndx0qs2jcpcwhzl86wm0000gn/T/tmph8o8T3.mozrunner"
> 0:02.82 LOG: MainThread INFO starting httpd
> 0:02.84 LOG: MainThread INFO running httpd on http://127.0.0.1:63917/
> 0:02.84 LOG: MainThread ERROR Failure during test execution.
>Traceback (most recent call last):
> File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
> runner.run_tests(tests)
> File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runner/base.py", line 881, in run_tests
> % '\n '.join(invalid_tests))
>Exception: Tests file names must starts with 'test_'. Invalid test names:
> /Users/mozilla/dev/firefox/testing/firefox-ui/tests/firefox_ui_tests/functional/security
> 0:02.84 LOG: MainThread ERROR Failure during harness setup
>Traceback (most recent call last):
> File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 89, in cli
> failed = harness_class(runner_class, parser_class, args=args).run()
> File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runtests.py", line 67, in run
> runner.run_tests(tests)
> File "/Users/mozilla/dev/firefox/testing/marionette/harness/marionette/runner/base.py", line 881, in run_tests
> % '\n '.join(invalid_tests))
> Exception: Tests file names must starts with 'test_'. Invalid test names:
> /Users/mozilla/dev/firefox/testing/firefox-ui/tests/firefox_ui_tests/functional/security
Reporter | ||
Comment 2•8 years ago
|
||
You can ignore the last part of comment 0 concerning "IOError: Timed out waiting for port!" as this was due to a bad local build of Firefox.
Assignee | ||
Comment 3•8 years ago
|
||
Took another quick look at this. The path mentioned in comment 0 and comment 1 (testing/firefox-ui/tests/firefox_ui_tests/*) doesn't exist; it's now testing/firefox-ui/tests/functional/security and running firefox-ui-functional with that works fine. Nevertheless it would be nice for Marionette runner to give a clearer error message when a directory doesn't exist.
Assignee | ||
Comment 4•8 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/46553/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/46553/
Attachment #8741536 -
Flags: review?(spolk)
Attachment #8741537 -
Flags: review?(dburns)
Assignee | ||
Comment 5•8 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/46555/diff/#index_header See other reviews: https://reviewboard.mozilla.org/r/46555/
Assignee | ||
Updated•8 years ago
|
Assignee: nobody → mjzffr
Status: NEW → ASSIGNED
Updated•8 years ago
|
Attachment #8741536 -
Flags: review?(spolk) → review+
Comment 6•8 years ago
|
||
Comment on attachment 8741536 [details] MozReview Request: Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r?sydpolk https://reviewboard.mozilla.org/r/46553/#review43267 I hate the way args handling is done in mozharness. But this looks right.
Updated•8 years ago
|
Attachment #8741537 -
Flags: review?(dburns) → review+
Comment 7•8 years ago
|
||
Comment on attachment 8741537 [details] MozReview Request: Bug 1262158 - Fail early if test path does not exist; r?AutomatedTester https://reviewboard.mozilla.org/r/46555/#review43653
https://hg.mozilla.org/integration/mozilla-inbound/rev/865e124224cd https://hg.mozilla.org/integration/mozilla-inbound/rev/f5f9a967030d
Comment 9•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/865e124224cd https://hg.mozilla.org/mozilla-central/rev/f5f9a967030d
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox48:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
Comment 10•8 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-aurora/rev/0ae4de45a952
status-firefox47:
--- → fixed
Comment 11•8 years ago
|
||
Maja, would the fx ui tests part also need to be uplifted? David only did this for the Marionette code changes.
Flags: needinfo?(mjzffr)
Comment 12•8 years ago
|
||
Btw. I have updated the MDN documentation for our tests so that the correct mach command and folders are listed. Thanks for noticing Dave!
Assignee | ||
Comment 13•8 years ago
|
||
If you want the mach command to use the default test manifest correctly in aurora, then yes, that needs to be uplifted, too.
Flags: needinfo?(mjzffr)
Comment 14•8 years ago
|
||
Only one patch got backported to 47.0. I would also like to see attachment 8741536 [details] landed on beta. Thanks.
Whiteboard: [checkin-needed-beta]
Comment 15•8 years ago
|
||
whimboo, this has conflicts in grafting 340390:865e124224cd "Bug 1262158 - Fix default tests being overwritten in mach firefox-ui-_; r=sydpolk" merging testing/firefox-ui/mach_commands.py warning: conflicts while merging testing/firefox-ui/mach_commands.py! when uplifting the other patch to beta, could you take a look, thanks!
Flags: needinfo?(hskupin)
Comment 16•8 years ago
|
||
In that case I would say we don't spend time in getting the patch backported. At least not as long no-one complains about broken functionality. Lets decide again if necessary. Thanks for pointing it out.
Flags: needinfo?(hskupin)
Whiteboard: [checkin-needed-beta]
You need to log in
before you can comment on or make changes to this bug.
Description
•