Closed
Bug 1302364
Opened 8 years ago
Closed 8 years ago
Add "mach test" support for Firefox UI tests
Categories
(Testing :: Firefox UI Tests, defect)
Tracking
(firefox52 fixed)
RESOLVED
FIXED
mozilla52
Tracking | Status | |
---|---|---|
firefox52 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Depends on 1 open bug)
Details
Attachments
(2 files, 1 obsolete file)
To allow execution of tests across the tree it would be good to get "mach test" support. Therefor we need specific fx-ui-test entries in context.py and appropriate moz.build files.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 4•8 years ago
|
||
mozreview-review |
Comment on attachment 8790868 [details]
Bug 1302364 - Update TEST_MANIFESTS documentation for install_root in resolve_tests().
https://reviewboard.mozilla.org/r/78478/#review77632
Attachment #8790868 -
Flags: review?(gps) → review+
Comment 5•8 years ago
|
||
mozreview-review |
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".
https://reviewboard.mozilla.org/r/78480/#review77634
Attachment #8790869 -
Flags: review?(gps) → review+
Comment 6•8 years ago
|
||
mozreview-review |
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".
https://reviewboard.mozilla.org/r/78480/#review77636
::: toolkit/toolkit.mozbuild:164
(Diff revision 1)
>
> DIRS += ['/addon-sdk']
>
> if CONFIG['ENABLE_MARIONETTE'] or CONFIG['MOZ_WIDGET_TOOLKIT'] not in ('gonk', 'android'):
> - DIRS += ['/testing/marionette']
> + DIRS += [
> + '/testing/firefox-ui',
This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect.
Comment 7•8 years ago
|
||
mozreview-review |
Comment on attachment 8790870 [details]
Bug 1302364 - Enable firefox-ui and puppeteer test flavors.
https://reviewboard.mozilla.org/r/78482/#review77638
Attachment #8790870 -
Flags: review?(gps) → review+
Assignee | ||
Comment 8•8 years ago
|
||
mozreview-review-reply |
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".
https://reviewboard.mozilla.org/r/78480/#review77636
> This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect.
It would mean I have to add an empty moz.build to the second commit, and modify it in commit 3. I think that I will better squash the latter two commits into a single one. Having them separate doesn't make that much sense anyway.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Assignee | ||
Updated•8 years ago
|
Attachment #8790870 -
Attachment is obsolete: true
Comment 11•8 years ago
|
||
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e92048c87179
Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps
https://hg.mozilla.org/integration/autoland/rev/1df8bde64853
Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
Comment 12•8 years ago
|
||
Backed out for failing test_emitter.py in e.g. spidermonkey tests:
https://hg.mozilla.org/integration/autoland/rev/f460b68500462bf7b4d9de3b8ef68470bede8b51
https://hg.mozilla.org/integration/autoland/rev/054e4e24a0c1a90db80c8ba33418bb0ab3be5672
Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1df8bde648531feb661af93efed50487e8a50c5d
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=3609594&repo=autoland
02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_manifest
02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error
02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error_unfiltered
02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_parent_support_files_dir
02:24:32 WARNING - TEST-UNEXPECTED-FAIL | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_shared_support_files, line 474: Items in the second set but not the first:
02:24:32 INFO - FAIL: Support files starting with '!' are given separate treatment, so their
02:24:32 INFO - Traceback (most recent call last):
02:24:32 INFO - File "/builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py", line 474, in test_test_manifest_shared_support_files
02:24:32 INFO - expected_deferred_installs)
02:24:32 INFO - AssertionError: Items in the second set but not the first:
02:24:32 INFO - u'!/child/data/**'
02:24:32 INFO - u'!/child/another-file.sjs'
02:24:32 INFO - u'!/child/test_sub.js'
Flags: needinfo?(hskupin)
Assignee | ||
Comment 13•8 years ago
|
||
I have absolutely no idea why this is happening, but I was able to narrow it down to the entries as added to mozbuild/testing.py:
https://hg.mozilla.org/integration/autoland/diff/1df8bde64853/python/mozbuild/mozbuild/testing.py
Gregory, do you have any idea what could be the problem here? It doesn't matter what I change, it always fails.
Flags: needinfo?(hskupin) → needinfo?(gps)
Assignee | ||
Comment 14•8 years ago
|
||
Failing test:
https://dxr.mozilla.org/mozilla-central/rev/fd0564234eca242b7fb753a110312679020f8059/python/mozbuild/mozbuild/test/frontend/test_emitter.py#459
So I checked what supported and child contain as deferred_installs.
Here without my changes:
supported: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js'])
child: set([])
And with my changes:
supported: set([])
child: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js'])
So it's vise-versa now. I'm still unclear why that happens due to my changes.
Assignee | ||
Comment 15•8 years ago
|
||
Chris, I have seen that you did a large rewrite of code in that area on bug 1242051. It also added this test. Do you have an idea what specifically could be wrong here?
Flags: needinfo?(gps) → needinfo?(cmanchester)
Comment 16•8 years ago
|
||
It looks like that test is relying on an unstable ordering (it's based on the keys of a dict, which I must not have realized when writing that test), so modifying the source dict created a spurious error.
Feel free to disable that test for now, and please file a follow up so we can get it fixed.
Flags: needinfo?(cmanchester)
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 19•8 years ago
|
||
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/793cc68cc5a2
Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps
https://hg.mozilla.org/integration/autoland/rev/2613526b22e9
Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
Comment 20•8 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/793cc68cc5a2
https://hg.mozilla.org/mozilla-central/rev/2613526b22e9
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox52:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in
before you can comment on or make changes to this bug.
Description
•