Closed Bug 1302364 Opened 8 years ago Closed 8 years ago

Add "mach test" support for Firefox UI tests

Categories

(Testing :: Firefox UI Tests, defect)

Product:
Testing
Component:
Firefox UI Tests
Version:
49 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox52 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
firefox52 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Depends on 1 open bug)

Details

Attachments

(2 files, 1 obsolete file)

Bug 1302364 - Update TEST_MANIFESTS documentation for install_root in resolve_tests().
58 bytes, text/x-review-board-request
gps
: review+
Details
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".
58 bytes, text/x-review-board-request
gps
: review+
Details
To allow execution of tests across the tree it would be good to get "mach test" support. Therefor we need specific fx-ui-test entries in context.py and appropriate moz.build files.
Comment on attachment 8790868 [details] Bug 1302364 - Update TEST_MANIFESTS documentation for install_root in resolve_tests(). https://reviewboard.mozilla.org/r/78478/#review77632
Attachment #8790868 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details] Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". https://reviewboard.mozilla.org/r/78480/#review77634
Attachment #8790869 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details] Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". https://reviewboard.mozilla.org/r/78480/#review77636 ::: toolkit/toolkit.mozbuild:164 (Diff revision 1) > > DIRS += ['/addon-sdk'] > > if CONFIG['ENABLE_MARIONETTE'] or CONFIG['MOZ_WIDGET_TOOLKIT'] not in ('gonk', 'android'): > - DIRS += ['/testing/marionette'] > + DIRS += [ > + '/testing/firefox-ui', This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect.
Comment on attachment 8790870 [details] Bug 1302364 - Enable firefox-ui and puppeteer test flavors. https://reviewboard.mozilla.org/r/78482/#review77638
Attachment #8790870 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details] Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". https://reviewboard.mozilla.org/r/78480/#review77636 > This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect. It would mean I have to add an empty moz.build to the second commit, and modify it in commit 3. I think that I will better squash the latter two commits into a single one. Having them separate doesn't make that much sense anyway.
Attachment #8790870 - Attachment is obsolete: true
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/e92048c87179 Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps https://hg.mozilla.org/integration/autoland/rev/1df8bde64853 Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
Backed out for failing test_emitter.py in e.g. spidermonkey tests: https://hg.mozilla.org/integration/autoland/rev/f460b68500462bf7b4d9de3b8ef68470bede8b51 https://hg.mozilla.org/integration/autoland/rev/054e4e24a0c1a90db80c8ba33418bb0ab3be5672 Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1df8bde648531feb661af93efed50487e8a50c5d Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=3609594&repo=autoland 02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_manifest 02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error 02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error_unfiltered 02:24:32 INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_parent_support_files_dir 02:24:32 WARNING - TEST-UNEXPECTED-FAIL | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_shared_support_files, line 474: Items in the second set but not the first: 02:24:32 INFO - FAIL: Support files starting with '!' are given separate treatment, so their 02:24:32 INFO - Traceback (most recent call last): 02:24:32 INFO - File "/builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py", line 474, in test_test_manifest_shared_support_files 02:24:32 INFO - expected_deferred_installs) 02:24:32 INFO - AssertionError: Items in the second set but not the first: 02:24:32 INFO - u'!/child/data/**' 02:24:32 INFO - u'!/child/another-file.sjs' 02:24:32 INFO - u'!/child/test_sub.js'
Flags: needinfo?(hskupin)
I have absolutely no idea why this is happening, but I was able to narrow it down to the entries as added to mozbuild/testing.py: https://hg.mozilla.org/integration/autoland/diff/1df8bde64853/python/mozbuild/mozbuild/testing.py Gregory, do you have any idea what could be the problem here? It doesn't matter what I change, it always fails.
Flags: needinfo?(hskupin) → needinfo?(gps)
Failing test: https://dxr.mozilla.org/mozilla-central/rev/fd0564234eca242b7fb753a110312679020f8059/python/mozbuild/mozbuild/test/frontend/test_emitter.py#459 So I checked what supported and child contain as deferred_installs. Here without my changes: supported: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js']) child: set([]) And with my changes: supported: set([]) child: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js']) So it's vise-versa now. I'm still unclear why that happens due to my changes.
Chris, I have seen that you did a large rewrite of code in that area on bug 1242051. It also added this test. Do you have an idea what specifically could be wrong here?
Flags: needinfo?(gps) → needinfo?(cmanchester)
It looks like that test is relying on an unstable ordering (it's based on the keys of a dict, which I must not have realized when writing that test), so modifying the source dict created a spurious error. Feel free to disable that test for now, and please file a follow up so we can get it fixed.
Flags: needinfo?(cmanchester)
Depends on: 1304316
Pushed by hskupin@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/793cc68cc5a2 Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps https://hg.mozilla.org/integration/autoland/rev/2613526b22e9 Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
https://hg.mozilla.org/mozilla-central/rev/793cc68cc5a2 https://hg.mozilla.org/mozilla-central/rev/2613526b22e9
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: