Closed Bug 1302364 Opened 8 years ago Closed 8 years ago

Add "mach test" support for Firefox UI tests

Categories

(Testing :: Firefox UI Tests, defect)

Product:
Testing
Component:
Firefox UI Tests
Version:
49 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox52 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
firefox52 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

(Depends on 1 open bug)

Details

Attachments

(2 files, 1 obsolete file)

Bug 1302364 - Update TEST_MANIFESTS documentation for install_root in resolve_tests().
58 bytes, text/x-review-board-request
gps
: review+
Details
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".
58 bytes, text/x-review-board-request
gps
: review+
Details 
To allow execution of tests across the tree it would be good to get "mach test" support. Therefor we need specific fx-ui-test entries in context.py and appropriate moz.build files.
Comment on attachment 8790868 [details]
Bug 1302364 - Update TEST_MANIFESTS documentation for install_root in resolve_tests().

https://reviewboard.mozilla.org/r/78478/#review77632
Attachment #8790868 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".

https://reviewboard.mozilla.org/r/78480/#review77634
Attachment #8790869 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".

https://reviewboard.mozilla.org/r/78480/#review77636

::: toolkit/toolkit.mozbuild:164
(Diff revision 1)
>  
>  DIRS += ['/addon-sdk']
>  
>  if CONFIG['ENABLE_MARIONETTE'] or CONFIG['MOZ_WIDGET_TOOLKIT'] not in ('gonk', 'android'):
> -    DIRS += ['/testing/marionette']
> +    DIRS += [
> +        '/testing/firefox-ui',

This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect.
Comment on attachment 8790870 [details]
Bug 1302364 - Enable firefox-ui and puppeteer test flavors.

https://reviewboard.mozilla.org/r/78482/#review77638
Attachment #8790870 - Flags: review?(gps) → review+
Comment on attachment 8790869 [details]
Bug 1302364 - Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test".

https://reviewboard.mozilla.org/r/78480/#review77636

> This will break moz.build processing because testing/firefox-ui/moz.build doesn't exist until the next commit. Ideally this addition would be deferred until the next commit so you don't break bisect.

It would mean I have to add an empty moz.build to the second commit, and modify it in commit 3. I think that I will better squash the latter two commits into a single one. Having them separate doesn't make that much sense anyway.
Attachment #8790870 - Attachment is obsolete: true
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e92048c87179
Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps
https://hg.mozilla.org/integration/autoland/rev/1df8bde64853
Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
Backed out for failing test_emitter.py in e.g. spidermonkey tests:

https://hg.mozilla.org/integration/autoland/rev/f460b68500462bf7b4d9de3b8ef68470bede8b51
https://hg.mozilla.org/integration/autoland/rev/054e4e24a0c1a90db80c8ba33418bb0ab3be5672

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1df8bde648531feb661af93efed50487e8a50c5d
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=3609594&repo=autoland

02:24:32     INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_manifest
02:24:32     INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error
02:24:32     INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_missing_test_error_unfiltered
02:24:32     INFO - TEST-PASS | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_parent_support_files_dir
02:24:32  WARNING - TEST-UNEXPECTED-FAIL | /builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py | TestEmitterBasic.test_test_manifest_shared_support_files, line 474: Items in the second set but not the first:
02:24:32     INFO - FAIL: Support files starting with '!' are given separate treatment, so their
02:24:32     INFO - Traceback (most recent call last):
02:24:32     INFO -   File "/builds/slave/autoland-m64-d-000000000000000/build/src/python/mozbuild/mozbuild/test/frontend/test_emitter.py", line 474, in test_test_manifest_shared_support_files
02:24:32     INFO -     expected_deferred_installs)
02:24:32     INFO - AssertionError: Items in the second set but not the first:
02:24:32     INFO - u'!/child/data/**'
02:24:32     INFO - u'!/child/another-file.sjs'
02:24:32     INFO - u'!/child/test_sub.js'
Flags: needinfo?(hskupin)
I have absolutely no idea why this is happening, but I was able to narrow it down to the entries as added to mozbuild/testing.py:

https://hg.mozilla.org/integration/autoland/diff/1df8bde64853/python/mozbuild/mozbuild/testing.py

Gregory, do you have any idea what could be the problem here? It doesn't matter what I change, it always fails.
Flags: needinfo?(hskupin) → needinfo?(gps)
Failing test:

https://dxr.mozilla.org/mozilla-central/rev/fd0564234eca242b7fb753a110312679020f8059/python/mozbuild/mozbuild/test/frontend/test_emitter.py#459

So I checked what supported and child contain as deferred_installs.

Here without my changes:
supported: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js'])
child: set([])

And with my changes:
supported: set([])
child: set(['!/child/data/**', '!/child/another-file.sjs', '!/child/test_sub.js'])

So it's vise-versa now. I'm still unclear why that happens due to my changes.
Chris, I have seen that you did a large rewrite of code in that area on bug 1242051. It also added this test. Do you have an idea what specifically could be wrong here?
Flags: needinfo?(gps) → needinfo?(cmanchester)
It looks like that test is relying on an unstable ordering (it's based on the keys of a dict, which I must not have realized when writing that test), so modifying the source dict created a spurious error.

Feel free to disable that test for now, and please file a follow up so we can get it fixed.
Flags: needinfo?(cmanchester)
Depends on: 1304316
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/793cc68cc5a2
Update TEST_MANIFESTS documentation for install_root in resolve_tests(). r=gps
https://hg.mozilla.org/integration/autoland/rev/2613526b22e9
Include firefox-ui and puppeteer in all-tests.json, and allow to run tests via "mach test". r=gps
https://hg.mozilla.org/mozilla-central/rev/793cc68cc5a2
https://hg.mozilla.org/mozilla-central/rev/2613526b22e9
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
status-firefox52: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: