Closed Bug 1668458 Opened 4 years ago Closed 4 years ago

Enable specialPowers for gecko-only tests

Categories

(Testing :: web-platform-tests, task)

Product:
Testing
Component:
web-platform-tests
Version:
Default
Type:
task

Tracking

(firefox83 fixed)

Status:
RESOLVED FIXED
Milestone:
83 Branch
Tracking Flags:
Tracking Status
firefox83 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

Details

Attachments

(2 files)

Bug 1668458 - Load specialPowers web extension in gecko-only wpt tests,
47 bytes, text/x-phabricator-request
Details | Review
Bug 1668458 - Ensure paths passed to Addons.install use os path separators only,
47 bytes, text/x-phabricator-request
Details | Review

No support for specialPowers is often cited as a reason for developers to write mochitests rather than wpt. Although using specialPowers means that the tests can't be upstreamed, and so should be an option of last-resort for things that can't be done in a cross-browser way, having it available in gecko-only tests should reduce the need to write mochitests and potentially means that a patch which previously would be written with just mochitests would instead be written with just wpt, a fraction of which would be unsuitable for sharing.

Lack of support for specialPowers is often cited as a reason for
developers to write mochitests rather than wpt. Although using
specialPowers means that the tests can't be upstreamed, having it
available in gecko-only tests should reduce the need to write
mochitests and potentially means that a patch which previously would
be written with just mochitests would instead be written with just
wpt, a fraction of which would be unsuitable for sharing.

Nevertheless a gecko-only specialPowers-using test should be
considered the option of last resort when cross-browser APIs such as
testdriver aren't sufficient.

This patch installs the specialPowers extension when the path to the
XPI is provided via a --specialpowers-path command line argument. To
ensure this only happens for gecko-only tests, the url_base is added
to the Test object, and the firefox settings() method is updated to
contain a specialpowers setting, which uses the url_base of the test
to determine if this is a gecko-only test.

Because the extension has to be enabled using marionette, we also need
to route this setting through the to executor, along with the provided
XPI path.

Assignee: nobody → james
Status: NEW → ASSIGNED

On Windows we can end up with a mixture of / and \ which makes the install fail.

Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/e45a73a47c37
Load specialPowers web extension in gecko-only wpt tests, r=jmaher,twisniewski
https://hg.mozilla.org/integration/autoland/rev/8739d75e13dc
Ensure paths passed to Addons.install use os path separators only, r=marionette-reviewers,whimboo
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/25982 for changes under testing/web-platform/tests
Upstream PR was closed without merging
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/5f2252c9e774
Load specialPowers web extension in gecko-only wpt tests, r=jmaher,twisniewski,whimboo

Backed out changeset 5f2252c9e774 (bug 1668458) for wpt failures.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=FbaWtX_hTFazIbAwdmxAvQ.0&searchStr=web%2Cplatform%2Ctests&fromchange=0395e722c91cbd364a62c13ee765ed1afda288b8&tochange=6ecde38faab9dce353aa7652ed4381f6be4cce89

Backout link: https://hg.mozilla.org/integration/autoland/rev/6ecde38faab9dce353aa7652ed4381f6be4cce89

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=317750357&repo=autoland&lineNumber=1397

...
[task 2020-10-06T13:13:00.872Z] 13:13:00     INFO -  'WORKING_DIR': '/builds/worker'}
[task 2020-10-06T13:13:00.872Z] 13:13:00     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/web-platform/runtests.py', '--log-raw=-', '--log-raw=/builds/worker/workspace/build/blobber_upload_dir/wpt_raw.log', '--log-wptreport=/builds/worker/workspace/build/blobber_upload_dir/wptreport.json', '--log-errorsummary=/builds/worker/workspace/build/blobber_upload_dir/wpt_errorsummary.log', '--binary=/builds/worker/workspace/build/application/firefox/firefox', '--symbols-path=None', '--stackwalk-binary=/builds/worker/fetches/minidump_stackwalk/minidump_stackwalk', '--stackfix-dir=/builds/worker/workspace/build/tests/bin', '--no-pause-after-test', '--instrument-to-file=/builds/worker/workspace/build/blobber_upload_dir/wpt_instruments.txt', '--specialpowers-path=/builds/worker/workspace/build/tests/extensions/specialpowers@mozilla.org.xpi', '--processes=2', '--install-fonts', '--test-type=reftest', '--setpref=media.peerconnection.mtransport_process=false', '--setpref=network.process.enabled=false', '--skip-timeout', '--skip-implementation-status=backlog', '--skip-implementation-status=not-implementing', '--test-groups=/builds/worker/fetches/wpt_tests_by_group.json', '--prefs-root=/builds/worker/workspace/build/tests/web-platform/prefs', '--config=/builds/worker/workspace/build/tests/web-platform/wptrunner.ini', '--ca-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/cacert.pem', '--host-key-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.key', '--host-cert-path=/builds/worker/workspace/build/tests/web-platform/tests/tools/certs/web-platform.test.pem', '--certutil-binary=/builds/worker/workspace/build/tests/bin/certutil', u'/css/css-values', u'/css/vendor-imports/mozilla', u'/html/rendering/non-replaced-elements', u'/css/css-scoping', u'/css/compositing/background-blending', u'/css/CSS2/syntax', u'/html/canvas/element', u'/shadow-dom', u'/portals', u'/css/cssom-view', u'/css/css-break', u'/mathml/presentation-markup/radicals', u'/css/css-transforms/group', u'/css/CSS2/visufx', u'/css/css-transforms/skewX', u'/html/the-xhtml-syntax/parsing-xhtml-documents', u'/css/css-flexbox/flex-lines', u'/svg/painting', u'/css/css-shapes/shape-outside', u'/css/css-text/word-break', u'/css/CSS2/backgrounds', u'/css/css-align/gaps', u'/css/css-text/text-align', u'/css/css-tables/height-distribution', u'/css/css-text/letter-spacing', u'/css/css-content', u'/web-animations/timing-model/animations', u'/css/css-masking/mask-svg-content', u'/css/css-fonts/variations', u'/css/css-grid/layout-algorithm', u'/svg/rendering/order', u'/html/semantics/embedded-content', u'/css/CSS2/normal-flow', u'/css/CSS2/colors', u'/css/css-lists/content-property', u'/css/CSS2/fonts', u'/acid/acid3'] with output_timeout 1000
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_buildid: 20201006125744
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_changeset: 5f2252c9e774825624ccb7d0aad59e338c7269b4
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_display_name: Nightly
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_name: Firefox
[task 2020-10-06T13:13:01.349Z] 13:13:01     INFO - mozversion application_remotingname: firefox
[task 2020-10-06T13:13:01.350Z] 13:13:01     INFO - mozversion application_vendor: Mozilla
[task 2020-10-06T13:13:01.350Z] 13:13:01     INFO - mozversion application_version: 83.0a1
[task 2020-10-06T13:13:01.350Z] 13:13:01     INFO - mozversion platform_buildid: 20201006125744
[task 2020-10-06T13:13:01.350Z] 13:13:01     INFO - mozversion platform_changeset: 5f2252c9e774825624ccb7d0aad59e338c7269b4
[task 2020-10-06T13:13:01.350Z] 13:13:01     INFO - mozversion platform_version: 83.0a1
[task 2020-10-06T13:13:08.188Z] 13:13:08     INFO - Closing logging queue
[task 2020-10-06T13:13:08.188Z] 13:13:08     INFO - queue closed
[task 2020-10-06T13:13:08.189Z] 13:13:08 CRITICAL - Test harness output was not a valid structured log message: 
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL - Traceback (most recent call last):
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/runtests.py", line 16, in <module>
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     rv = wptrunner.main()
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 416, in main
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     return start(**kwargs)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 400, in start
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     rv = not run_tests(**kwargs) or logged_critical.has_log
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 190, in run_tests
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     **kwargs)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py", line 92, in get_loader
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     chunker_kwargs=chunker_kwargs)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testloader.py", line 247, in __init__
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     self._load_tests()
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testloader.py", line 310, in _load_tests
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     for test_path, test_type, test in self.iter_tests():
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testloader.py", line 303, in iter_tests
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     yield test_path, test_type, self.get_test(manifest_file, test, inherit_metadata, test_metadata)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/testloader.py", line 263, in get_test
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     return wpttest.from_manifest(manifest_file, manifest_test, inherit_metadata, test_metadata)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wpttest.py", line 662, in from_manifest
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     return test_cls.from_manifest(manifest_file, manifest_test, inherit_metadata, test_metadata)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/wpttest.py", line 550, in from_manifest
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL -     mismatch_refs)
[task 2020-10-06T13:13:08.190Z] 13:13:08 CRITICAL - TypeError: __init__() takes at least 7 arguments (6 given)
[task 2020-10-06T13:13:08.394Z] 13:13:08    ERROR - Return code: 1
[task 2020-10-06T13:13:08.395Z] 13:13:08    ERROR - No checks run.
[task 2020-10-06T13:13:08.395Z] 13:13:08    ERROR - No suite end message was emitted by this harness.
[task 2020-10-06T13:13:08.395Z] 13:13:08 CRITICAL - # TBPL FAILURE #
[task 2020-10-06T13:13:08.395Z] 13:13:08  WARNING - setting return code to 2
...

Upstream PR was closed without merging
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/7be538be385a
Load specialPowers web extension in gecko-only wpt tests, r=jmaher,twisniewski,whimboo
https://hg.mozilla.org/integration/autoland/rev/aea82e971e8c
Ensure paths passed to Addons.install use os path separators only, r=marionette-reviewers,whimboo

https://hg.mozilla.org/mozilla-central/rev/7be538be385a
https://hg.mozilla.org/mozilla-central/rev/aea82e971e8c

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
status-firefox83: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 83 Branch
Flags: needinfo?(james)
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9438172e8c66
[wpt PR 25982] - [Gecko Bug 1668458] Load specialPowers web extension in gecko-only wpt tests, a=testonly
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: