1572448 - --setpref options are not forwarded to pytestrunner

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With bug 1350905 the --setpref command line option has been added, so that custom preferences can be specified. While this works great for wpt in general, those options sadly don't get forwarded to the pytestrunner. As result the Wd jobs completely ignore them.

This is needed for the fission jobs, which use --setpref fission.autostart=true.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The capabilities for the WebDriverProtocol and here specifically the preferences for Firefox are set at:

https://searchfox.org/mozilla-central/rev/9ae20497229225cb3fa729a787791f424ff8087b/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py#125-127

Right now this is only for network.dns.localDomains, and nothing else. So to correctly pass-over custom preferences to Firefox, they will have to be added here.

But note that those preferences will be set when creating a session, and as such only work for those which do not need a restart of Firefox. The preference for Fission enabled would have to be set when the profile gets created, and before Firefox gets started.

So I see two ways in doing that:

1. Having an option for geckodriver like --setpref to allow setting preferences when the profile gets created
2. Having another pref section beside moz:firefoxOptions["prefs"] in the capabilities like forcedPrefs which would be set during profile creation

Andreas and James, what would you suggest?

Another option could be to request a FISSION environment variable which Firefox would respect. Something similar to HEADLESS.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

False alarm, the problem actually is that preferences as specified via --setpref on the command line are all handled as string, and trying to set set "true" for a boolean pref indeed fails.

So we need a type conversion in wptrunner when we read-in those values and cast to the right type.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ace5b42b8cc1550a13cfc44da9501a83c298ea4b

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1572448 - [wptrunner] Cast preference values from --setpref to their real type. r=#webdriver

The extra preferences as set via --setpref will by default be of type string.
To actually set those for the browser, they will have to be stored with their
real type.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1572448 - [wptrunner] Forward custom preferences from --setpref WebDriverProtocol. r=#webdriver

Depends on D41281

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

From try build:

https://taskcluster-artifacts.net/WwibeCnPREufAhq4NQ-TPg/0/public/logs/live_backing.log

[task 2019-08-08T19:05:55.609Z] + : mozharness args --test-type=wdspec '--setpref="fission.autostart=true"' --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --allow-software-gl-layers --total-chunk=2 --this-chunk=2 --download-symbols=ondemand
[..]
[task 2019-08-08T19:09:58.136Z] 19:09:58     INFO - PID 898 | 1565291398127	webdriver::server	DEBUG	-> POST /session {"capabilities": {"alwaysMatch": {"moz:firefoxOptions": {"binary": "/builds/worker/workspace/build/application/firefox/firefox", "prefs": {"network.process.enabled": false, "fission.autostart": true, "network.dns.localDomains": "www1.xn--n8j6ds53lwwkrqhv28a.web-platform.test,xn--lve-6lad.www1.web-platform.test,[..],www2.www.not-web-platform.test", "media.peerconnection.mtransport_process": false}}}}}

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1572448 - [mozprofile] Cast boolean preference values case-insensitive. r=jgraham

Depends on D41282

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b880c3941ed3
[wptrunner] Forward custom preferences from --setpref WebDriverProtocol. r=webdriver-reviewers,jgraham
https://hg.mozilla.org/integration/autoland/rev/46206b874a55
[mozprofile] Cast boolean preference values case-insensitive. r=jgraham

Comment 9

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/18351 for changes under testing/web-platform/tests

Comment 10

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 11

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7adc0ab9aa56
Backed out changeset 46206b874a55 for causing test_preferences.py to perma fail CLOSED TREE

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The mozprofile fix was optional and given that mozbase tests exist which check that the case remains, I just abandoned the revision.

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b880c3941ed3