1744249 - [wdspec] Several tests fail to run due to wd_server_config.json not found

Status: RESOLVED FIXED

(Testing :: geckodriver, defect, P2)

Description

[David Burns :automatedtester]

When working on bug 1700093 it was noticed that the collections tests are not being run due to error in wpt. See https://treeherder.mozilla.org/logviewer?job_id=360070977&repo=try&lineNumber=25410-25416

Example error in case tc loses the logs

 1:12.37 INFO STDOUT: _______________________________________________________________________________ ERROR at setup of test_shadow_root _______________________________________________________________________________
 1:12.37 INFO STDOUT:     @pytest.fixture
 1:12.37 INFO STDOUT:     def server_config():
 1:12.37 INFO STDOUT: >       with open(os.environ.get("WD_SERVER_CONFIG_FILE"), "r") as f:
 1:12.37 INFO STDOUT: E       FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/hw/_lj4mq2j1p9__s43mgfjt9j40000gn/T/pytest-j7ptg6w_/wd_server_config.json'
 1:12.37 INFO STDOUT: testing/web-platform/tests/webdriver/tests/support/fixtures.py
 1:12.37 INFO STDOUT: :64: FileNotFoundError
 1:12.37 INFO STDOUT: 

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Note that there are way more tests that are currently marked as expected ERROR and it looks like it's mainly the same problem:
https://searchfox.org/mozilla-central/search?q=ERROR&path=meta%2Fwebdriver%2F

And that doesn't seem to be an issue with Firefox only. As wpt.fyi shows all the browsers are affected.

Running a git bisect ended with the following result:

✦2 ➜ git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[cd82725023b0bbde750858397134bbdc0df1fd36] Pass server config to WebDriver via a file instead of an env variable. (#28834)

So that is https://github.com/web-platform-tests/wpt/pull/28834

James, would you have the time to check what's wrong here?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The WPT sync for that change was done via bug 1709566 for Firefox 90.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I had a quick peak and as it looks like the server config file gets deleted by Pytest itself. It seems to clean-up the temporary folder as passed in via --basetemp and as such the file is no longer available. Using another subfolder for pytest results seems to work fine. I'll check Pytest code to see if that is an expected behavior or a bug.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

And indeed the Pytest documentation for the default base temporary directory gives this warning:

The contents of mydir will be completely removed, so make sure to use a directory for that purpose only.

As such lets use a different path for Pytest specific tmp file usage.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1744249 - [wptrunner] Use a dedicated pytest cache to prevent removal of wd_server_config.json.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1744249 - [wdspec] Fix broken usage of inline fixture in get computed label/role tests.

Depends on D133184

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1744249 - [wdspec] Updated expectation data for tests that errored due to missing wd_server_config.json file.

Depends on D133185

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fa1bcb99f5c4
[wptrunner] Use a dedicated pytest cache to prevent removal of wd_server_config.json. r=jgraham
https://hg.mozilla.org/integration/autoland/rev/c33bd46c07ae
[wdspec] Fix broken usage of inline fixture in get computed label/role tests. r=webdriver-reviewers,jgraham
https://hg.mozilla.org/integration/autoland/rev/9573b719e2c1
[wdspec] Updated expectation data for tests that errored due to missing wd_server_config.json file. r=webdriver-reviewers,jgraham

Comment 9

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/31954 for changes under testing/web-platform/tests

Comment 10

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/fa1bcb99f5c4
https://hg.mozilla.org/mozilla-central/rev/c33bd46c07ae
https://hg.mozilla.org/mozilla-central/rev/9573b719e2c1

Comment 11

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:whimboo, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, given that we are early in the beta phase lets get all of these patches uplifted to have a better test coverage for wdspec tests on the beta branch. Thanks.

Comment 14

[Dianna Smith [:diannaS]]

https://hg.mozilla.org/releases/mozilla-beta/rev/c28883b33e2a
https://hg.mozilla.org/releases/mozilla-beta/rev/6385f4d65729
https://hg.mozilla.org/releases/mozilla-beta/rev/3103fedbcd3a