1678663 - Make web-platform-tests run with py3 in CI

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task)

Description

[James Graham [:jgraham]]

This depends on upstream switching over; the timetable for that is https://github.com/web-platform-tests/rfcs/blob/master/rfcs/py_3.md Based on that timetable, we should expect to switch at the start of December.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Only install enum for python 2,

It's not the same as the stdlib for 3.6+ and is present on 3

Comment 2

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Use correct error for socket timeout in Py3,

Comment 3

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Don't try to set the environment for firefox_android,

This doesn't make sense since tests are running on the device, not
locally.

Comment 4

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Change wptrunner multiprocessing start method to spawn,

This makes Linux match Windows and macOS, and avoids a py3-only bug
where mozrunner desctructors can end up being called incorrectly in
the child process, so causing tests to fail.

To do this correctly turns out to require using an explicit context
which we pass down to everywhere we use multiprocessing. That's
because wptrunner isn't usually invoked directly but from a frontend
like wpt run or mach. Therefore spawned processed don't go through
the same entrypoint as the initial process and so don't set the
context themselves.

This unfortunately requires backwards-incompatible changes in mozlog,
so that will need to be released for this to land upstream.

Comment 5

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Fix TypeError under Python 3,

data[bytes] must exist but can be None; comaping None with an integer
is an error in Py3

Comment 6

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Fix stash unittests with multiprocessing.spawn,

The problem was that classes defined in functions can't be recreated in
child processes. So we need to make the classes global instead.

Comment 7

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Run wpt tests under Python 3 on Taskcluster,

Comment 8

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Fix usage of bytes in cookie tests,

Comment 9

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Fix use of iteritems in WebDriver helpers,

Comment 10

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Support passing adb binary in to wptrunner,

Comment 11

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Fix rather broken handling of errors during setup,

Comment 12

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Allow mozrunner.Runner instances to avoid cleanup in __del__,

The change in behaviour of the gc in Python 3 means that del is
being called in cases where it previously wasn't, and this can have
unfortunate side effects where del is being used to clean up
resources.

In particular with web-platform-tests running on Android, we end up
closing the test application unexpectedly. That causes widespread test
failures.

In general using del for this kind of cleanup is problematic
because it's not defined exactly when, or if, it's called. It's much
better to require explcit cleanup (and better still to make that
scope-based, using a context manager, although this patch doesn't do
so).

Therefore we add an option to only do explicit cleanup. This is
currently opt-in to make the breakage as small as possible, but
ultimately ought to be the default and only behaviour. We also change
the wpt android runner to use this behaviour.

Comment 13

[Phabricator Automation]

Comment on attachment 9192219 [details]
Bug 1678663 - Fix use of iteritems in WebDriver helpers,

Revision D99253 was moved to bug 1680114. Setting attachment 9192219 [details] to obsolete.

Comment 14

[James Graham [:jgraham]]

Attachment: Bug 1678663 - python3 - malformed \N character escape is-parseable (py3),

Comment 15

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Ignore connection errors during webdriver shutdown,

Comment 16

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Swallow exceptions during shutdown of fix_stacks.py,

Under Python 3 on Windows we sometimes get an OSError which may be a
race with the process exiting. There's not much we can do about
exceptions during shutdown so just ignore them.

Comment 17

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Add long timeout to test that times out on win 7 opt in py3,

This test was previously stable but seems to run a litle slower in
Python 3 so we see some timeouts. Make the timeout longer so avoid the
failure.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c9931fc09331
Don't try to set the environment for firefox_android, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/c584e6f853e6
Support passing adb binary in to wptrunner, r=bc
https://hg.mozilla.org/integration/autoland/rev/6d4a8738da36
Only install enum for python 2, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/4fbd1c323b6d
Fix rather broken handling of errors during setup, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/aeb183f1c3f0
Use correct error for socket timeout in Py3, r=marionette-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/e63f988a3d82
Fix TypeError under Python 3, r=bc
https://hg.mozilla.org/integration/autoland/rev/38d6e1fcab14
Change wptrunner multiprocessing start method to spawn, r=karlcow,bc
https://hg.mozilla.org/integration/autoland/rev/4a753391c683
Allow mozrunner.Runner instances to avoid cleanup in __del__, r=whimboo
https://hg.mozilla.org/integration/autoland/rev/3ab9b82d7122
Fix stash unittests with multiprocessing.spawn, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/3fdacb63d3c3
Fix usage of bytes in cookie tests, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/00c401c41d16
Ignore connection errors during webdriver shutdown, r=webdriver-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/d5afbee022e5
Swallow exceptions during shutdown of fix_stacks.py, r=bc
https://hg.mozilla.org/integration/autoland/rev/4967a4a57ab2
Add long timeout to test that times out on win 7 opt in py3, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/70a2e2a4b156
Run wpt tests under Python 3 on Taskcluster, r=bc

Comment 19

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Failed to create upstream wpt PR due to merge conflicts. This requires fixup from a wpt sync admin.

Comment 20

[Natalia Csoregi [:nataliaCs]]

Backed out aeb183f1c3f04a60f59841de3f0c231bc836b3b0 for causing marionette failures

backout: https://hg.mozilla.org/integration/autoland/rev/dff02098b9a627cce58cbfd8553597833c807456

push: https://treeherder.mozilla.org/jobs?repo=autoland&revision=70a2e2a4b156e8188b62988f9bc5e19aeab33a83&searchStr=marionette&selectedTaskRun=PbOVw8JaRdO8o6azb-sfFQ.0

failure log: https://treeherder.mozilla.org/logviewer?job_id=324458602&repo=autoland&lineNumber=51968

[task 2020-12-14T16:18:44.291Z] 16:18:44 INFO - Failed to gather test failure debug: Process crashed (Exit code: 1) (Reason: [Errno 10054] An existing connection was forcibly closed by the remote host)
[task 2020-12-14T16:18:44.292Z] 16:18:44 ERROR - TEST-UNEXPECTED-ERROR | testing/marionette/harness/marionette_harness/tests/unit/test_crash.py TestCrash.test_crash_chrome_process | NameError: global name 'TimeoutError' is not defined
[task 2020-12-14T16:18:44.292Z] 16:18:44 INFO - Traceback (most recent call last):
[task 2020-12-14T16:18:44.292Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_harness\marionette_test\testcases.py", line 214, in run
[task 2020-12-14T16:18:44.293Z] 16:18:44 INFO - testMethod()
[task 2020-12-14T16:18:44.293Z] 16:18:44 INFO - File "Z:\task_1607961860\build\tests\marionette\tests\testing\marionette\harness\marionette_harness\tests\unit\test_crash.py", line 130, in test_crash_chrome_process
[task 2020-12-14T16:18:44.294Z] 16:18:44 INFO - self.assertRaisesRegexp(IOError, "Process crashed", self.crash, parent=True)
[task 2020-12-14T16:18:44.294Z] 16:18:44 INFO - File "c:\mozilla-build\python\lib\unittest\case.py", line 993, in assertRaisesRegexp
[task 2020-12-14T16:18:44.294Z] 16:18:44 INFO - callable_obj(*args, **kwargs)
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - File "Z:\task_1607961860\build\tests\marionette\tests\testing\marionette\harness\marionette_harness\tests\unit\test_crash.py", line 101, in crash
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - "about:crash{}".format("parent" if parent else "content")
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_driver\marionette.py", line 1510, in navigate
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - self._send_message("WebDriver:Navigate", {"url": url})
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_driver\decorators.py", line 27, in _
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - return func(*args, **kwargs)
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_driver\marionette.py", line 617, in _send_message
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - msg = self.client.request(name, params)
[task 2020-12-14T16:18:44.295Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_driver\transport.py", line 288, in request
[task 2020-12-14T16:18:44.296Z] 16:18:44 INFO - return self.receive()
[task 2020-12-14T16:18:44.296Z] 16:18:44 INFO - File "Z:\task_1607961860\build\venv\lib\site-packages\marionette_driver\transport.py", line 163, in receive
[task 2020-12-14T16:18:44.296Z] 16:18:44 INFO - except (TimeoutError, socket.timeout):
[task 2020-12-14T16:18:44.296Z] 16:18:44 INFO - TEST-INFO took 199ms

Comment 21

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c9931fc09331
https://hg.mozilla.org/mozilla-central/rev/c584e6f853e6
https://hg.mozilla.org/mozilla-central/rev/6d4a8738da36
https://hg.mozilla.org/mozilla-central/rev/4fbd1c323b6d
https://hg.mozilla.org/mozilla-central/rev/e63f988a3d82
https://hg.mozilla.org/mozilla-central/rev/38d6e1fcab14
https://hg.mozilla.org/mozilla-central/rev/4a753391c683
https://hg.mozilla.org/mozilla-central/rev/3ab9b82d7122
https://hg.mozilla.org/mozilla-central/rev/3fdacb63d3c3
https://hg.mozilla.org/mozilla-central/rev/00c401c41d16
https://hg.mozilla.org/mozilla-central/rev/d5afbee022e5
https://hg.mozilla.org/mozilla-central/rev/4967a4a57ab2
https://hg.mozilla.org/mozilla-central/rev/70a2e2a4b156

Comment 22

[James Graham [:jgraham]]

Attachment: Bug 1678663 - Pass multiprocessing context in when starting server,

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d41b11ea020
Pass multiprocessing context in when starting server, r=emilio

Comment 24

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/4d41b11ea020

Comment 25

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/26940 for changes under testing/web-platform/tests

Comment 26

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 27

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c45207c0ba80
[wpt PR 26940] - [Gecko Bug 1678663] Don't try to set the environment for firefox_android, a=testonly

Comment 29

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c45207c0ba80