Closed Bug 1672181 Opened 4 years ago Closed 3 years ago

Create a 'python-version' key for the 'test' kind


(Firefox Build System :: Task Configuration, task, P2)



(Not tracked)



(Reporter: ahal, Assigned: ahal)


(Blocks 1 open bug)



(1 obsolete file)

Many of our test harnesses are Python 3 compatible, though they still run with Python 2 in CI since mozharness hasn't been converted yet.

We should set things up such that tasks can specify which version of Python they should run with (similar to the source-test kind). This way we'll be able to easily identify what failures in mozharness need to be addressed in order to turn them on in CI with Python 3.

This will also allow us to convert the test portions of bug 1532427 piece by piece, rather than all at once.

Assignee: nobody → ahal
Pushed by
[taskgraph] Make test tasks py3 by default and add a py2 override r=bc,bhearsum

I confirmed again that the only diff in the taskgraph was an additional PYTHON=python2.7 environment variable. As far as I can tell, this env should only be used in where the value it is replacing was also python2.7. I have no idea how this patch could have caused an intermittent failure anywhere :/..

I guess I'll try renaming the environment variable to make sure there's nothing else that was using it.

Depends on: 1679467

The errors are still reproducible after changing the environment:

There's something fishy going on here.. I'm 99.2% sure that these failures aren't the result of my patch, but rather something to do with docker-worker. I filed bug 1679467 to investigate further.

This same error was caused by a patch that only touches a comment in So once bug 1679467 is fixed, this can be re-landed.

Pushed by
[taskgraph] Make test tasks py3 by default and add a py2 override r=bc,bhearsum
See Also: → 1679778
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → 85 Branch
Regressions: 1679952
Resolution: FIXED → ---
Target Milestone: 85 Branch → ---

The backout was for bug 1679952.

After this change, Android 4.0 API16+ pgo Bpgo(run) tasks

have 2 exceptions like this:

[task 2020-11-30T19:43:02.329Z] 19:43:02     INFO - run-tests - Wall time: 892s; CPU: 25%; Read bytes: 262144; Write bytes: 1362161664; Read time: 0; Write time: 60016
[task 2020-11-30T19:48:02.347Z] Exception ignored in: <bound method Marionette.__del__ of <marionette_driver.marionette.Marionette object at 0x7fae1816f5f8>>
[task 2020-11-30T19:48:02.347Z] Traceback (most recent call last):
[task 2020-11-30T19:48:02.347Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/", line 532, in __del__
[task 2020-11-30T19:48:02.347Z]     self.cleanup()
[task 2020-11-30T19:48:02.347Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/", line 517, in cleanup
[task 2020-11-30T19:48:02.347Z]     self.delete_session()
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/", line 1220, in delete_session
[task 2020-11-30T19:48:02.348Z]     self._send_message("WebDriver:DeleteSession")
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/", line 37, in _
[task 2020-11-30T19:48:02.348Z]     m._handle_socket_failure()
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/workspace/build/venv/lib/python3.6/site-packages/marionette_driver/", line 684, in _handle_socket_failure
[task 2020-11-30T19:48:02.348Z]     returncode = self.instance.runner.wait(timeout=self.shutdown_timeout)
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/", line 153, in wait
[task 2020-11-30T19:48:02.348Z]     while self.is_running():
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/", line 186, in is_running
[task 2020-11-30T19:48:02.348Z]     return self.returncode is None
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozrunner/mozrunner/base/", line 136, in returncode
[task 2020-11-30T19:48:02.348Z]     if self.app_ctx.device.process_exist(self.app_ctx.remote_process):
[task 2020-11-30T19:48:02.348Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/", line 3118, in process_exist
[task 2020-11-30T19:48:02.349Z]     if self._pidof(app, timeout=timeout):
[task 2020-11-30T19:48:02.349Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/", line 1133, in _pidof
[task 2020-11-30T19:48:02.349Z]     procs = self.get_process_list(timeout=timeout)
[task 2020-11-30T19:48:02.350Z]   File "/builds/worker/checkouts/gecko/testing/mozbase/mozdevice/mozdevice/", line 2951, in get_process_list
[task 2020-11-30T19:48:02.350Z]     raise ADBTimeoutError("%s" % adb_process)
[task 2020-11-30T19:48:02.350Z] mozdevice.adb.ADBTimeoutError: args: /builds/worker/workspace/build/android-sdk-linux/platform-tools/adb wait-for-device shell ps; echo adb_returncode=$?, exitcode: None, stdout: 

and the Bpgo(run) task times out frequently on Android.

bc: are you looking at this? It looks like maybe the short term solution is to ensure we set PYTHON=python2.7 before invoking the tests. The actual error seems like it could be a change in the way the gc runs under py3? Having stuff in del methods is pretty scary... Hopefully there's some more straightforward solution :)

Flags: needinfo?(bob)

I hadn't planned on it though I had hoped to make use of this in my other python3 work. I haven't looked at the patch closely but it seems weird this is showing up as an ADBTimeoutError on Android. Is this the only reason the patch was backed out?

I'll keep the NI active to remind me and will revisit after I get my other python3 work landed.

This is I think the main thing blocking using wpt as Py3 which is on a bit of a deadline. So I can look and see if there's a simple workaround here.

See Also: → 1680752

jgraham: I think your approach in bug 1680752 is the approach to take where we run under python2.7 by default unless we've explicitly requested the test run under python3. There are many places in the tree even after my recent work on python3 compatibility where the various packages do not yet support python3. Attempting to switch to python3 before we resolve those issues will only lead to pain.

Flags: needinfo?(bob)

Looks like bug 1678663 implemented a way to specify Python 3 in the configs in the meantime here. While it's opt-in rather than opt-out, it serves the same purpose and is likely good enough.

Closed: 4 years ago3 years ago
Resolution: --- → DUPLICATE
Attachment #9188985 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.