Closed Bug 1714406 Opened 4 months ago Closed 3 months ago

Run test-verify and test-coverage tasks with Python 3

Categories

(Testing :: General, task)

Default
task

Tracking

(firefox91 fixed)

RESOLVED FIXED
91 Branch
Tracking Status
firefox91 --- fixed

People

(Reporter: ahal, Assigned: gbrown)

References

(Blocks 2 open bugs)

Details

Attachments

(1 file)

Run test-verify tasks with Python 3. :bwc recently discovered these were still running with Python 2. Since many of the underlying harnesses are already switched over, we should switch TV as well.

Blocks: 1714411

From bug 1713239, contrasting
https://treeherder.mozilla.org/jobs?repo=try&revision=4a1b5f5af399f0500a03a17a8ca5b723f520e06e
https://treeherder.mozilla.org/jobs?repo=try&revision=49419f976423004f1299869a256f789f14fa0280

I see py2 used for create_virtualenv on TV only.
https://firefoxci.taskcluster-artifacts.net/UhP4H-3zRwOfH7xS2TT6CA/0/public/logs/live_backing.log

[task 2021-05-28T17:43:32.849Z] 17:43:32     INFO - Creating virtualenv Z:\task_1622222660\build\venv
[task 2021-05-28T17:43:32.849Z] 17:43:32     INFO - Running command: ['c:\\mozilla-build\\python\\python.exe', 'Z:\\task_1622222660\\mozharness\\third_party\\python\\virtualenv\\virtualenv.py', '--always-copy', 'Z:\\task_1622222660\\build\\venv'] in Z:\task_1622222660\build
[task 2021-05-28T17:43:32.849Z] 17:43:32     INFO - Copy/paste: c:\mozilla-build\python\python.exe Z:\task_1622222660\mozharness\third_party\python\virtualenv\virtualenv.py --always-copy Z:\task_1622222660\build\venv
[task 2021-05-28T17:43:32.849Z] 17:43:32     INFO - Using partial env: {'VIRTUALENV_NO_DOWNLOAD': '1'}
[task 2021-05-28T17:43:34.028Z] 17:43:34     INFO -  created virtual environment CPython2.7.15.final.0-64 in 857ms
Assignee: nobody → whole.grains

...and that's probably because...

[taskcluster 2021-05-28T17:42:39.881Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python\python.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --verify --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --download-symbols ondemand

vs

[taskcluster 2021-05-28T19:30:35.923Z] Executing command 0: C:/mozilla-build/python3/python3.exe run-task -- c:\mozilla-build\python3\python3.exe -u mozharness\scripts\desktop_unittest.py --cfg mozharness\configs\unittests\win_unittest.py --setpref=webgl.out-of-process=false --mochitest-suite=mochitest-media --setpref=media.peerconnection.mtransport_process=false --setpref=network.process.enabled=false --download-symbols ondemand
See Also: → 1716860

super hack:
https://treeherder.mozilla.org/jobs?repo=try&revision=2b4cf9aa66990891914f3c700c7d7d71e8b98ebc

those two variables use:
log_errorsummary=[<_io.TextIOWrapper name='/builds/worker/workspace/build/blobber_upload_dir/mochitest-browser-chrome-test1_errorsummary.log' mode='w' encoding='UTF-8'>],

Yeah, maybe? Go ahead and take this bug if you like -- I don't have a full solution pending.

I think you will also need something like this for Android TV:
https://hg.mozilla.org/try/rev/89b55837e3af8a47c480c1a6e1837ffdef97a72b#l4.13

and a clone of your mochitest changes in runreftest.py.

But it might be worth pondering follow-up on https://searchfox.org/mozilla-central/rev/c114db74a92cf15096dfda02255e125949b0e070/layout/tools/reftest/runreftest.py#603 instead.

See Also: → 1714688
Blocks: 1714688

:jmaher verified that test-coverage works okay on py3 without changes, so I'll roll that into this bug:
https://treeherder.mozilla.org/jobs?repo=try&revision=69f5ab042ea0737e027c6147bb37ea0fe6a79c86

Summary: Run test-verify tasks with Python 3 → Run test-verify and test-coverage tasks with Python 3

Run TV and TC tasks with python 3. Several issues are addressed for TV:

  • minor Android-only str/int type confusion for mozinfo parameter;
  • the type of mozlog-generated log* options changes under py3, causing
    difficulty when deepcopy'ing options; avoided by removing the need
    for deepcopy;
  • on Windows, with py 3.6, repeated process launches trigger a python
    bug; avoided by running self-tests only once per TV session.
Pushed by whole.grains@protonmail.com:
https://hg.mozilla.org/integration/autoland/rev/7b0edcdf1dbe
Run test-verify and test-coverage tasks with python 3; r=jmaher
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 91 Branch
Regressions: 1718298
You need to log in before you can comment on or make changes to this bug.