1677532 - Test verification does not use mozharness create-virtualenv action

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Bob Clary [:bc] (inactive)]

Attempting to add Python3 support, I ran into the following error with Tvg

[task 2020-11-13T18:16:07.532Z] 18:16:07    ERROR - Exception during post-action for download-and-extract: Traceback (most recent call last):
[task 2020-11-13T18:16:07.533Z] 18:16:07    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2304, in run_action
[task 2020-11-13T18:16:07.534Z] 18:16:07    ERROR -     method(action, success=success and self.return_code == 0)
[task 2020-11-13T18:16:07.535Z] 18:16:07    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/verify_tools.py", line 53, in find_tests_for_verification
[task 2020-11-13T18:16:07.536Z] 18:16:07    ERROR -     self.find_modified_tests()
[task 2020-11-13T18:16:07.537Z] 18:16:07    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/per_test_base.py", line 353, in find_modified_tests
[task 2020-11-13T18:16:07.538Z] 18:16:07    ERROR -     self._find_misc_tests(dirs, changed_files, gpu=True)
[task 2020-11-13T18:16:07.539Z] 18:16:07    ERROR -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/per_test_base.py", line 125, in _find_misc_tests
[task 2020-11-13T18:16:07.540Z] 18:16:07    ERROR -     man.load(path)
[task 2020-11-13T18:16:07.541Z] 18:16:07    ERROR -   File "/builds/worker/workspace/build/tests/reftest/manifest/__init__.py", line 108, in load
[task 2020-11-13T18:16:07.542Z] 18:16:07    ERROR -     line = six.ensure_text(line)
[task 2020-11-13T18:16:07.543Z] 18:16:07    ERROR - AttributeError: 'module' object has no attribute 'ensure_text'

I had attempted to add six and vcversioner to the virtualenv via

|  diff --git a/testing/mozharness/configs/unittests/linux_unittest.py b/testing/mozharness/configs/unittests/linux_unittest.py
|  --- a/testing/mozharness/configs/unittests/linux_unittest.py
|  +++ b/testing/mozharness/configs/unittests/linux_unittest.py
|  @@ -30,6 +30,7 @@ else:
|   #####
|   config = {
|       ###
|  +    "virtualenv_modules": ["six==1.13.0", "vcversioner==2.16.0.0"],
|       "installer_path": INSTALLER_PATH,
|       "binary_path": BINARY_PATH,
|       "xpcshell_name": XPCSHELL_NAME,

But it appears that TVg does not use the mozharness action create-virtualenv. I couldn't see where the actions were set. six.ensure_* were added in six 1.12.0 and it appears the system version on that machine was earlier than that.

It would be good if verifcation supported create-virtualenv.

gbrown: Is this something you would like to take?

Comment 1

[Geoff Brown [:gbrown]]

Sure! I don't have immediate insight, but can have a look this week.

Comment 2

[Geoff Brown [:gbrown]]

I don't think test-verify does anything special related to create-virtualenv: the mozharness steps are generally defined by the script, like desktop_unittest.py.

test-verify (or -gpu, -wpt) tasks do run create-virtualenv, but only after download-and-extract. You can see that in successful test-verify runs on autoland:
https://treeherder.mozilla.org/logviewer?job_id=321937438&repo=autoland&lineNumber=858

That behavior seems fairly consistent across tasks and across test platforms: practically everything run from desktop_unittest.py, android_emulator_unittest.py, etc. runs create-virtualenv, generally after download-and-extract.

It might be easy to run create-virtualenv before download-and-extract, but that won't help: the virtualenv is created for the use of the test harness (like runreftest.py); but the exception in comment 0 is in the mozharness script itself.

I am a little hazy still on where the mozharness script finds six -- maybe in mozharness.zip? ...

Comment 3

[Bob Clary [:bc] (inactive)]

Mmm. I didn't see where verify set the actions in the same fashion as the others but I do see that the error is occurring before the create-virtualenv has a chance to run. I doubt we can bundle six with mozharness though. It would be nice if we could count on six being available from the system but I suppose that isn't doable either. I guess the real problem is that download-and-extract collects the tests which ends up executing code from the tests prior to the creation of the virtualenv. Perhaps we can split that out of download-and-extract and use a different action to collect the test information after create-virtualenv?

Comment 4

[Geoff Brown [:gbrown]]

We don't seem to be on the same page. Sorry about that.

six.py (version 1.13) is included in mozharness.zip, right in the mozharness root. It looks to me like that <mozharness>/six.py is used by test-verify when loading the reftest manifest, and it includes six.ensure_text(). That shouldn't have anything to do with create-virtualenv, or any venv.
https://treeherder.mozilla.org/logviewer?job_id=322014400&repo=try&lineNumber=853

But my observations do not explain why or how your try push failed. :(

Comment 5

[Bob Clary [:bc] (inactive)]

With some slightly updated patches which mostly removed six from mozharness scripts and configs, TVg is green for my try push. Don't ask me why.

Comment 6

[Geoff Brown [:gbrown]]

Oh, that's nice!

Re-ordering of create-virtualenv and the manifest operations is tricky: create-virtualenv relies on artifacts downloaded by download-and-extract, and test-verify checks manifests to better determine which artifacts need to be downloaded, giving the current order: check manifests, download-and-extract, create-virtualenv.

And such changes should not be necessary: the manifest loading that uses six should be using <mozharness-zip-extraction-point>/six.py, which is 1.13 and has ensure_text(). I verified this in try pushes against recent m-c, using print(six) and print(dir(six)) after the troublesome import; maybe you could do the same if this gives you any more trouble?

Comment 7

[Bob Clary [:bc] (inactive)]

Will do. Thanks.

Comment 8

[Geoff Brown [:gbrown]]

:bc - Have you had any more trouble of this sort? OK to resolve this now?

Comment 9

[Bob Clary [:bc] (inactive)]

I think I have worked around this issue by raising the minimum version of six to 1.13.0 in Bug 1680345 but the underlying problem is probably still there. I think it would be good if we could unify the various virtual environments we use so that we don't end up with one for mozharness, a different one for raptor and who knows what else.

But the underlying error is gone and we can wontfix or otherwise resolve this as you see fit.

Comment 10

[Geoff Brown [:gbrown]]

The use of create-virtualenv in mozharness introduces an additional runtime environment and the potential for confusion. It also provides the ability to customize the venv to the task. I am reluctant to lose that flexibility.