1162230 - Mozharness looks for virtualenv.py in a soon to be deprecated path on Windows 2008 slaves

Status: RESOLVED DUPLICATE of bug 1297515

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Mark Cornmesser [:markco] OOO 2024/04/15]

It seems that mozharness is hardcoded to look for the virtlenv.py in C:\mozilla-build\buildbotve\:

11:53:13     INFO - Creating virtualenv c:\builds\moz2_slave\try-w32_g-00000000000000000000\build\venv
11:53:13     INFO - Running command: ['c:\\mozilla-build\\python27\\python.exe', 'c:/mozilla-build/buildbotve/virtualenv.py', '--no-site-packages', '--distribute', 'c:\\builds\\moz2_slave\\try-w32_g-00000000000000000000\\build\\venv'] in c:\builds\moz2_slave\try-w32_g-00000000000000000000\build
11:53:13     INFO - Copy/paste: c:\mozilla-build\python27\python.exe c:/mozilla-build/buildbotve/virtualenv.py --no-site-packages --distribute c:\builds\moz2_slave\try-w32_g-00000000000000000000\build\venv
11:53:13     INFO -  c:\mozilla-build\python27\python.exe: can't open file 'c:/mozilla-build/buildbotve/virtualenv.py': [Errno 2] No such file or directory
11:53:13    ERROR - Return code: 2 

As we move into Puppet configuration for 2008 the buildbotve directory is going to be deprecated in order have the Windows directories more consistent with posix buildbot virtual environment directories. For the creation of the new directories there is a virtualenv.py located at C:\mozilla-build. We may want to use that one. 


As a temp. work around, in the Pupept managed startbuildbot.bat a symbolic link is created between C:\mozilla-build\virtualenv.py and C:\mozilla-build\buildbotve\virtualenv.py. It seems to work for now. However, I would like to remove it as soon as possible.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

A really simple solution to this particular issue may be to just include a copy of virtualenv.py in mozharness.

We'll also run into a problem with other things referencing 'buildbotve' directly:

configs/web_platform_tests/test_config_windows.py:        'virtualenv': [sys.executable, 'c:/mozilla-source/cedar/python/virtualenv/virtualenv.py'], #'c:/mozilla-build/buildbotve/virtualenv.py'],
configs/web_platform_tests/prod_config_windows.py:        'virtualenv': [sys.executable, 'c:/mozilla-build/buildbotve/virtualenv.py'],
configs/b2g/desktop_windows32.py:            'c:\\mozilla-build\\buildbotve\\scripts\\buildbot'
configs/b2g/desktop_windows32.py:            'c:/mozilla-build/buildbotve/virtualenv.py'
configs/b2g/desktop_windows32.py:                'C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_base_windows_32_builds.py:            'c:\\mozilla-build\\buildbotve\\scripts\\buildbot'
configs/builds/releng_base_windows_32_builds.py:            'c:/mozilla-build/buildbotve/virtualenv.py'
configs/builds/releng_base_windows_32_builds.py:                'C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_base_windows_64_builds.py:            'c:\\mozilla-build\\buildbotve\\scripts\\buildbot'
configs/builds/releng_base_windows_64_builds.py:            'c:/mozilla-build/buildbotve/virtualenv.py'
configs/builds/releng_base_windows_64_builds.py:        'PATH': 'C:/mozilla-build/python27;C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_sub_windows_configs/32_debug.py:                'C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_sub_windows_configs/32_mulet.py:                'C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_sub_windows_configs/64_debug.py:        'PATH': 'C:/mozilla-build/python27;C:/mozilla-build/buildbotve/scripts;'
configs/builds/releng_sub_windows_configs/64_graphene.py:                'C:/mozilla-build/buildbotve/scripts;'
configs/talos/windows_config.py:        'virtualenv': [PYTHON, 'c:/mozilla-build/buildbotve/virtualenv.py'],
configs/unittests/win_unittest.py:        'virtualenv': [sys.executable, 'c:/mozilla-build/buildbotve/virtualenv.py'],
configs/marionette/windows_config.py:        'virtualenv': ['c:/mozilla-build/python27/python', 'c:/mozilla-build/buildbotve/virtualenv.py'],

Those may be less easy to fix..

Comment 2

[Armen [:armenzg]]

+1!

Comment 3

[Selena Deckelmann :selenamarie :selena]

Attachment: bundle_virtualenv.diff

Attempt to add virtualenv to mozharness.

Comment 4

[Selena Deckelmann :selenamarie :selena]

Attachment: Adding virtualenv 13.0.3 to mozharness

Oops, forgot to properly export this patch!

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8616175 [details] [diff] [review]
Adding virtualenv 13.0.3 to mozharness

Review of attachment 8616175 [details] [diff] [review]:
-----------------------------------------------------------------

<3

It might be worth adding a virtualenv-README.txt to external_tools/ describing how to go about updating it (download from where, untar, any other modifications)

Jordan's review would be good here, too.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8616175 [details] [diff] [review]
Adding virtualenv 13.0.3 to mozharness

..as the de facto mozharness module owner :)

Comment 7

[Jordan Lund (:jlund)]

Comment on attachment 8616175 [details] [diff] [review]
Adding virtualenv 13.0.3 to mozharness

Review of attachment 8616175 [details] [diff] [review]:
-----------------------------------------------------------------

neato, but, just to get me up to speed here, why is this package being added directly into mh? My instinct would lean me to think that we should have this installed on our machines with change mgmt tools like we do with other software. Is this painful to do for puppet windows or GPO?

Does the decision of putting this in mh change if mh is going in-tree within the next few weeks?

has this been tested anywhere on try?

Comment 8

[Jordan Lund (:jlund)]

Comment on attachment 8616175 [details] [diff] [review]
Adding virtualenv 13.0.3 to mozharness

Review of attachment 8616175 [details] [diff] [review]:
-----------------------------------------------------------------

I just re-read comment 0 so I'm guessing the issue is that there could be two paths depending on if it's puppet or GPO. Since we are going to have to change all the paths anyway in comment 1 to point at external_tools/virtualenv, could we not just let the symbolic link markco setup persist until we are completely on puppet windows and then fix the config paths? Or am I reading this wrong? thanks for bearing my ignorance :)

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Really, the issue is that this is a Python dependency of MH, which is otherwise dependency-free.  When the dependency was added, someone did the windows equivalent of 'find c:\ -name virtualenv.py' and used the first thing that popped up, and that path later moved because the dependency was unknown.

As a tool that's meant to be used directly from its repo, mozharness should "vendor" all of its dependencies.  This is its only dependency (you could describe hgtool.py and gittool.py as dependencies, too, and they are also vendored, in a way)

As for moving in-tree, virtualenv is already there - https://dxr.mozilla.org/mozilla-central/source/python/virtualenv

Comment 10

[Armen [:armenzg]]

I don't think you need to commit the whole repo. Only the module virtualenv.py is sufficient since it is self contained.

See in here https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild

This adds a lot of value as it removes complexity for deployments.

Side note, Mozilla-build included virtualenv.py and pip bug 782958.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Interesting, so we have *two* copies of virtualenv in the gecko codebase.

You're right, just virtualenv.py is sufficient -- that's also what puppet does:
  https://github.com/mozilla/build-puppet/blob/master/modules/python/manifests/virtualenv/prerequisites.pp#L15

Comment 12

[Armen [:armenzg]]

We should also think as if we're going to use this virtualenv.py as the default go-to and overwrite that value in the configs if wanted.

Comment 13

[Jordan Lund (:jlund)]

(In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #12)
> We should also think as if we're going to use this virtualenv.py as the
> default go-to and overwrite that value in the configs if wanted.

+1 to making this configurable.

I guess it also be nice to use the same in-gecko-tree virtualenv.py if possible rather than adding another copy of it

Comment 14

[Selena Deckelmann :selenamarie :selena]

(In reply to Jordan Lund (:jlund) from comment #13)
> (In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #12)
> > We should also think as if we're going to use this virtualenv.py as the
> > default go-to and overwrite that value in the configs if wanted.
> 
> +1 to making this configurable.

Can we talk about why this is being requested? I would like to understand why we would want to support multiple versions of virtualenv rather than a single version of virtualenv that meets our needs.

> I guess it also be nice to use the same in-gecko-tree virtualenv.py if
> possible rather than adding another copy of it

That's a good idea! The one that's in-tree is reasonably recent as well. With mozharness being standalone, it seems challenging to invoke that in tests without pulling in the tree. Should I just mock something up?

Comment 15

[Armen [:armenzg]]

(In reply to Selena Deckelmann :selenamarie :selena from comment #14)
> (In reply to Jordan Lund (:jlund) from comment #13)
> > (In reply to Armen Zambrano G. (:armenzg - Toronto) from comment #12)
> > > We should also think as if we're going to use this virtualenv.py as the
> > > default go-to and overwrite that value in the configs if wanted.
> > 
> > +1 to making this configurable.
> 
> Can we talk about why this is being requested? I would like to understand
> why we would want to support multiple versions of virtualenv rather than a
> single version of virtualenv that meets our needs.
> 
The nature of mozharness allows you to take default values and overwrite them in a config.
This is not a feature request.
We simply want to agree that we're using this checked-in version unless otherwise stated (explicitely set in the configs).

Moving towards only using one version of virtualenv.py requires a bit more careful changes (remove explicit paths) and testing.
I would not be surprised if the paths of the virtualenv files on each platform is scattered across puppet and other repos beyond mozharness.

In both cases we need to test it on try.

Comment 16

[Selena Deckelmann :selenamarie :selena]

Met today with :jlund and plan is: 

* get a patch together with virtualenv in-MH and test on all platforms to suss out any bugs
* post a full try run once it looks pretty good
* Deal with switching to using the other virtualenv in-tree later