1632300 - Setting $PYTHON3 causes failure on MozillaBuild

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

$ ./mach configure
 0:00.72 Clobber not needed.
 0:00.73 Adding make options from C:\Users\Kagami\Documents\GitHub\gecko-dev\mozconfig
    AUTOCLOBBER=1
    MOZ_OBJDIR=c:/Users/Kagami/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32
    OBJDIR=c:/Users/Kagami/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32
    FOUND_MOZCONFIG=C:/Users/Kagami/Documents/GitHub/gecko-dev/mozconfig
    export FOUND_MOZCONFIG
 0:00.78 c:\mozilla-build\bin\mozmake.EXE -f client.mk -s configure
 0:00.82 cd c:/Users/Kagami/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32
 0:00.84 c:/Users/Kagami/Documents/GitHub/gecko-dev/configure
 0:01.26 Creating Python 3 environment
 0:01.37 Using base prefix 'c:\\mozilla-build\\python38'
 0:02.95 New python executable in C:\Users\Kagami\Documents\GitHub\gecko-dev\obj-x86_64-pc-mingw32\_virtualenvs\init_py3\Scripts\python.exe
 0:03.82 Installing setuptools, pip, wheel...
 0:14.98 done.
 0:15.17 c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\virtualenv.py:457: UserWarning: Hacking environment to allow binary Python extensions to build. You can make this warning go away by installing Visual Studio 2008. You can download the Express
Edition installer from http://go.microsoft.com/?linkid=7729279
 0:15.17   warnings.warn('Hacking environment to allow binary Python '
 0:15.25 Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
 0:15.26 Re-executing in the virtualenv
 0:16.03 Creating Python 3 environment
 0:16.14 Using base prefix 'C:\\mozilla-build\\python3'
 0:16.16 Overwriting C:\Users\Kagami\Documents\GitHub\gecko-dev\obj-x86_64-pc-mingw32\_virtualenvs\init_py3\lib\orig-prefix.txt with ne
w content
 0:16.16 New python executable in C:\Users\Kagami\Documents\GitHub\gecko-dev\obj-x86_64-pc-mingw32\_virtualenvs\init_py3\Scripts\python
3.exe
 0:16.17 Traceback (most recent call last):
 0:16.17   File "C:/Users/Kagami/Documents/GitHub/gecko-dev\third_party\python\virtualenv\virtualenv.py", line 2634, in <module>
 0:16.17     main()
 0:16.17   File "C:/Users/Kagami/Documents/GitHub/gecko-dev\third_party\python\virtualenv\virtualenv.py", line 870, in main
 0:16.17     symlink=options.symlink,
 0:16.17   File "C:/Users/Kagami/Documents/GitHub/gecko-dev\third_party\python\virtualenv\virtualenv.py", line 1162, in create_environm
ent
 0:16.17     install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
 0:16.18   File "C:/Users/Kagami/Documents/GitHub/gecko-dev\third_party\python\virtualenv\virtualenv.py", line 1567, in install_python
 0:16.18     shutil.copyfile(python_dll, os.path.join(os.path.dirname(py_executable), py_executable_dll))
 0:16.18   File "C:\mozilla-build\python3\lib\shutil.py", line 121, in copyfile
 0:16.18     with open(dst, 'wb') as fdst:
 0:16.18 PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Kagami\\Documents\\GitHub\\gecko-dev\\obj-x86_64-pc-mingw32\\_virtu
alenvs\\init_py3\\Scripts\\python3.dll'
 0:16.19 Traceback (most recent call last):
 0:16.19   File "c:/Users/Kagami/Documents/GitHub/gecko-dev/configure.py", line 181, in <module>
 0:16.19     sys.exit(main(sys.argv))
 0:16.19   File "c:/Users/Kagami/Documents/GitHub/gecko-dev/configure.py", line 52, in main
 0:16.19     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
 0:16.19   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 445, in run
 0:16.19     self.include_file(path)
 0:16.19   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 436, in include_file
 0:16.19     exec_(code, self)
 0:16.19   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\util.py", line 53, in exec_
 0:16.19     exec(object, globals, locals)
 0:16.20   File "C:/Users/Kagami/Documents/GitHub/gecko-dev/moz.configure", line 7, in <module>
 0:16.20     include('build/moz.configure/init.configure')
 0:16.20   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 773, in include_impl
 0:16.20     self.include_file(what)
 0:16.20   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 436, in include_file
 0:16.20     exec_(code, self)
 0:16.20   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\util.py", line 53, in exec_
 0:16.20     exec(object, globals, locals)
 0:16.20   File "C:/Users/Kagami/Documents/GitHub/gecko-dev/build/moz.configure/init.configure", line 255, in <module>
 0:16.20     def virtualenv_python3(env_python, build_env, mozillabuild, mozconfig, help):
 0:16.20   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 756, in decorator
 0:16.21     depends = DependsFunction(self, func, dependencies, when=when)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 129, in __init__
 0:16.21     sandbox._value_for(self)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 542, in _value_for
 0:16.21     return self._value_for_depends(obj)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\util.py", line 1018, in method_call
 0:16.21     cache[args] = self.func(instance, *args)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 551, in _value_for_de
pends
 0:16.21     value = obj.result()
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\util.py", line 1018, in method_call
 0:16.21     cache[args] = self.func(instance, *args)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 155, in result
 0:16.21     return self._func(*resolved_args)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\configure\__init__.py", line 1161, in wrapped
 0:16.21     return new_func(*args, **kwargs)
 0:16.21   File "C:/Users/Kagami/Documents/GitHub/gecko-dev/build/moz.configure/init.configure", line 365, in virtualenv_python3
 0:16.21     manager.build(python)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\virtualenv.py", line 520, in build
 0:16.21     self.create(python)
 0:16.21   File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python\mozbuild\mozbuild\virtualenv.py", line 258, in create
 0:16.21     raise Exception(
 0:16.21 Exception: Failed to create virtualenv: C:/Users/Kagami/Documents/GitHub/gecko-dev/obj-x86_64-pc-mingw32\_virtualenvs\init_py3
 (virtualenv.py retcode: 1)
 0:16.26 *** Fix above errors and then restart with\
 0:16.26                "./mach build"
 0:16.33 mozmake.EXE: *** [client.mk;115: configure] Error 1

Virtualenv first copies Python3.8 files and runs Python3.7 on it, causing failure. This doesn't happen when modifying hardcoded path in https://searchfox.org/mozilla-central/rev/41c3ea3ee8eab9ce7b82932257cb80b703cbba67/build/moz.configure/init.configure#335.

Comment 1

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: Bug 1632300 - Always prefer virtualenv python

Currently MozillaBuild with $PYTHON3 first creates virtualenv with the path from environment variable and then reexecutes with its own bundled executable. This patch forces it to run with virtualenv python when reexecuting.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d2a8c353d4a5
Always prefer virtualenv python r=glandium

Comment 3

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/d2a8c353d4a5