1331219 - MinGW/Python build failure on Windows 10

Status: RESOLVED DUPLICATE of bug 1275437

(Firefox Build System :: General, defect)

Description

[saber-nyan]

When I tried to build Firefox under Windows 10 with MinGW, I followed this ( https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Compiling_Mozilla_With_Mingw ) instruction. Building stopped at step `make -f client.mk` with following python traceback:

Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/mozilla/configure.py", line 124, in <module>
    sys.exit(main(sys.argv))
  File "/mozilla/configure.py", line 29, in main
    sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 334, in run
    self.include_file(path)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
    exec_(code, self)
  File "/mozilla/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "/mozilla/moz.configure", line 7, in <module>
    include('build/moz.configure/init.configure')
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 630, in include_impl
    self.include_file(what)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
    exec_(code, self)
  File "/mozilla/python/mozbuild/mozbuild/util.py", line 59, in exec_
    exec(object, globals, locals)
  File "/mozilla/build/moz.configure/init.configure", line 145, in <module>
    @imports('distutils.sysconfig')
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 613, in decorator
    depends = DependsFunction(self, func, dependencies, when=when)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 69, in __init__
    sandbox._value_for(self)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
    return self._value_for_depends(obj, need_help_dependency)
  File "/mozilla/python/mozbuild/mozbuild/util.py", line 925, in method_call
    cache[args] = self.func(instance, *args)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
    return obj.result
  File "/mozilla/python/mozbuild/mozbuild/util.py", line 943, in __get__
    setattr(instance, name, self.func(instance))
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 90, in result
    return self.func(*resolved_args)
  File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 938, in wrapped
    return new_func(*args, **kwargs)
  File "/mozilla/build/moz.configure/init.configure", line 192, in virtualenv_python
    manager.build(python)
  File "/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 447, in build
    result = self._log_process_output(args, cwd=self.topsrcdir)
  File "/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 174, in _log_process_output
    stderr=subprocess.STDOUT, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
*** Fix above errors and then restart with\
               "make -f client.mk build"

.mozconfig file: http://pastebin.com/Ybxca5XR
Full build log attached to the report.

Comment 1

[saber-nyan]

Attachment: Verbose build log

Comment 2

[Tooru Fujisawa [:arai]]

the error seems to be that python executable is not found while building virtualenv.

https://dxr.mozilla.org/mozilla-central/rev/ac3275723df59db0f09198fdb61b51e7002c391a/python/mozbuild/mozbuild/virtualenv.py#444-447
>         args = [self.python_path, __file__, 'populate', self.topsrcdir,
>             self.topobjdir, self.virtualenv_root, self.manifest_path]
> 
>         result = self._log_process_output(args, cwd=self.topsrcdir)

can you check the value of |self.python_path| there, and also see if the file exists?

Comment 3

[saber-nyan]

(In reply to Tooru Fujisawa [:arai] from comment #2)
> the error seems to be that python executable is not found while building
> virtualenv.
> 
> https://dxr.mozilla.org/mozilla-central/rev/
> ac3275723df59db0f09198fdb61b51e7002c391a/python/mozbuild/mozbuild/virtualenv.
> py#444-447
> >         args = [self.python_path, __file__, 'populate', self.topsrcdir,
> >             self.topobjdir, self.virtualenv_root, self.manifest_path]
> > 
> >         result = self._log_process_output(args, cwd=self.topsrcdir)
> 
> can you check the value of |self.python_path| there, and also see if the
> file exists?

Yes, there is a file `virtualenv.py`:
> $ ls -lh C:\cygwin64\mozilla\python\mozbuild\mozbuild\virtualenv.py
> -rw-r--r-- 1 rasva rasva 22K Jan 15 12:12 'C:\cygwin64\mozilla\python\mozbuild\mozbuild\virtualenv.py'

I changed the file to show `self.python_path`:
> $ grep -C 5 'args = \[self.python_path, __file__,' C:\cygwin64\mozilla\python\mozbuild\mozbuild\virtualenv.py
> 
>         # We need to populate the virtualenv using the Python executable in
>         # the virtualenv for paths to be proper.
> 
>         print("self.python_path == %s" % self.python_path) ## TODO: Remove!
>         args = [self.python_path, __file__, 'populate', self.topsrcdir,
>             self.topobjdir, self.virtualenv_root, self.manifest_path]
> 
>         result = self._log_process_output(args, cwd=self.topsrcdir)
> 
>         if result != 0:

During build it is printed:
> $ make -f client.mk -d
> * * *
> New python executable in /out/_virtualenv/bin/python2.7
> Also creating executable in /out/_virtualenv/bin/python
> Installing setuptools, pip, wheel...done.
> self.python_path == /out/_virtualenv/Scripts/python.exe ## Python binary path exists!
> Traceback (most recent call last):
>   File "/mozilla/configure.py", line 124, in <module>
>     sys.exit(main(sys.argv))
>   File "/mozilla/configure.py", line 29, in main
>     sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 334, in run
>     self.include_file(path)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
>     exec_(code, self)
>   File "/mozilla/python/mozbuild/mozbuild/util.py", line 59, in exec_
>     exec(object, globals, locals)
>   File "/mozilla/moz.configure", line 7, in <module>
>     include('build/moz.configure/init.configure')
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 630, in include_impl
>     self.include_file(what)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 325, in include_file
>     exec_(code, self)
>   File "/mozilla/python/mozbuild/mozbuild/util.py", line 59, in exec_
>     exec(object, globals, locals)
>   File "/mozilla/build/moz.configure/init.configure", line 145, in <module>
>     @imports('distutils.sysconfig')
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 613, in decorator
>     depends = DependsFunction(self, func, dependencies, when=when)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 69, in __init__
>     sandbox._value_for(self)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 409, in _value_for
>     return self._value_for_depends(obj, need_help_dependency)
>   File "/mozilla/python/mozbuild/mozbuild/util.py", line 925, in method_call
>     cache[args] = self.func(instance, *args)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 419, in _value_for_depends
>     return obj.result
>   File "/mozilla/python/mozbuild/mozbuild/util.py", line 943, in __get__
>     setattr(instance, name, self.func(instance))
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 90, in result
>     return self.func(*resolved_args)
>   File "/mozilla/python/mozbuild/mozbuild/configure/__init__.py", line 938, in wrapped
>     return new_func(*args, **kwargs)
>   File "/mozilla/build/moz.configure/init.configure", line 192, in virtualenv_python
>     manager.build(python)
>   File "/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 448, in build
>     result = self._log_process_output(args, cwd=self.topsrcdir)
>   File "/mozilla/python/mozbuild/mozbuild/virtualenv.py", line 174, in _log_process_output
>     stderr=subprocess.STDOUT, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
>     raise child_exception
> OSError: [Errno 2] No such file or directory
> *** Fix above errors and then restart with\
>                "make -f client.mk build"

Comment 4

[Tooru Fujisawa [:arai]]

do you have /out/_virtualenv/Scripts/python.exe and/or /out/_virtualenv/bin/python.exe ?

if you have /out/_virtualenv/bin/python.exe and not /out/_virtualenv/Scripts/python.exe , it could be the issue in the following lines:
https://dxr.mozilla.org/mozilla-central/rev/5ce3882eec21be3a70e4afc050959ca2f76bfa76/python/mozbuild/mozbuild/virtualenv.py#73-81

Comment 5

[saber-nyan]

I have only `/out/_virtualenv/bin/python` (not .exe!):
> $ ls -lh /out/_virtualenv/Scripts/python.exe
> ls: cannot access '/out/_virtualenv/Scripts/python.exe': No such file or directory
> 
> $ ls -lh /out/_virtualenv/bin/python.exe
> ls: cannot access '/out/_virtualenv/bin/python.exe': No such file or directory
> 
> $ ls -lh /out/_virtualenv/bin/python
> lrwxrwxrwx 1 rasva rasva 9 Jan 15 12:18 /out/_virtualenv/bin/python -> python2.7

Comment 6

[Tooru Fujisawa [:arai]]

okay, then that's the issue in those 2 functions
https://dxr.mozilla.org/mozilla-central/rev/5ce3882eec21be3a70e4afc050959ca2f76bfa76/python/mozbuild/mozbuild/virtualenv.py#72-89

can you check the value of IS_CYGWIN, IS_NATIVE_WIN and IS_MSYS2 there?
maybe, the function need to check IS_MSYS2 and then use non-windows specific path.

Comment 7

[Tooru Fujisawa [:arai]]

fwiw, those 3 variables are defined here, so it depends on sys.platform and os.sep
https://dxr.mozilla.org/mozilla-central/rev/5ce3882eec21be3a70e4afc050959ca2f76bfa76/python/mozbuild/mozbuild/virtualenv.py#19-21

Comment 8

[saber-nyan]

(In reply to Tooru Fujisawa [:arai] from comment #7)
> fwiw, those 3 variables are defined here, so it depends on sys.platform and
> os.sep
> https://dxr.mozilla.org/mozilla-central/rev/
> 5ce3882eec21be3a70e4afc050959ca2f76bfa76/python/mozbuild/mozbuild/virtualenv.
> py#19-21

I checked these three variables in python shell:
> $ python2.7.exe
> Python 2.7.12 (default, Oct 10 2016, 12:56:26)
> [GCC 5.4.0] on cygwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys, os
> >>> (sys.platform == 'win32' and os.sep == '\\') # IS_NATIVE_WIN
> False
> >>> (sys.platform == 'win32' and os.sep == '/') # IS_MSYS2
> False
> >>> (sys.platform == 'cygwin') # IS_CYGWIN
> True
> >>> sys.platform
> 'cygwin'
> >>> os.sep
> '/'

Comment 9

[Tooru Fujisawa [:arai]]

which shell are you using? (cygwin or mingw or msys or...)
also, what's the path of python2.7.exe you used?
it looks like the python is part of cygwin, not mingw

Comment 10

[saber-nyan]

I use CMD.EXE with following %PATH% variable:
> $ cat C:\cygwin64\mozset.bat
> @echo off
> set MOZ_TOOLS=C:\cygwin64\moztools
> set PATH=C:\MinGW\bin;C:\cygwin64\bin;%MOZ_TOOLS%\bin;%PATH%
> set HOME=C:\cygwin64
> set BUILD_OFFICIAL=1
> 
> $ echo %PATH%
> C:\MinGW\bin;C:\cygwin64\bin;C:\cygwin64\moztools\bin;C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu;C:\Program Files\ConEmu\ConEmu;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\system32\config\systemprofile\.dnx\bin;C:\Program Files\Microsoft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Git\cmd;C:\Users\rasva\AppData\Local\Microsoft\WindowsApps;
> 
> $ where python2
> C:\cygwin64\bin\python2

But in MinGW shell:
> $ python
> bash.exe": python: command not found
The manual ( https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Compiling_Mozilla_With_Mingw#Get_MinGW ) does not say to install python in MinGW. I have to do it?

Comment 11

[Gregory Szorc [:gps]]

The build system and the Python in MinGW don't exactly work well together. This was one of the more painful parts of getting the build system to work in msys2. We had to patch our vendored copy of virtualenv so it handled the MinGW Python properly. See bug 1275437 for the gory details.