814333 - webrtc makefile generation doesn't use $PYTHON

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

As I was setting up a chroot to do linux x86 builds, i got an interesting failure during configure:

generating WebRTC Makefiles...
Updating projects from gyp files...
/bin/sh: 1: python: not found
Traceback (most recent call last):
  File "/home/mh/mozilla-central/media/webrtc/trunk/build/gyp_chromium", line 175, in <module>
    sys.exit(gyp.main(args))
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/__init__.py", line 511, in main
    return gyp_main(args)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/__init__.py", line 494, in gyp_main
    options.circular_check)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/__init__.py", line 133, in Load
    depth, generator_input_info, check, circular_check)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 2378, in Load
    depth, check)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 384, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 975, in ProcessVariablesAndConditionsInDict
    variables, build_file, 'variables')
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 982, in ProcessVariablesAndConditionsInDict
    expanded = ExpandVariables(value, phase, variables, build_file)
  File "/home/mh/mozilla-central/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 714, in ExpandVariables
    (contents, p.returncode))
Exception: Call to 'python ./build/dir_exists.py ./third_party/platformsdk_win7/files/redist/x86' returned exit status 127. while trying to load /home/mh/mozilla-central/media/webrtc/trunk/peerconnection.gyp
configure: error: failed to generate WebRTC Makefiles

While earlier in configure, there was:
checking for python2.7... /usr/bin/python2.7

As it happens, i hadn't installed python, but python2.7, so i only had the python2.7 executable, and not the python executable (actually, wrapper).

This means gyp doesn't use whatever python it was run under.

Comment 1

[(not currently active) Ted Mielczarek]

This appears to just be something stupid in the GYP files themselves:
http://mxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/build/common.gypi#878

They're shelling out to raw `python`. I don't know if GYP has a variable for "the Python executable in use", but it would be straightforward to add that if it doesn't.

Comment 2

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> They're shelling out to raw `python`. I don't know if GYP has a variable for
> "the Python executable in use", but it would be straightforward to add that
> if it doesn't.

... or just use sys.executable.

Comment 3

[Markus Adrario [:Taraman]]

I'm seeing the same (or at least very similar) thing in Windows XP Sp3 with Mozilla Build 1.6.1 and VC++ 2008 & 2010
The output there is like this:

Updating projects from gyp files...
python: can't open file './build/dir_exists.py': [Errno 2] No such file or direc
tory
Traceback (most recent call last):
  File "d:/mozilla/comm-central/mozilla/media/webrtc/trunk/build/gyp_chromium",
line 175, in <module>
    sys.exit(gyp.main(args))
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\_
_init__.py", line 511, in main
    return gyp_main(args)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\_
_init__.py", line 494, in gyp_main
    options.circular_check)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\_
_init__.py", line 133, in Load
    depth, generator_input_info, check, circular_check)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\i
nput.py", line 2378, in Load
    depth, check)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\i
nput.py", line 384, in LoadTargetBuildFile
    build_file_data, PHASE_EARLY, variables, build_file_path)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\i
nput.py", line 975, in ProcessVariablesAndConditionsInDict
    variables, build_file, 'variables')
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\i
nput.py", line 982, in ProcessVariablesAndConditionsInDict
    expanded = ExpandVariables(value, phase, variables, build_file)
  File "d:\mozilla\comm-central\mozilla\media\webrtc\trunk\tools\gyp\pylib\gyp\i
nput.py", line 714, in ExpandVariables
    (contents, p.returncode))
Exception: Call to 'python ./build/dir_exists.py ./third_party/platformsdk_win7/
files/redist/x86' returned exit status 2. while trying to load d:\mozilla\comm-c
entral\mozilla\media\webrtc\trunk\peerconnection.gyp
configure: error: failed to generate WebRTC Makefiles
configure: error: /d/mozilla/comm-central/mozilla/configure failed for mozilla


The detection of Phyton is like this:

checking for python2.7... (cached) /c/mozilla-build/python/python2.7
Creating Python environment
New python executable in d:/mozilla/objdir-tb-release-cen/mozilla/_virtualenv\Sc
ripts\python2.7.exe
Also creating executable in d:/mozilla/objdir-tb-release-cen/mozilla/_virtualenv
\Scripts\python.exe

Comment 4

[Cellier fabien [:fcellier]]

Attachment: use env to set python executable in gypi

I don't know how you will handle it but this is a patch for people which compile firefox under archlinux. I use an env variable as a workaround.

Comment 5

[Landry Breuil (:gaston)]

(In reply to Cellier fabien [:fcellier] from comment #4)
> Created attachment 713345 [details] [diff] [review]
> use env to set python executable in gypi
> 
> I don't know how you will handle it but this is a patch for people which
> compile firefox under archlinux. I use an env variable as a workaround.

This would be nice if your solution actually worked :

On OpenBSD i dont have 'python' in the path, only python2.7 and PYTHON=/usr/local/bin/python2.7 set via .mozconfig, but there's still a chicken & egg problem here :
gyp: Call to 'python ./build/find_python.py' returned exit status 127

media/webrtc/trunk/build/common.gypi still calls 'python' to find the actual python...

the build-system creates a virtualenv, there's an env var for it too.. any way to fetch that info to gyp ?

Comment 6

[Mike Hommey [:glandium]]

The best would be to use sys.executable in python instead of relying on an environment variable.

Comment 7

[Landry Breuil (:gaston)]

(In reply to Mike Hommey [:glandium] from comment #6)
> The best would be to use sys.executable in python instead of relying on an
> environment variable.

sure, but gyp/gypi files dont seem to be real python code, and the <!(python constructs only seem to send commands to a shell.. so where should sys.executable go ?

Comment 8

[Mike Hommey [:glandium]]

(In reply to Landry Breuil (:gaston) from comment #7)
> (In reply to Mike Hommey [:glandium] from comment #6)
> > The best would be to use sys.executable in python instead of relying on an
> > environment variable.
> 
> sure, but gyp/gypi files dont seem to be real python code, and the <!(python
> constructs only seem to send commands to a shell.. so where should
> sys.executable go ?

Make python a predefined variable in gyp with an initial value of sys.executable.

Comment 9

[Landry Breuil (:gaston)]

Attachment: make gyp provide a PYTHON default variable initialized to sys.executable and reuse it in .gyp files instead of hardcording python

This patch allows me to configure webrtc on OpenBSD without a 'python' in the path/env - it takes PYTHON from sys.executable and adds it to default_variables, so that it can be reused in .gypi files. Apparently in gyp style, default variables are uppercased (see http://code.google.com/p/gyp/wiki/InputFormatReference#Predefined_Variables)

Now, to see how to properly upstream that to the gyp developers.

Comment 10

[(not currently active) Ted Mielczarek]

Comment on attachment 715989 [details] [diff] [review]
make gyp provide a PYTHON default variable initialized to sys.executable and reuse it in .gyp files instead of hardcording python

Review of attachment 715989 [details] [diff] [review]:
-----------------------------------------------------------------

This looks pretty sensible, but we should get it upstream as well.

Comment 11

[Landry Breuil (:gaston)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e27e1c6fe079

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/e27e1c6fe079