800847 - mozmake.py doesn't create working dependencies of Makefiles on gyp files

Status: RESOLVED FIXED

(Core :: WebRTC, defect)

Description

[Randell Jesup [:jesup] (needinfo me)]

[I could swear there's an open bug on this, but can't find it.  This appears to be causing frequent need to clobber inbound.  It's unclear why, as gyp files shouldn't be changing there; it may be somehow related to the dep builds or some incomplete clobbers after a gyp change.]

Dependencies of Makefiles on gypfiles is broken; you typically (during export) see:
https://tbpl.mozilla.org/php/getParsedLog.php?id=16012230&full=1&branch=mozilla-inbound

    ProcessConditionsInDict(the_dict, phase, variables, build_file)
  File "/builds/slave/m-in-osx64/build/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 885, in ProcessConditionsInDict
    if eval(ast_code, {'__builtins__': None}, variables):
  File "<string>", line 1, in <module>
NameError: name 'use_system_libvpx' is not defined while evaluating condition 'use_system_libvpx==0' in /builds/slave/m-in-osx64/build/media/webrtc/trunk/src/modules/video_coding/codecs/vp8/vp8.gyp while loading dependencies of /builds/slave/m-in-osx64/build/media/webrtc/trunk/src/modules/modules.gyp while loading dependencies of /builds/slave/m-in-osx64/build/media/webrtc/trunk/peerconnection.gyp while trying to load /builds/slave/m-in-osx64/build/media/webrtc/trunk/peerconnection.gyp

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Correctly regenerate Makefiles from gyp files

Note: includes temporary fix for bug 798969 from the patch there

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 670735 [details] [diff] [review]
Correctly regenerate Makefiles from gyp files

Ok, it may not be the only problem (or maybe it is), but a primary issue is that common.mk generated by mozmake.py is missing options specified in configure.in, in particular --include ${srcdir}/media/webrtc/webrtc_config.gypi

This passes the --include through a -D value as well, and thus also ensures it survives in the common.mk file and doesn't disappear when common.mk regenerates the makefiles (including itself).

In testing it appears to both correctly trigger re-runs of gyp, but also common.mk doesn't change after a re-run.

As mentioned, this has the hacky solution for the path issue to find gyp (bug 798969)

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

I'd also add a comment to configure.in that some changes to gyp invocation will require modifying mozmake.py/common.mk as well.

Comment 5

[(not currently active) Ted Mielczarek]

Related: bug 800546

Comment 6

[(not currently active) Ted Mielczarek]

Comment on attachment 670735 [details] [diff] [review]
Correctly regenerate Makefiles from gyp files

Review of attachment 670735 [details] [diff] [review]:
-----------------------------------------------------------------

This is gross, but let's unbreak things. Maybe we can add that info to gyp and get it upstreamed to do this a clean way.

Comment 7

[Tanvi Vyas[:tanvi]]

.mozconfig - http://pastebin.mozilla.org/1865284.  

hg log -l 5 - http://pastebin.mozilla.org/1865282.  

Errors after a clobber - http://pastebin.mozilla.org/1865281

Using Xcode 4.5.1 with Command Line tools installed on Mac OS X 10.7.5

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5a5563b58d5b

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/5a5563b58d5b

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

ted: should we uplift this to Aurora to keep (re)builds there sane?

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 670735 [details] [diff] [review]
Correctly regenerate Makefiles from gyp files

[Approval Request Comment]
Bug caused by (feature/regressing bug #): webrtc
User impact if declined: Touching gyp files or anything else that triggers a gyp dependency build (and something on builders occasionally does without a gyp change it appears) will leave the build busted until a clobber is done.  This was causing frequent clobbers on m-i/m-c, and had messed up some of the windows buildsers.
Testing completed (on m-c, etc.): Stable on m-c since last Friday
Risk to taking this patch (and alternatives if risky): No risk to build, only an developer/infrastructure issue.
String or UUID changes made by this patch: None

Comment 12

[(not currently active) Ted Mielczarek]

Yes, I think so.

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/3bb7ee11c85c