982185 - "ERROR PROCESSING MOZBUILD FILE [...] The error appears to be part of the mozbuild.frontend.reader Python module itself!" with tweak to toolkit.mozbuild

Status: RESOLVED DUPLICATE of bug 981780

(Firefox Build System :: General, defect)

Description

[Daniel Holbert [:dholbert] (PTO until 2/24)]

Attachment: toolkit.mozbuild tweak (apply and build with --disable-webrtc in mozconfig)

STR:
 1. Add this to your .mozconfig:
ac_add_options --disable-webrtc

 2. Apply the included patch.
 3. Start a clobber build.

ACTUAL RESULTS: After all the configure stuff completes, during "Reticulating splines", this output:
{
 0:20.51 Reticulating splines...
 0:20.94 /scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/obj/_virtualenv/bin/python: can't open file './build/dir_exists.py': [Errno 2] No such file or directory
 0:20.94 Traceback (most recent call last):
 0:20.94   File "./config.status", line 955, in <module>
 0:20.95     config_status(**args)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/config_status.py", line 148, in config_status
 0:20.95     summary = the_backend.consume(definitions)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/backend/base.py", line 186, in consume
 0:20.95     for obj in objs:
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/frontend/emitter.py", line 99, in emit
 0:20.95     for out in output:
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/frontend/reader.py", line 726, in read_mozbuild
 0:20.95     raise bre
 0:20.95 mozbuild.frontend.reader.BuildReaderError: ==============================
 0:20.95 ERROR PROCESSING MOZBUILD FILE
 0:20.95 ==============================
 0:20.95 
 0:20.95 The error occurred while processing the following file:
 0:20.95 
 0:20.95     /scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/libyuv/moz.build
 0:20.95 
 0:20.95 The error appears to be part of the mozbuild.frontend.reader Python module itself! It is possible you have stumbled across a legitimate bug.
 0:20.95 
 0:20.95 Traceback (most recent call last):
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/frontend/reader.py", line 722, in read_mozbuild
 0:20.95     descend=descend, metadata=metadata):
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/frontend/reader.py", line 819, in _read_mozbuild
 0:20.95     non_unified_sources = non_unified_sources):
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/python/mozbuild/mozbuild/frontend/gyp_reader.py", line 111, in read_from_gyp
 0:20.95     params=params)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/__init__.py", line 132, in Load
 0:20.95     params['parallel'])
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 2560, in Load
 0:20.95     variables, includes, depth, check, True)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 391, in LoadTargetBuildFile
 0:20.95     build_file_data, PHASE_EARLY, variables, build_file_path)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 1142, in ProcessVariablesAndConditionsInDict
 0:20.95     variables, build_file, 'variables')
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 1149, in ProcessVariablesAndConditionsInDict
 0:20.95     expanded = ExpandVariables(value, phase, variables, build_file)
 0:20.95   File "/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/webrtc/trunk/tools/gyp/pylib/gyp/input.py", line 881, in ExpandVariables
 0:20.95     (contents, p.returncode))
 0:20.95 GypError: Call to '/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/obj/_virtualenv/bin/python ./build/dir_exists.py ./third_party/platformsdk_win7/files/redist/x86' returned exit status 2. while trying to load /scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/libyuv/libyuv.gyp
 0:20.98 *** Fix above errors and then restart with\
 0:20.98                "/usr/bin/make -f client.mk build"
 0:20.98 make[2]: *** [configure] Error 1
 0:20.98 make[1]: *** [/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/../obj/Makefile] Error 2
 0:20.98 make: *** [build] Error 2
}

I'm taking the output at its word ("It is possible you have stumbled across a legitimate bug.") and filing a build system bug.

(This seems particularly problematic: "can't open file './build/dir_exists.py'")

Comment 1

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(BTW, the attached patch [which makes us hit this build error] is an attempted partial fix for bug 981780.)

Comment 2

[Daniel Holbert [:dholbert] (PTO until 2/24)]

Note: I'm assuming there *is actually* a problem with the moz.build file. If that's the case, that issue should probably be fixed as part of bug 981780.

I filed this bug here to cover the the "The error appears to be part of the mozbuild.frontend.reader Python module itself" part of the output. (and/or to make the error output from issues like this a bit more readable)

Comment 3

[Benjamin Chen [:bechen]]

Hi Daniel:
Any update here? As your comment Bug 981780 comment 6, this bug should be fixed before Bug 981780.
And also I'm not familiar with the build system, please take Bug 981780 if you can.
Thanks for your help.

Comment 4

[Daniel Holbert [:dholbert] (PTO until 2/24)]

I'm not particularly familiar with the build system either. I just happened to have --disable-webrtc in my mozconfig and hence ran up against these bugs.

Comment 5

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(In reply to Benjamin Chen [:bechen] from comment #3)
> As your comment Bug 981780 comment 6, this bug should be
> fixed before Bug 981780.

Actually, that's not quite what I meant to suggest.  I think this bug is independent and probably doesn't need to be fixed first.

What I meant to say was, there are two distinct issues brought up by this build error:
 (A) It seems to be complaining about something that's broken in a moz.build file.

 (B) The specific complaint may also be indicative of a bug in mach or in the moz.build parser. (At best, the error output could be made to be more helpful)


Part (A) is probably covered by bug 981780 and will need to be fixed as part of that bug (if we can figure out what the problem is).
Part (B) is covered by this bug and doesn't necessarily need to be fixed first (depending on the nature of the bug).

Comment 6

[Markus Adrario [:Taraman]]

I have the same error since This week:

 3:12.62 ERROR PROCESSING MOZBUILD FILE
 3:12.62 ==============================
 3:12.62
 3:12.62 The error occurred while processing the following file:
 3:12.62
 3:12.62     d:/mozilla/comm-central/mozilla/media/libyuv/moz.build
 3:12.62
 3:12.62 The error appears to be part of the mozbuild.frontend.reader Python module itself! It is possible you have stumbled across a legitimate bug.

since this file has not been changed lately, I suspect the error to be somewhere else.

I'm building on Win XP using mozilla-build 1.7
removing --disable-webrtc does not help.

Comment 7

[Markus Adrario [:Taraman]]

Meanwhile I updated mozbuild to 1.9, but no help.

MSVC version is 10

Comment 8

[Daniel Holbert [:dholbert] (PTO until 2/24)]

:Taraman, my bet would be that your issue was introduced by bug 981780's fix. Can you try reverting that locally, to see?

Comment 9

[Markus Adrario [:Taraman]]

Reverting to rev 183046 changes the message to:

 3:40.67 The error occurred while processing the following file:
 3:40.67
 3:40.67     d:/mozilla/comm-central/mozilla/media/webrtc/moz.build

with "ac_add_options --disable-webrtc" in mozconfig the build proceeds past this point. If it succeeds I will see later.

So your guess was right.
Is there any workaround for this at the moment?

Comment 10

[Daniel Holbert [:dholbert] (PTO until 2/24)]

Not that I know of.

(So far, you're the first person I've heard of to hit this in an actual stock build. I personally only hit it when applying a local patch, the one I've attached here.)

gps, do you know what might be going on here, or can you suggest someone who might be able to take a look and/or suggest a workaround?

Comment 11

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(In reply to Markus Adrario [:Taraman] from comment #9)
> Reverting to rev 183046 changes the message to:

(Sorry, I meant "revert" in the sense of "back out", not in the sense of "revert to before". Not a big deal, though.  So, you've proven that bug 981780 *or something newer* caused this. For completeness' sake, could you also test revision 183047 -- that bug's patch -- to be sure that this did actually *start with* bug 981780's landing? [maybe you already did that])

Comment 12

[StevenLee[:slee]]

Hi all,

I grab the latest m-c. I can build on Win7 with msvc 10 and mozilla-build 1.9.

Comment 13

[StevenLee[:slee]]

:Taraman
Have you removed the obj folder and tried?

Comment 14

[Daniel Holbert [:dholbert] (PTO until 2/24)]

(In reply to StevenLee[:slee] from comment #12)
> I grab the latest m-c. I can build on Win7 with msvc 10 and mozilla-build 1.9.

(It looks like Taraman is building comm-central, not just mozilla-central; presumably that's at least somewhat involved in triggering this.)

Comment 15

[Markus Adrario [:Taraman]]

Yes, I did a clobber before each try and yes, I am building comm-central.

I tried to update the mozilla dir to the next revs and the Problem returns wit rev 183048.
If time permits, I will see at which pint in comm-central this starts to happen.

Comment 16

[Markus Adrario [:Taraman]]

I turned time back on c-c to rev 15800 without any success.
This would have been somewhere in February, where my builds were working perfectly.

So this problem must be within the interaction between c-c & m-c.
I wonder if someone else is seeing this.

I also tried to delete and rebuild the local dir.

Comment 17

[Gregory Szorc [:gps]]

The real error here is inside GYP. I'm guessing this is fallout from bug 778236.

glandium: does this ring a bell?

Comment 19

[Daniel Holbert [:dholbert] (PTO until 2/24)]

I don't think these are duplicates. Per comment 0, I explicitly spun this off for an unrelated "ERROR PROCESSING MOZBUILD FILE" issue that we can hit with a particular configuration (either by applying my patch here to an old build, or by trying to build comm-central w/ current trunk, according to :Taraman).

Hence un-duping. If you really do think this should be duped, please explain why.

Comment 20

[Mike Hommey [:glandium]]

You actual error, with --disable-webrtc, is:

0:20.95 GypError: Call to '/scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/obj/_virtualenv/bin/python ./build/dir_exists.py ./third_party/platformsdk_win7/files/redist/x86' returned exit status 2. while trying to load /scratch/work/builds/mozilla-central/mozilla-central.11-08-17.14-31/mozilla/media/libyuv/libyuv.gyp

Bug 981780 added media/libyuv/build/dir_exists.py as part of fixing --disable-webrtc. How less duplicate can that be?

Comment 21

[Daniel Holbert [:dholbert] (PTO until 2/24)]

OK, thanks for the explanation.

That makes :Taraman's build issue different & a bit more mysterious, since his build has that bug's fix (and really, the issue he's hitting *started* with that bug's fix).

So -- Taraman, could you spin off a new bug for the issue that you're hitting, with a bit more build output than what you've got in comment 6? (e.g. if you see anything like the "Traceback" in comment 0, it would probably be helpful to include that.)

Comment 22

[Markus Adrario [:Taraman]]

Sure:
Bug 1015570