829793 - IDL generation busted after bug 826606

Status: RESOLVED INCOMPLETE

(Firefox Build System :: General, defect)

Description

[Karsten Düsterloh]

Building current SeaMonkey trunk under Linux 64 bit with external linkage fails when generating IDLs:

nsIDOMUserProximityEvent.idl
Traceback (most recent call last):
  File "/home/kd/projects/mozilla/mozilla.org/src/trunk/mozilla/config/pythonpath.py", line 56, in <module>
    main(sys.argv[1:])
  File "/home/kd/projects/mozilla/mozilla.org/src/trunk/mozilla/config/pythonpath.py", line 48, in main
    execfile(script, frozenglobals)
  File "/home/kd/moz/obj/sr/mozilla/dist/sdk/bin/header.py", line 523, in <module>
    idl = p.parse(open(file).read(), filename=file)
  File "/home/kd/moz/obj/sr/mozilla/dist/sdk/bin/xpidl.py", line 1507, in parse
    idl = self.parser.parse(lexer=self)
  File "/home/kd/moz/obj/sr/mozilla/dist/sdk/bin/ply/yacc.py", line 265, in parse
    return self.parseopt_notrack(input,lexer,debug,tracking,tokenfunc)
  File "/home/kd/moz/obj/sr/mozilla/dist/sdk/bin/ply/yacc.py", line 1047, in parseopt_notrack
    tok = self.errorfunc(errtoken)
  File "/home/kd/moz/obj/sr/mozilla/dist/sdk/bin/xpidl.py", line 1478, in p_error
    raise IDLError("invalid syntax", location)
xpidl.IDLError: error: invalid syntax, /home/kd/projects/mozilla/mozilla.org/src/trunk/mozilla/dom/interfaces/events/nsIDOMDeviceProximityEvent.idl line 24:18
   double value = Infinity;
                  ^

Comment 1

[Karsten Düsterloh]

Non-external linkage build fail as well.
Clobbering doesn't help either.

Comment 2

[[e-mail]]

Might be old .pyc files in your mozilla srcdir. Try hg st -in

Comment 3

[Karsten Düsterloh]

I removed all .pyc files from my tree before, it didn't help.

Comment 4

[Landry Breuil (:gaston)]

I have the exact same issue on my buildbot - when trying to build tb from c-c. Removing objdir & all .pyc files from the src tree didnt help.

See http://buildbot.rhaalovely.net/builders/comm-central-amd64 for the logs. Last successful nightly build was on the 7 with rev a976311210e4e, so the breakage window is the same.

Comment 5

[Karsten Düsterloh]

JDTR: cleaning out/turning off ccache doesn't help either.

Comment 6

[Karsten Düsterloh]

So, it's not even an comm-central problem at all, since I fail to build FF with the very same error trace …
- Linux 64 bit on an AMD Athlon X2 double core
- Python 2.7.3

Comment 7

[Karsten Düsterloh]

FWIW, I don't see the problem on OS X 10.6.8 with clang 3.1 and Python 2.6.1(?!).
I'm using gcc 4.7.2 on Linux.
Both Linux and Mac are debug builds.

Comment 8

[William Chen [:wchen]]

I'm not sure why this is happening. My guess is that it has something to do with yacc. My only idea right now is that there are some yacc/lex cache files somewhere in your tree. Try removing any files matching xpidllex.py* and xpidlyacc.py*.

Comment 9

[Ilja Sekler]

(In reply to William Chen [:wchen] from comment #8)

> [...] Try removing any files matching xpidllex.py*
> and xpidlyacc.py*.

This indeed fixes the build.

Fedora 17 x86_64, gcc 4.7.2, python 2.7.3.

Comment 10

[William Chen [:wchen]]

(In reply to Ilja Sekler from comment #9)
> (In reply to William Chen [:wchen] from comment #8)
> 
> > [...] Try removing any files matching xpidllex.py*
> > and xpidlyacc.py*.
> 
> This indeed fixes the build.
> 
> Fedora 17 x86_64, gcc 4.7.2, python 2.7.3.
Do you remember where these files were located in your tree? I'm suspect that there is a problem in the build script. On my Mac and Linux machines, these cache files are located in the objdir so clobbering should have fixed the problem and it looks like there is some code to delete these files. Something is causing these old cache files to stick around.

I'm not familiar with the build system but I'm guessing the problem is around here:
http://dxr.mozilla.org/mozilla-central/xpcom/idl-parser/Makefile.in.html#l32
http://dxr.mozilla.org/mozilla-central/xpcom/idl-parser/header.py.html#l502

Comment 11

[Landry Breuil (:gaston)]

The list of files to remove in addition to the objdir might be those in $srcdir :

xpcom/idl-parser/xpidl.pyc
xpcom/idl-parser/xpidllex.py{,c}
xpcom/idl-parser/xpidlyacc.py{,c}

They're not under version control/hg but strangely hg status doesnt show them as untracked - although only .pyc files are ignored by .hgignore.. not the two xpidl{lex,yacc}.py files.

Comment 12

[Karsten Düsterloh]

Okay, blowing away xpidllex.py/xpidlyacc.py as well lets me build again (I already deleted objdir and *.pyc in my source tree before, but to no avail).
The cache were located where gaston pointed out.

Comment 13

[Karsten Düsterloh]

(In reply to Karsten Düsterloh from comment #12)
> The cache were located where gaston pointed out.

To clarify: I had two sets of these files, one set in $objdir/mozilla/dist/sdk/bin/, one set in $source/mozilla/xpcom/idl-parser/. The .py files in the objdir are symlinked to their source tree equivalents, the .pyc caches were local to both directories (and diffrent due to different paths and such included).

Comment 14

[[e-mail]]

OK, so apparently between 2.15 and 2.16 the generation of the xpidl*.py* was moved from the srcdir to the objdir, so if you start with a clean 2.16 checkout you won't get the problem, but if you pull a tree previously used to build 2.15 or earlier then you will have those stale files and you need to remove them so that the versions in the objdir get used.

Comment 15

[Jens Hatlak (:InvisibleSmiley)]

Just hit this one on Windows (Win 7 x64) trying to build SM trunk. Since even before that my build failed due to an old mozilla/build/mozinfo.pyc, I'll now bite the bullet and have each and every untracked file deleted before building. I don't understand why anything is created/modified outside the objdir at all, but since this is about the third time that I cannot build due to some old pyc's lying around and seemingly no attempt was made to prevent this, I'll now take care of this myself. Sad but true. :-(

Neil: Thanks for the hint in comment 2. Will use that command for rm list identification purposes.

Comment 16

[Gregory Szorc [:gps]]

Fixing the pyc problem requires writing a custom Python importer and is tracked in bug 795995. This is a hard problem, which is why it hasn't been implemented yet. While the problem is annoying, things only break maybe once every few months. There are much bigger fish to fry.

Comment 17

[Jesse Ruderman]

> things only break maybe once every few months

And whenever anyone tries to bisect.

Comment 18

[Gregory Szorc [:gps]]

We haven't seen these problems recently.