723861 - xpidllex.py and xpidlyacc.py are missing in Gecko10.0 for both Win & Mac

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[paddy]

User Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11

Steps to reproduce:

I am trying to generate header and type lib using Gecko10.0(http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/10.0/sdk/xulrunner-10.0.en-US.win32.sdk.zip) for my XPCOM to support FF10.

I am trying following commands:
...\xulrunner-sdk\sdk\bin>python header.py ISample.idl -I..\..\idl -o ISample.h
...\xulrunner-sdk\sdk\bin>python typelib.py ISample.idl -I..\..\idl -o ISample.xpt


Actual results:

I am getting following errors:

...\xulrunner-sdk\sdk\bin>python header.py ISample.idl -I..\..\idl -o ISample.h
Traceback (most recent call last):
  File "header.py", line 497, in <module>
    p = xpidl.IDLParser(outputdir=options.cachedir)
  File "...\xulrunner-sdk\sdk\bin\xpidl.py", line 1453, in __init__
    optimize=1)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 1004, in lex
    lexobj.writetab(lextab,outputdir)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 175, in writetab
    filename = os.path.join(outputdir,basetabfilename)+".py"
  File "C:\Python25\lib\ntpath.py", line 90, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()

#################################################################################

...\xulrunner-sdk\sdk\bin>python typelib.py ISample.idl -I..\..\idl -o ISample.xpt
Traceback (most recent call last):
  File "typelib.py", line 313, in <module>
    p = xpidl.IDLParser(outputdir=options.cachedir)
  File "...\xulrunner-sdk\sdk\bin\xpidl.py", line 1453, in __init__
    optimize=1)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 1004, in lex
    lexobj.writetab(lextab,outputdir)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 175, in writetab
    filename = os.path.join(outputdir,basetabfilename)+".py"
  File "C:\Python25\lib\ntpath.py", line 90, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()


Expected results:

On analysis I have found the xpidllex.py and xpidlyacc.py are missing in Gecko10.0. I have copied then from Gecko9.0 to Gecko10.0 and tried to generate the header and type and they worked fine. Then I built my XPCOM with these header and typelib and XPCOM now works fine with FF10.

So following are my queries:

Why xpidllex.py and xpidlyacc.py are missing in Gecko10.0?
Is there any other way to generate header and typelib for Gecko10.0?
Is use of xpidllex.py and xpidlyacc.py from Gecko9.0 in Gecko10.0 harmful?

Please suggest.

Comment 1

[Ed Morley [:emorley]]

As I understand it, bug 691781 removed the checked in copies, they're now generated as part of the build.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Using the xpidllex.py and xpidlyacc.py from Gecko 9 will work fine as a workaround.

Comment 3

[(not currently active) Ted Mielczarek]

Wasn't there another bug on this (or was that a mailing list post)?

Comment 4

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Yeah, I think so, but I can't find it.

Comment 5

[Benjamin Smedberg]

Bug 722632, I don't know which way to dup this.

Comment 7

[paddy]

Please let me know resolution on this issue. 

xpidllex.py and xpidlyacc.py are missing in Gecko11.0 too.

Comment 9

[Mike Hommey [:glandium]]

Attachment: Ship xpidllex.py and xpidlyacc.py

Comment 10

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/013b99c7ae65

Comment 11

[Mike Hommey [:glandium]]

Comment on attachment 598235 [details] [diff] [review]
Ship xpidllex.py and xpidlyacc.py

[Approval Request Comment]
This prevents the SDK from being completely useful (though this can be worked around by giving a --cachedir to the header.py and typelib.py scripts). This only ships two additional files in the SDK and changes nothing else.
Risk to taking this patch (and alternatives if risky): none
String changes made by this patch: none

I'd request approval for esr, but the flag doesn't show up.

Comment 12

[Jonathan Kew [:jfkthame]]

Since this landed, my local -inbound tree fails to build on OS X (10.6), with the error:

make[7]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
/Users/jonathan/mozdev/mozilla-inbound/obj-ff/config/nsinstall -D ../../../dist/sdk/bin/ply
/Users/jonathan/mozdev/mozilla-inbound/obj-ff/config/nsinstall -L /Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl -m 755 /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/__init__.py /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/lex.py /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/yacc.py ../../../dist/sdk/bin/ply
make[8]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.

Obviously, this isn't affecting tinderbox, as tbpl shows happy builds ... I don't know why it's failing locally for me.

Comment 13

[Josh Aas]

I have the same build failure on Mac OS X 10.7.

Comment 14

[Mike Hommey [:glandium]]

So, after investigation, Josh and Jonathan's problem was leftovers from times when xpidllex.py and xpidlyacc.py were created in the srcdir instead of the objdir.

Comment 15

[Stefan]

(In reply to Mike Hommey [:glandium] from comment #14)
> So, after investigation, Josh and Jonathan's problem was leftovers from
> times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> of the objdir.

Sure? I get the error of Comment #12 even after hg co -C && hg purge. In order to compile m-c I had to backout 013b99c7ae65.

Comment 16

[Mike Hommey [:glandium]]

(In reply to Stefan from comment #15)
> (In reply to Mike Hommey [:glandium] from comment #14)
> > So, after investigation, Josh and Jonathan's problem was leftovers from
> > times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> > of the objdir.
> 
> Sure? I get the error of Comment #12 even after hg co -C && hg purge. In
> order to compile m-c I had to backout 013b99c7ae65.

rm $objdir/xpcom/idl-parser/*.pyc ; they are hgignored, and hg purge doesn't remove them.

Comment 17

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #16)
> rm $objdir/xpcom/idl-parser/*.pyc ; they are hgignored, and hg purge doesn't
> remove them.

erm, that's rm $srcdir/xpcom/idl-parser/*.pyc

Comment 18

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/013b99c7ae65

Comment 19

[Alex Keybl [:akeybl]]

Please see https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for information on landing on mozilla-esr10 before 3/2.

Comment 20

[Mike Hommey [:glandium]]

http://hg.mozilla.org/releases/mozilla-aurora/rev/1f89179f19d7
http://hg.mozilla.org/releases/mozilla-beta/rev/76add95f5ce4

Also landed on the COMM110_2012022204_RELBRANCH by mistake :(
http://hg.mozilla.org/releases/mozilla-beta/rev/5738e8b0a963

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

[Triage comment]

Hi Mike, this bug is being tracked for landing on ESR branch.  Please land patches on http://hg.mozilla.org/releases/mozilla-esr10/ before Thursday March 1, 2012 in order to be ready for go-to-build on Friday March 2, 2012.

Comment 22

[Mike Hommey [:glandium]]

http://hg.mozilla.org/releases/mozilla-esr10/rev/52637db4f742

Comment 23

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Mike Hommey [:glandium] from comment #14)
> So, after investigation, Josh and Jonathan's problem was leftovers from
> times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> of the objdir.

Could we add an rm command somewhere during configure or make so that this automatically gets fixed?

Comment 24

[Jonathan Kew [:jfkthame]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) (away February 21 to March 17) from comment #23)
> (In reply to Mike Hommey [:glandium] from comment #14)
> > So, after investigation, Josh and Jonathan's problem was leftovers from
> > times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> > of the objdir.
> 
> Could we add an rm command somewhere during configure or make so that this
> automatically gets fixed?

In principle, I don't think configure or make should be modifying the source tree in any way.

Comment 25

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Sure, but in this case they'd just be undoing a modification that shouldn't have happened in the first place.

Comment 26

[Stefan]

(In reply to Jonathan Kew (:jfkthame) from comment #24)
> In principle, I don't think configure or make should be modifying the source
> tree in any way.

Sure, if you build outside of the source tree. But building outside the source tree (on which I removed the write permission) fails:

/bin/sh: ../m-c/.mozconfig.out: Permission denied

client.mk:

    136 run_for_side_effects := \
    137   $(shell $(TOPSRCDIR)/$(MOZCONFIG_LOADER) $(TOPSRCDIR) $(TOPSRCDIR)/.mo        zconfig.mk > $(TOPSRCDIR)/.mozconfig.out)
                                   ^

Comment 27

[ISHIKAWA, Chiaki]

Can someone possibly re-open this bug by changing the bug status?
I got bitten by the same bug (I think), but since its status was fixed, it didn't show up on the first simple search.

I my case, I had to install system's python-ply (lex/yacc tool) first, and run make clean and then recompile everything. I am not sure why, but both the new python tools and make clean was necessary on my Debian [wheezy] system.)

Looks to me that these are all related:

691836 	nor 	-- 	Linu 	nobody 	NEW 	--- 	Build system doesn't correctly
reparse IDL files when the xpidl parser has changed

729752 	nor 	-- 	All 	nobody 	NEW 	--- 	Old .pyc files in srcdir shouldn't
break the build
and this one.

Comment 28

[Mike Hommey [:glandium]]

For what it's worth, "make clean" removes the problematic .pyc files from the source directory because of http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/Makefile.in#250

Comment 29

[Andreas Gal :gal]

comment 28: I ran into this bug, didn't think of make clean since I blew away my obj dir but it was still breaking. We should auto-clean these files when building.