Last Comment Bug 723861 - xpidllex.py and xpidlyacc.py are missing in Gecko10.0 for both Win & Mac
: xpidllex.py and xpidlyacc.py are missing in Gecko10.0 for both Win & Mac
Status: RESOLVED FIXED
[qa-][read Comment 17 if you're hitti...
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Windows 7
: -- normal (vote)
: mozilla13
Assigned To: Mike Hommey [:glandium]
:
Mentors:
: 714138 722632 781525 (view as bug list)
Depends on: 729752
Blocks:
  Show dependency treegraph
 
Reported: 2012-02-03 02:35 PST by paddy
Modified: 2015-09-07 15:29 PDT (History)
15 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Ship xpidllex.py and xpidlyacc.py (904 bytes, patch)
2012-02-17 07:34 PST, Mike Hommey [:glandium]
ted: review+
lukasblakk+bugs: approval‑mozilla‑aurora+
lukasblakk+bugs: approval‑mozilla‑beta+
Details | Diff | Splinter Review

Description paddy 2012-02-03 02:35:37 PST
User Agent: Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11

Steps to reproduce:

I am trying to generate header and type lib using Gecko10.0(http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/10.0/sdk/xulrunner-10.0.en-US.win32.sdk.zip) for my XPCOM to support FF10.

I am trying following commands:
...\xulrunner-sdk\sdk\bin>python header.py ISample.idl -I..\..\idl -o ISample.h
...\xulrunner-sdk\sdk\bin>python typelib.py ISample.idl -I..\..\idl -o ISample.xpt


Actual results:

I am getting following errors:

...\xulrunner-sdk\sdk\bin>python header.py ISample.idl -I..\..\idl -o ISample.h
Traceback (most recent call last):
  File "header.py", line 497, in <module>
    p = xpidl.IDLParser(outputdir=options.cachedir)
  File "...\xulrunner-sdk\sdk\bin\xpidl.py", line 1453, in __init__
    optimize=1)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 1004, in lex
    lexobj.writetab(lextab,outputdir)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 175, in writetab
    filename = os.path.join(outputdir,basetabfilename)+".py"
  File "C:\Python25\lib\ntpath.py", line 90, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()

#################################################################################

...\xulrunner-sdk\sdk\bin>python typelib.py ISample.idl -I..\..\idl -o ISample.xpt
Traceback (most recent call last):
  File "typelib.py", line 313, in <module>
    p = xpidl.IDLParser(outputdir=options.cachedir)
  File "...\xulrunner-sdk\sdk\bin\xpidl.py", line 1453, in __init__
    optimize=1)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 1004, in lex
    lexobj.writetab(lextab,outputdir)
  File "...\xulrunner-sdk\sdk\bin\ply\lex.py", line 175, in writetab
    filename = os.path.join(outputdir,basetabfilename)+".py"
  File "C:\Python25\lib\ntpath.py", line 90, in join
    assert len(path) > 0
TypeError: object of type 'NoneType' has no len()


Expected results:

On analysis I have found the xpidllex.py and xpidlyacc.py are missing in Gecko10.0. I have copied then from Gecko9.0 to Gecko10.0 and tried to generate the header and type and they worked fine. Then I built my XPCOM with these header and typelib and XPCOM now works fine with FF10.

So following are my queries:

Why xpidllex.py and xpidlyacc.py are missing in Gecko10.0?
Is there any other way to generate header and typelib for Gecko10.0?
Is use of xpidllex.py and xpidlyacc.py from Gecko9.0 in Gecko10.0 harmful?

Please suggest.
Comment 1 Ed Morley [:emorley] 2012-02-03 05:56:46 PST
As I understand it, bug 691781 removed the checked in copies, they're now generated as part of the build.
Comment 2 Kyle Huey [:khuey] (khuey@mozilla.com) 2012-02-03 06:03:36 PST
Using the xpidllex.py and xpidlyacc.py from Gecko 9 will work fine as a workaround.
Comment 3 Ted Mielczarek [:ted.mielczarek] 2012-02-03 07:31:23 PST
Wasn't there another bug on this (or was that a mailing list post)?
Comment 4 Kyle Huey [:khuey] (khuey@mozilla.com) 2012-02-03 07:36:20 PST
Yeah, I think so, but I can't find it.
Comment 5 Benjamin Smedberg AWAY UNTIL 2-AUG-2016 [:bsmedberg] 2012-02-03 08:15:51 PST
Bug 722632, I don't know which way to dup this.
Comment 6 Kyle Huey [:khuey] (khuey@mozilla.com) 2012-02-03 08:21:17 PST
*** Bug 722632 has been marked as a duplicate of this bug. ***
Comment 7 paddy 2012-02-13 00:50:05 PST
Please let me know resolution on this issue. 

xpidllex.py and xpidlyacc.py are missing in Gecko11.0 too.
Comment 8 Ted Mielczarek [:ted.mielczarek] 2012-02-17 06:57:05 PST
*** Bug 714138 has been marked as a duplicate of this bug. ***
Comment 9 Mike Hommey [:glandium] 2012-02-17 07:34:29 PST
Created attachment 598235 [details] [diff] [review]
Ship xpidllex.py and xpidlyacc.py
Comment 11 Mike Hommey [:glandium] 2012-02-21 23:39:51 PST
Comment on attachment 598235 [details] [diff] [review]
Ship xpidllex.py and xpidlyacc.py

[Approval Request Comment]
This prevents the SDK from being completely useful (though this can be worked around by giving a --cachedir to the header.py and typelib.py scripts). This only ships two additional files in the SDK and changes nothing else.
Risk to taking this patch (and alternatives if risky): none
String changes made by this patch: none

I'd request approval for esr, but the flag doesn't show up.
Comment 12 Jonathan Kew (:jfkthame) 2012-02-22 04:12:33 PST
Since this landed, my local -inbound tree fails to build on OS X (10.6), with the error:

make[7]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
/Users/jonathan/mozdev/mozilla-inbound/obj-ff/config/nsinstall -D ../../../dist/sdk/bin/ply
/Users/jonathan/mozdev/mozilla-inbound/obj-ff/config/nsinstall -L /Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl -m 755 /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/__init__.py /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/lex.py /Users/jonathan/mozdev/mozilla-inbound/other-licenses/ply/ply/yacc.py ../../../dist/sdk/bin/ply
make[8]: Entering directory `/Users/jonathan/mozdev/mozilla-inbound/obj-ff/xpcom/typelib/xpidl'
make[8]: *** No rule to make target `../../../xpcom/idl-parser/xpidllex.py', needed by `libs'.  Stop.

Obviously, this isn't affecting tinderbox, as tbpl shows happy builds ... I don't know why it's failing locally for me.
Comment 13 Josh Aas 2012-02-22 06:23:19 PST
I have the same build failure on Mac OS X 10.7.
Comment 14 Mike Hommey [:glandium] 2012-02-22 07:37:51 PST
So, after investigation, Josh and Jonathan's problem was leftovers from times when xpidllex.py and xpidlyacc.py were created in the srcdir instead of the objdir.
Comment 15 Stefan 2012-02-22 09:37:41 PST
(In reply to Mike Hommey [:glandium] from comment #14)
> So, after investigation, Josh and Jonathan's problem was leftovers from
> times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> of the objdir.

Sure? I get the error of Comment #12 even after hg co -C && hg purge. In order to compile m-c I had to backout 013b99c7ae65.
Comment 16 Mike Hommey [:glandium] 2012-02-22 09:43:54 PST
(In reply to Stefan from comment #15)
> (In reply to Mike Hommey [:glandium] from comment #14)
> > So, after investigation, Josh and Jonathan's problem was leftovers from
> > times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> > of the objdir.
> 
> Sure? I get the error of Comment #12 even after hg co -C && hg purge. In
> order to compile m-c I had to backout 013b99c7ae65.

rm $objdir/xpcom/idl-parser/*.pyc ; they are hgignored, and hg purge doesn't remove them.
Comment 17 Mike Hommey [:glandium] 2012-02-22 09:44:22 PST
(In reply to Mike Hommey [:glandium] from comment #16)
> rm $objdir/xpcom/idl-parser/*.pyc ; they are hgignored, and hg purge doesn't
> remove them.

erm, that's rm $srcdir/xpcom/idl-parser/*.pyc
Comment 18 Ed Morley [:emorley] 2012-02-22 10:25:50 PST
https://hg.mozilla.org/mozilla-central/rev/013b99c7ae65
Comment 19 Alex Keybl [:akeybl] 2012-02-23 15:13:09 PST
Please see https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for information on landing on mozilla-esr10 before 3/2.
Comment 21 Lukas Blakk [:lsblakk] use ?needinfo 2012-02-27 10:50:02 PST
[Triage comment]

Hi Mike, this bug is being tracked for landing on ESR branch.  Please land patches on http://hg.mozilla.org/releases/mozilla-esr10/ before Thursday March 1, 2012 in order to be ready for go-to-build on Friday March 2, 2012.
Comment 22 Mike Hommey [:glandium] 2012-02-28 00:04:23 PST
http://hg.mozilla.org/releases/mozilla-esr10/rev/52637db4f742
Comment 23 Robert O'Callahan (:roc) (Exited; email my personal email if necessary) 2012-02-28 05:50:22 PST
(In reply to Mike Hommey [:glandium] from comment #14)
> So, after investigation, Josh and Jonathan's problem was leftovers from
> times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> of the objdir.

Could we add an rm command somewhere during configure or make so that this automatically gets fixed?
Comment 24 Jonathan Kew (:jfkthame) 2012-02-28 07:03:23 PST
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) (away February 21 to March 17) from comment #23)
> (In reply to Mike Hommey [:glandium] from comment #14)
> > So, after investigation, Josh and Jonathan's problem was leftovers from
> > times when xpidllex.py and xpidlyacc.py were created in the srcdir instead
> > of the objdir.
> 
> Could we add an rm command somewhere during configure or make so that this
> automatically gets fixed?

In principle, I don't think configure or make should be modifying the source tree in any way.
Comment 25 Robert O'Callahan (:roc) (Exited; email my personal email if necessary) 2012-02-28 07:11:55 PST
Sure, but in this case they'd just be undoing a modification that shouldn't have happened in the first place.
Comment 26 Stefan 2012-02-28 07:24:33 PST
(In reply to Jonathan Kew (:jfkthame) from comment #24)
> In principle, I don't think configure or make should be modifying the source
> tree in any way.

Sure, if you build outside of the source tree. But building outside the source tree (on which I removed the write permission) fails:

/bin/sh: ../m-c/.mozconfig.out: Permission denied

client.mk:

    136 run_for_side_effects := \
    137   $(shell $(TOPSRCDIR)/$(MOZCONFIG_LOADER) $(TOPSRCDIR) $(TOPSRCDIR)/.mo        zconfig.mk > $(TOPSRCDIR)/.mozconfig.out)
                                   ^
Comment 27 ISHIKAWA, Chiaki 2012-02-28 22:06:49 PST
Can someone possibly re-open this bug by changing the bug status?
I got bitten by the same bug (I think), but since its status was fixed, it didn't show up on the first simple search.

I my case, I had to install system's python-ply (lex/yacc tool) first, and run make clean and then recompile everything. I am not sure why, but both the new python tools and make clean was necessary on my Debian [wheezy] system.)

Looks to me that these are all related:

691836 	nor 	-- 	Linu 	nobody 	NEW 	--- 	Build system doesn't correctly
reparse IDL files when the xpidl parser has changed

729752 	nor 	-- 	All 	nobody 	NEW 	--- 	Old .pyc files in srcdir shouldn't
break the build
and this one.
Comment 28 Mike Hommey [:glandium] 2012-02-29 03:25:53 PST
For what it's worth, "make clean" removes the problematic .pyc files from the source directory because of http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/Makefile.in#250
Comment 29 Andreas Gal :gal 2012-04-07 18:56:30 PDT
comment 28: I ran into this bug, didn't think of make clean since I blew away my obj dir but it was still breaking. We should auto-clean these files when building.
Comment 30 :Ms2ger 2012-08-10 23:49:59 PDT
*** Bug 781525 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.