98811 - Remove GPL and LGPLed libIDL code from tree

Status: VERIFIED FIXED

(Core :: XPCOM, defect, P4)

Description

[Gervase Markham [:gerv]]

As part of a licensing cleanup, we would like to remove the GPLed and LGPLed
libIDL stuff from the tree and just make it an external dependency on Mac, like
it is on Linux.

Gerv

Comment 1

[John Bandhauer]

Patrick will know better, but AFAIK the libIDL changes needed by Mac never got 
merged back into the 'real' libIDL. We either need to figure out how to make 
that happen - and get a new round of distributions for Mac made available, 
distributed, and installed. Or, we need to set up hosting for the bits 
ourselves.

An alternative that was discussed back in the day was to 'simply' fork libIDL 
and maintain within mozilla the code that will work for all platforms. This 
looked especially inviting when we contemplated making significant changes to 
libIDL for our own purposes. Last I heard the ownership of libIDL was doubtful. 
But that was some time ago. A complication here was the fact that the whole 
thing did not seem to be natively buildable on Win32. Apparently the owner was 
doing lex/yacc code generation stuff on Linux and then feeding that output into 
the Win32 build to make the binary distributions.

This whole mess had mccabe stalled out on making any changes to the scheme. 
Maybe it is now time to resolve this.

Comment 2

[David Bradley]

I think we'd gain a good bit if we could build from the .y and .l files. This
would allow us to address certain errors and states that we currently cannot
detect just using the back-end modules. Take the orphan #endif  issue that came
up recently as an example. 

I did a little investigating just to figure out where we stand currently.
Basically I'm able to generate and compile the files using flex and bison on
Win32.  The build built ok, but I encountered an assert with the typeindex being
negative. So there is still some work and testing that would need to be done.

Comment 3

[David Bradley]

Quick update, I think xpidl.exe didn't get recompiled and caused the probem with
typeIndex. Everything seems to be working fine now.

Comment 4

[Gervase Markham [:gerv]]

So, I'm slightly confused. dbradley: what's the status here?

Comment 5

[David Bradley]

Well we have two options.

1. Is the "easy" method remove the libIDL stuff and make it an external
dependency for the Mac.
2. Fork libIDL and make it part of the build. I've been able to build it on
Win32 using the .y and .l files. This has an additional advantage to allowing us
to make changes to the grammar not just the back-ends. 

Option 2 is probably the best long-term solution, if we can afford the wait.
I'll need someone to back me up on the Mac. I can do the Windows and Linux build
mods and test, but I don't have a Mac. I'm thinking we're probably looking at a
week and a half to get it in. I've got the basics done for Windows, I'd need to
finish it up and then do the same for Linux. Then get all the reviews.

If you need something quick then option 1 is probably the best solution.

Comment 6

[Mike Shaver (:shaver emeritus)]

I like option 1 a lot better.  It's what we do on _all_ other platforms, and I
don't think we've been constrained by the inability to alter the grammar.

Also, I don't like needing bison/yacc as part of the build, which I don't
believe we currently do.

Comment 7

[David Bradley]

The security/PSM module appears to use bison/flex ->
mozilla\security\nss\cmd\modutil\installparse.y

(Didn't quickly see any optional build of this, other than the normal PSM switch).

So that just leaves the potential benefit of modifying the grammar or the .y
file. I'm more concerned with changing the code inside the grammar than with the
grammar itself. There are probably just a handful of bugs that could be adressed
by changing the .y file.

I'm fine with either option. I can't say I see an immediate large payoff for
bringing libIDL into our tree. So less work would seem to be the overriding
advantage of option 1.

To get 1. done, we just need someone with a Mac to make the project adjustments
and then make available a version of the library. 

Comment 8

[David Bradley]

Bugs that could be addressed by forking libIDL - bug 103495, bug 13424, maybe
bug 94650. I just wanted to get these recorded here.

Comment 9

[Mike Shaver (:shaver emeritus)]

If we fork libIDL, we really want to get it under our triple license.  I don't
think that the GNOME project is still actively developing libIDL, so I could
carry some water to them and see if the authors are willing to do that.

Comment 10

[David Bradley]

Moving to 0.9.8

Comment 11

[David Bradley]

Moving to 0.9.9

Comment 12

[David Bradley]

Attachment: Removes the Mac specific libIDL from the build

This is my first time modifying a Mac XML project file. I think this is
correct, but I need someone to test it on the Mac. My intent with the patch is
to remove the libidl project from the build and to add in references to a
prebuilt library. I'll need to get a build engineer to give me a hand in
getting a prebuilt libIDL in the proper place.

Comment 13

[David Bradley]

Retargetting. This may get moved back after reassessing, and is currently under
evaluation for MachV

Comment 14

[Gervase Markham [:gerv]]

As input to that reevaluation, please be aware that sorting out licensing issues
is a top priority for mozilla.org and we'd very much like to have this done
before 1.0.

Gerv

Comment 15

[hacker formerly known as [e-mail]]

> The security/PSM module appears to use bison/flex ->
> mozilla\security\nss\cmd\modutil\installparse.y

We don't build any of the command line utilities of NSS.  We just build the
libraries.

I'm all for removing the 3rd party libraries from the tree. 

Comment 16

[Gervase Markham [:gerv]]

cls: how do we get to a stage where we can do that?

Gerv

Comment 17

[hacker formerly known as [e-mail]]

Gerv, first, I think we need buy-in from the build team that this is
collectively what we want to do.  From there, I think dbradley's patch is a good
start though someone probably needs to test it.

Do the mac hacks to libIDL require any other mozilla modules to be built...like
NSPR?

How would one add the prebuilt library & header to their build?

Comment 18

[Gervase Markham [:gerv]]

[CCing scc, who's been involved with this issue.]

Gerv

Comment 19

[TimMaroney]

I am unclear on the implications of the bug but I suspect it is very important to embedding users who are 
looking for cross-platform solutions. libIDL is linked with Mozilla and would need to be distributed with the Mac 
code of any embedding project, right? If so, a GPL or LGPL license would be highly problematic for certain 
embedding developers. For that reason -- if I'm understanding correctly, and apologies if I'm not -- this bug 
should be marked topembed.

Comment 20

[Gervase Markham [:gerv]]

libIDL is not linked with Mozilla; it's used to build the IDL files which are
part of Mozilla. So there are no distribution issues.

Still, we need to make progress on this bug. cls: did you ever find answers to
your questions?

Gerv

Comment 21

[TimMaroney]

Thanks for the clarification, Gerv. I was thinking libIDL was used at run time by XPCOM, not just build time. 
Also I did not realize that Mozilla uses LGPL libraries.

On looking at the mozilla 1.0 source tree, I found three files named COPYING, a GPL/LGPL signature. They 
bear on libIDL (as noted in this bug) and on glib. It looks like glib is in the same situation as libIDL. The 
README.mac file in glib says:

This directory contains a snapshot of glib 1.2.1 slightly modified for
building on the Macintosh only. When these changes are folded back into
the main distribution of glib, this directory will go away.

So, I wonder, how is glib used? Is it run time or build time? Thanks for any help.

Comment 22

[hacker formerly known as [e-mail]]

I crashed the macdev meeting and basically, all we need to do is come up with
some place to put the pre-built package and instructions on how to integrate it
with the build.  The first is easy; we can put it up on ftp.m.o with the win32
prebuilt tools. The second is going to require someone familiar with the cfm
system to determine the best way to automate the install process if possible.  
Or at the very least, make the libIDL install not be a per-tree process.

Tim, libIDL requires glib to build and creates a runtime dependency upon glib. 
The only program that uses libIDL is xpidl and that is primarily used at build
time to create headers from the .idl files.  It is not used by the browser at
runtime and is only packaged for development purposes.

Comment 23

[hacker formerly known as [e-mail]]

Attachment: Add project files for glib & libIDL as well as remove them from xpidl

This patch adds project files for glib & libIDL in a macbuild subdir in the
respective package directories.  This will allow us to build libIDL & glib w/o
having to reintegrate them into our build system.  The libIDL & glib srcs were
removed from the xpidl project file.  Right now, xpidl expects to find the
glib/libIDL headers & static libraries inside the CodeWarrior installation.  I
arbitrarily chose the directory '3rd Party Libraries' and created it in the
root of the toplevel CodeWarrior directory.  The additional search paths look
like {Compiler}:::3rd Party Libraries:glib: & {Compiler}:::3rd Party
Libraries:libIDL: .

Comment 24

[hacker formerly known as [e-mail]]

Attachment: glib-1.2.1-mac.sit

Comment 25

[hacker formerly known as [e-mail]]

Attachment: libIDL-0.6.5-mac.sit

Comment 26

[David Bradley]

I've been trying to verify this, in somewhat of an emperical manner. I thought
everything was good, but when I pulled down a new development environment and
tried to rebuild xpidl my build is not building it. I'm trying to figure out if
this is just a bungling on my part or a problem in the patch.

Comment 27

[David Bradley]

We'll I've built with both release and debug, and Code Warrior stops
unexpectedly, at least that's the message box I get. It's in the _xpcomidl.mcp
project. xpidl has been built along with the linker and prefs. If I just try and
compile a single IDL file it bombs. So something's not quite right. I thought
there might have been a miss match between debug/release but I've tried both
from scratch and they both crash. Is there possibly some other miss match? I'm
using version 7 of Code Warrior, that I got of the MacDev server.

And one last thing. I initially didn't reset my build environment and it built.
Presumeably because it didn't actually rebuild xpidl. So that's when I reset the
build environment and tried building and got this crash.

I'll see if I can come up with more info, but that's all I've got at the moment.

Comment 28

[David Bradley]

After another round of builds here's what I've been able to determine. If I use
the attached glib and libidl libraries CW dies. However if I build them myself
and use the attached patch, it builds fine.

So I think the patch is probably ok. What I don't know is what the difference is
between Christopher's system and mine that would cause this. I tried building
release as well and still had the same crash.

This kind of worries me, as if this is going to crash on trivial version issues,
this could be a bit of a headache.

Any Mac guru's know what might be going on?

Comment 29

[hacker formerly known as [e-mail]]

FWIW, those libs were built using CW 7.2 from the macdev server.  I'm guessing
that  we're hitting a backwards compatibility problem.  I upgraded because CW7.0
stopped building ppembed.  Are you not seeing that problem?

Comment 30

[David Bradley]

Comment on attachment 89153 [details] [diff] [review]
Add project files for glib & libIDL as well as remove them from xpidl

I'm going to give the patch an r=, as I think it's fine.  It looks like we may
just have to provide version specific libs. Given that it worked fine when I
built my own libs and installed in the same location.

I was building on a PowerBook Mac running OSX and a build environment that I
copyed from the Mac dev server, I think 7.0.

I'm still rather new to the Mac dev environment. Anyone know if there are any
other factors that would cause us to need other flavors of these libs, such as
carbon builds vs non-carbon builds.

Comment 31

[David Bradley]

To satisfy my curiousity I pulled down the Codewarrior 7.2 image and built with
the posted libs and it works fine. Do we need to support 7.0? I assume we'll
need to come up with 8.0 built lib's at least.

Comment 32

[Simon Fraser [no longer active]]

7.2 is the currently supported build system.

Comment 33

[David Bradley]

Comment on attachment 89213 [details]
glib-1.2.1-mac.sit

r=dbradley

This worked for 7.2 so they're fine

Comment 34

[David Bradley]

Comment on attachment 89214 [details]
libIDL-0.6.5-mac.sit

r=dbradley

This worked for 7.2 so they're fine

Comment 35

[Simon Fraser [no longer active]]

Can someone summarize how Mac builds work with these changes? Would I have to go
and download something, or do we just check built libs into the tree?

Comment 36

[David Bradley]

Currently the Windows builds expects them to be downloaded manually before the
build. I "assumed" this is how the Mac would work. But if it's easier to put the
binaries in the tree for Mac users I would think that could be done.

Comment 37

[hacker formerly known as [e-mail]]

Per comment 23,
1) download the files (eventually from ftp)
2) create the '3rd Party Libraries' subdir at the top of your CodeWarrior
installation
3) create glib & libIDL subdirs under '3rd Party Libraries'
4) unsit the packages into the respective glib & libIDL subdirs so that the
headers & libs are found as ':3rd Party Libraries:glib:glib.h' etc.

I'd like to avoid the COOL situation that we have in the commercial tree.  We
don't check in any of these binaries for other platforms and I think that it'd
be a bad precedent to start.

Comment 38

[Mark McLoughlin]

Hi guys. I haven't had time to read this bug in detail, but libIDL is currently
actively maintained in GNOME CVS (by me :-). No real development is going on
apart from pretty minor portability fixes.

I'd be very interested in working to see your work merged back upstream rather
than having a fork ... one blocker for you might be that libIDL HEAD (0.8.0)
requires glib 2.0 ...

Anyway, just an FYI

Comment 39

[David Bradley]

Moving out to 1.3. If this needs to be in before 1.3 please comment.

Comment 40

[David Bradley]

Moving to 1.4 Alpha

Comment 41

[David Bradley]

Moving out

Comment 42

[hacker formerly known as [e-mail]]

xpcom/typelib/xpidl/glib & xpcom/typelib/xpidl/libidl have been cvs removed from
the trunk.  You can repull them using the CFM_LAST_RITES tag if need be.

Comment 43

[Phil Schwartau]

Marking Verified -