570361 - Package Core build products from exported locations

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Over time, we've built up a collection of Core build products that we package/link from their internal locations, rather than from locations where final build products are exported.  (See also bug 299356 comment 10, wherein mento first starts cleaning this up.)  

I noticed this recently with 1.9.3 packaging (bug 569782), when xpconnect and caps move into gklayout; static linking failed for xpconnect (because the internal .a is no more) but not for caps, because the internal .a was still there, yet both are now gone from staticlib/components, where we should be pulling them from.

We should try to pull things from dist/bin, dist/bin/components, staticlib, and staticlib/components if at all possible, then dist/lib, and finally from internal locations if the libs are not exported (but we should figure out why we're using things that aren't exported and if we still need to do so).

I have some more detailed analysis to follow.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Things we can re-reference right now (format is as follows):

Desired location: (build type)
/* file */ path = current location;

-----

../dist/bin/components: (all)

/* places.xpt */ path = ../toolkit/components/places/public/_xpidlgen/places.xpt;
/* xuldoc.xpt */ path = ../content/xul/document/public/_xpidlgen/xuldoc.xpt;

../dist/bin/components: (shared)

/* libuniversalchardet.dylib */ path = ../extensions/universalchardet/src/xpcom/libuniversalchardet.dylib;
/* libappcomps.dylib */ path = ../xpfe/components/build/libappcomps.dylib;
/* libtxmgr.dylib */ path = ../editor/txmgr/src/libtxmgr.dylib;
/* libgkplugin.dylib */ path = ../modules/plugin/base/src/libgkplugin.dylib;

../staticlib: (all)

/* libmorkreader_s.a */ path = ../db/morkreader/libmorkreader_s.a;
/* libucvutil_s.a */ path = ../intl/uconv/util/libucvutil_s.a;

/* libunicharutil_s.a */ path = ../dist/lib/libunicharutil_s.a;

../staticlib: (all)

/* libjsj.a */ path = ../dist/lib/libjsj.a;
/* libgkgfx.a */ path = ../dist/lib/libgkgfx.a;

../staticlib/components: (static)

/* libappcomps.a */ path = ../xpfe/components/build/libappcomps.a;
/* libcaps.a */ path = ../caps/src/libcaps.a;
/* libchardet.a */ path = ../intl/chardet/src/libchardet.a;
/* libchrome.a */ path = ../rdf/chrome/build/libchrome.a;
/* libcookie.a */ path = ../extensions/cookie/libcookie.a;
/* libdocshell.a */ path = ../docshell/build/libdocshell.a;
/* libembedcomponents.a */ path = ../embedding/components/build/libembedcomponents.a;
/* libgkgfxthebes.a */ path = ../gfx/src/thebes/libgkgfxthebes.a;
/* libgklayout.a */ path = ../layout/build/libgklayout.a;
/* libgkplugin.a */ path = ../modules/plugin/base/src/libgkplugin.a;
/* libi18n.a */ path = ../intl/build/libi18n.a;
/* libimglib2.a */ path = ../modules/libpr0n/build/libimglib2.a;
/* libjar50.a */ path = ../modules/libjar/libjar50.a;
/* libnecko.a */ path = ../netwerk/build/libnecko.a;
/* libpipboot.a */ path = ../security/manager/boot/src/libpipboot.a;
/* libpipnss.a */ path = ../security/manager/ssl/src/libpipnss.a;
/* libpref.a */ path = ../modules/libpref/src/libpref.a;
/* librdf.a */ path = ../rdf/build/librdf.a;
/* libtxmgr.a */ path = ../editor/txmgr/src/libtxmgr.a;
/* libuconv.a */ path = ../intl/uconv/src/libuconv.a;
/* libuniversalchardet.a */ path = ../extensions/universalchardet/src/xpcom/libuniversalchardet.a;
/* libwebbrwsr.a */ path = ../embedding/browser/build/libwebbrwsr.a;
/* libwidget_mac.a */ path = ../widget/src/cocoa/libwidget_mac.a;
/* libxpconnect.a */ path = ../js/src/xpconnect/src/libxpconnect.a;

../dist/bin: (all)

/* libnssdbm3.dylib */ path = ../dist/lib/libnssdbm3.dylib;

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Things we can't re-reference, and which we should investigate whether are still required (with some hints about when/why we started linking them):

http://www.mozdev.org/source/browse/chimera/Chimera.pbproj/project.pbxproj#rev1.11

/* libembed_base_s.a */ path = ../embedding/base/libembed_base_s.a;

Bug 181752 (get the profile dir for profile dir migration)

/* libprofdirserviceprovider_s.a */ path = ../profile/dirserviceprovider/src/libprofdirserviceprovider_s.a;

Build on 1.9.2 (inherited from libmozlcms; linker errors otherwise)

/* libmozqcms.a */ path = ../dist/lib/libmozqcms.a;

http://hg.mozilla.org/camino/rev/15454ca0cd95 (pink, use libmozz 1.2 instead of system zlib)

/* libmozz.a */ path = ../dist/lib/libmozz.a;

http://hg.mozilla.org/camino/rev/624499debacb (bryner; "changes for static builds")

/* libcrmf.a */ path = ../dist/lib/libcrmf.a;

The following four moved out of staticlib by bug 78081:

Bug 298406 ("reference objects in link stage rather than use linker flags"; not sure why we don't need them in shared, too)

/* libmozcairo.a */ path = ../gfx/cairo/cairo/src/libmozcairo.a;
/* libmozjpeg.a */ path = ../jpeg/libmozjpeg.a;

Bug 317077 (in gklayout in shared)

/* libmozlibpixman.a */ path = ../gfx/cairo/libpixman/src/libmozlibpixman.a;
/* libmozpng.a */ path = ../modules/libimg/png/libmozpng.a;


I'm pretty sure we got linker errors without libmozqcms when we started 1.9.2 builds, so I feel comfortable about that; likewise mento's comments on libmozpixman and libmozpng (though in all cases it seems lame they don't end up in staticlib).

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Fix

Stuart, see previous comments (as a follow-up to what I mentioned on irc the other day).  We can try to poke mento for additional info/clarity, too ;)

Comment 4

[Stuart Morgan]

Comment on attachment 449506 [details] [diff] [review]
Fix

sr=smorgan

Comment 5

[Stuart Morgan]

> libembed_base_s.a

Appears to be the source for NS_InitEmbedding, so we definitely want that.

> libprofdirserviceprovider_s.a

We either need this, or a replacement lib if there is such a thing. I don't know if the reasons we stopped using a lib are still good now.

> libmozz.a
> libcrmf.a

What's wrong with these? Isn't dist/lib a reasonable place to link them from?

Comment 6

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #5)
> > libprofdirserviceprovider_s.a
> 
> We either need this, or a replacement lib if there is such a thing. I don't
> know if the reasons we stopped using a lib are still good now.

I checked, and the alternative today is linking the whole toolkit XUL app module (xre), which we definitely don't want to do, or its internal profile _s.a lib, which gets us the toolkit profile manager that we don't want any more than we wanted the more-bloated xpfe profile manager years ago.

> > libmozz.a
> > libcrmf.a
> 
> What's wrong with these? Isn't dist/lib a reasonable place to link them from?

Only that 90% of our libs don't come from there, but otherwise, yes, it's a fine place. (We do link the libmozz.dylib in shared; it just comes from dist/bin like everything else, so I'd've expected the .a to come from staticlib/ somewhere.)

Well, and for libcrmf, I'm not sure why we're linking this one apparently-internal NSS lib, other than we needed it for static builds in June 2002 ;)

Comment 7

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #4)
> (From update of attachment 449506 [details] [diff] [review])
> sr=smorgan

Captain Bitrot, reporting for duty: http://hg.mozilla.org/camino/rev/580ecfee4326

Comment 8

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Addendum 1, oji

For reasons that are inexplicable to me, my previous grepping somehow missed oji. :( (which will trigger the need to do another merge-patch for 1.9.3 :P )

Comment 9

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Attachment: Addendum 1.1, fix oji and sqlite

:sigh: 

Eye-fail; I missed that libsqlite does in fact appear in dist/bin, too.

Comment 10

[Stuart Morgan]

Comment on attachment 449578 [details] [diff] [review]
Addendum 1.1, fix oji and sqlite

sr=smorgan

Comment 11

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #10)
> (From update of attachment 449578 [details] [diff] [review])
> sr=smorgan

http://hg.mozilla.org/camino/rev/87a56a7ed9f7

(In reply to comment #6)

> Well, and for libcrmf, I'm not sure why we're linking this one
> apparently-internal NSS lib, other than we needed it for static builds in June
> 2002 ;)

I want to see if we can figure out whether we still need to link that, and then we can call this bug FIXED.

Comment 12

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

(In reply to comment #11)
> > Well, and for libcrmf, I'm not sure why we're linking this one
> > apparently-internal NSS lib, other than we needed it for static builds in June
> > 2002 ;)
> 
> I want to see if we can figure out whether we still need to link that, and then
> we can call this bug FIXED.

It gets stuffed into NSS_LIBS in congfigure: http://mxr.mozilla.org/mozilla1.9.2/source/configure.in#4256

NSS_LIBS shows up in the static rules: http://mxr.mozilla.org/mozilla1.9.2/source/config/static-config.mk#74 (and, for completeness, in the libXUL config, too: http://mxr.mozilla.org/mozilla1.9.2/source/toolkit/library/libxul-rules.mk#39 ).  Incidentally, I believe we see our other static-only libs mentioned there, too.  I love this bizarro world we live in where shared and static are not mirror images of each other with different libraryname extensions, but hey, this is the stuff that keeps mento in demand wherever he is!

So, cleanup's all done; we're left with a few static-only link references that come out of dist/lib and internal file locations, and two of what we'll call "embedding-only" always-static libs coming out of their internal locations, and that's life.  Thanks for bearing with me. :-)