57470 - examine IID explosion

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect, P3)

Description

[Chris Waterson]

Cuddled up with "readelf" this week: it appears that GetIID()-generated
constants are accounting for 252KB in the .data section!

$ readelf --syms libgklayout.so \
  | awk '$7 == 17 { print; }' \
  | grep iid \
  | awk '{ size += $3 } END { print size; }'
252176

(Section 17 was .data in the build I was testing.)

I built with the following options:

   --enable-optimize --disable-debug --enable-strip-libs.

Comment 1

[Chris Waterson]

Ok, digging into the .data section (using the same technique as on .text, which 
is to build a hashtable of all unique byte strings referenced from .symtab), I 
see that there are 17,216 entries, of which 1,600 are unique; in terms of raw 
space, 147KB of the 426KB .data segment contains unique values.

Comment 2

[Chris Waterson]

Attachment: map of redundant cruft in .data segment

Comment 3

[Chris Waterson]

Attachment: same thing, not gzip'd

Comment 4

[Chris Waterson]

Note that the first 300 or so "redundant" values are actually bonafide global 
variables.

Comment 5

[Brendan Eich [:brendan]]

See bug 8581 (missing GetIID-via-NS_DECLARE_STATIC_IID_ACCESSOR-etc. method in 
layout interfaces); bug 13004 (i'm a lazy bastard for not fixing it earlier); 
and other bugs.  The lxr filename finding URL 
http://lxr.mozilla.org/mozilla/find?string=IIDs from 13004 may help.

Should one person sweep through layout unifying CIDs and IIDs, or should we try 
to divide the labor?

Waterson, can you share your text-redundancy script and results?  I am keen to 
measure basic-block redundancy now.

/be

Comment 6

[jim_nance]

To follow up on my post to the newsgroup, I found the options I was talking
about in the gcc man page.  They are:

-fno-implement-inlines
-fno-implicit-inline-templates

I am building under Red Hat 6.2, and that version of the compiler only supports
the first option.  I will try under Red Hat 7.0 latter.  Anyway the results are
quite supprising:

-rwxr-xr-x    1 jlnance  jlnance   6321099 Oct 21 22:37 libgklayout.so*
-rwxr-xr-x    1 jlnance  jlnance   5969592 Oct 21 21:58 libgklayout.so.orig*

so the file actually got bigger when I told the compiler not to make non-inline
coppies of inline functions.  However if I run size I see:

   text    data     bss     dec     hex filename
3527777  461936    4156 3993869  3cf10d build/libgklayout.so
3166512  461932    4156 3632600  376dd8 build/libgklayout.so.orig

So even though the file is bigger, its text section is significantly smaller.
Neither of these files was compiled with the --enable-strip-libs option, so
the difference in file size may be something that would get stripped out.  I
need to do more work and use a more recent compiler.

Another thing I thought about was that when we build a shared lib we basically
take the entire contents of our .o files and stick them into the lib because
we use the --whole-archive flag.  It may be possible to tell the linker just to
stick the stuff we need into the lib if we know the entry points that need to
be externally visable.  I dont know how XPCOM works.  Is there a fixed set of
entry points that needs to be visable from the outside?

Comment 7

[Brendan Eich [:brendan]]

Jim, if I'm reading your numbers right, text size did increase with RH7.0, by 
about the same amount as the entire .so did.  Does -fno-implement-inlines turn 
off inlining?  That should make things smaller, not larger.  If it does what 
your words suggest (turns off out-of-line copies of inline functions, inlining 
those methods and functions *everywhere*) then it's bound to make things bigger. 
Can we turn off inlining with the latest gcc?

I think XPCOM does have a well-defined set of entry points, although they are 
not extern "C" declared or called out anywhere -- but I may be missing a Windows 
export def file somewhere.

/be

Comment 8

[jim_nance]

Yes, you are correct.  I am reading things wrong.  These are both under RH6.2.
The only difference is the -fno-implement-inlines flag.  Here is what the flag
is supposed to do.  Given:

inline int fun() {return 3;}

all the calls to fun() will be inlined, but the library will still contain a
copy of fun() in case someone called it w/o supplying the definition.  If you
supply the flag its supposed to not put a copy of fun() into the library.  Since
there is no way this could make the library larger, something else is going on.
Im going to play with this some more tonight.

I am not familiar with windows export def files, but it sounds like exactly what
I need.  I would like to tell the linker "here is a list of unresolved symbols",
make a shared lib that resolves them.

Comment 9

[jim_nance]

I moved to redhat 7.0 and added --enable-strip-libs.  The size of the library
drops to 3.5M.  I dont know if this is because the compiler is better, because
the compiler is using a shared c++ library, or because of --enable-strip-libs,
but its an impressive improvement.

Comment 10

[jim_nance]

I cant reproduce that 3.5M lib.  I must have done something wrong

Comment 11

[jim_nance]

OK, I found the problem.  The elf-gc-dynstr program does not run if you just
type make libgklayout.so.  That program brings the library size down from 5
something M to 3.35.  I can knock another 100K off the lib by making the linker
only resolve symbols needed for the XPCOM interface, but thats a lot less than
what I was hoping for.

Comment 12

[jim_nance]

Attachment: patch to strip libs when built

Comment 13

[jim_nance]

Since it confused me that half the stripping was done when the libs were built
and the other half when they were installed, I created a patch to make things
consistent.  It has the added feature of only running the elf stripper one time
rather than two times.  cls, can you take a look at this and add it if you like
it?

Comment 14

[Bruce Mitchener]

Is this an interesting flag for gcc or no:

       -fexternal-templates
              Produce smaller code for template declarations,  by
              generating  only  a  single  copy  of each template
              function where it is defined (C++  only).   To  use
              this  option  successfully,  you must also mark all
              files  that  use  templates  with  either  `#pragma
              implementation'   (the   definition)   or  `#pragma
              interface' (declarations).

              When     your     code     is     compiled     with
              `-fexternal-templates', all template instantiations
              are external.  You must arrange for  all  necessary
              instantiations  to  appear  in  the  implementation
              file; you can do this with a  typedef  that  refer-
              ences  each instantiation needed.  Conversely, when
              you    compile    using    the    default    option
              `-fno-external-templates',  all template instantia-
              tions are explicitly internal.

Comment 15

[Chris Waterson]

Attachment: foldelf.cpp

Comment 16

[jim_nance]

I though I would see what the effect of compiling with -Os (optimize for size)
was.  It knocks about 100K off the size of the text section.

Comment 17

[Chris Waterson]

Which compiler? I was seeing an *increase* in text size using egcs-2.91.66 (the
stock compiler w/ RH6.2)

Comment 18

[jim_nance]

Sorry for the misinformation.  Your right Chris.  I was comparing build done
with 2 different compilers.  It does get larger with -Os and egcs-2.91.66.

Comment 19

[Ken Faulkner]

adding to cc

Comment 20

[Chris Waterson]

Steve Adamski ([e-mail]) has discovered that, for some unknown 
reason, the .data and .rodata are *significantly* smaller when the app is built 
using the latest gcc release (2.95.2).

Comment 21

[Chris Waterson]

assigning to steve, who'll look into this on the latest gcc...

Comment 22

[Pierre Phaneuf]

Bug #21326 is related to this.

Comment 23

[chris hofmann]

should this one get closed out now??