905646 - Enable -gsplit-dwarf by default on local builds.

Status: NEW

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

+++ This bug was initially created as a clone of Bug #904979 +++

Comment 1

[Mike Hommey [:glandium]]

Attachment: Use -gsplit-dwarf when supported for local builds

Patch as it was attached to bug 904979. Review in bug 904979 comment 19.

Comment 2

[ISHIKAWA, Chiaki]

Hi,
as you have already noted in 
https://bugzilla.mozilla.org/show_bug.cgi?id=904979#c20
> Note that a mozilla contributor has submitted patches to https://bugzilla.samba.org/show_bug.cgi?id=10005 . 
> I don't know how well they work, but we could build and provide binaries, and add a test whether ccache supports the dwo files.

Actually, I have been using this version of ccache on my local PC for a few days, and it seems to work well.
The only thing I have not tried is to debug the final binary using GDB.
(You have to have up-to-date binutils: objcopy that understands --extract-dwo, for example.
We need more testers for this ccache version. I think it works well (and check the operation by looking at the log), but who knows what bug it may still have. 

Some people may wonder why ccache is useful?
In a perfect world, there is no unnecessary recompilation.
In a real world, due to many configuration issues, still being sorted out,
sometimes unwanted recompilation of files, which seemed unwarranted still happen.

Also, using mercurial with many local patches, sometimes, I find myself
popping patches and pushing patches again. These operations touch files and force re-compilation even though the final file content is the same (I thought the operations preserve mtime, but seem not to.)

ccache come in handy in these cases.

By the way, using -gsplit-dwarf
has cut down my link time of libxul from somewhere near 5 minutes to 1m30s range (on a 32 bit linux that runs under VMPlayer environment. My CPU, a couple of generations architecture P620, 32bit memory space, and virtual environment seems to be the speed limit now. Since my main target is thunderbird that runs in 32bit mode, I am hesitant to move to 64 bits.)

This speed-up by -gsplit-dwarf is a killer for local build.

TIA

Comment 3

[Justin Lebar (not reading bugmail)]

Thanks so much for writing ccache patches for this.  I have some experience hacking on ccache, so feel free to ask if you need help getting these patches landed.

Comment 4

[ISHIKAWA, Chiaki]

(In reply to Justin Lebar [:jlebar] from comment #3)
> Thanks so much for writing ccache patches for this.  I have some experience
> hacking on ccache, so feel free to ask if you need help getting these
> patches landed.

Will do. We need more testers, er, users for now :-)

TIA

Comment 5

[Nathan Froyd [:froydnj]]

Per http://gcc.gnu.org/wiki/DebugFission, you may want to make sure we're linking with gold and pass -Wl,--gdb-index so that debugging works.  Perhaps bfd ld supports this option by now, too..

Comment 6

[Mike Hommey [:glandium]]

--gdb-index is currently broken in gold. Its only benefit is to make loading symbols lazy in gdb.

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

It would be nice to have an easy way to disable -gsplit-dwarf without losing other default debugging options.  (Or is it unlikely that they'll ever change from -g?)  I depend on being able to rsync dist/bin from a debug build to another machine and then debug the result, but I'd rather not have to track whatever other changes we make to debugging options.

Comment 8

[ISHIKAWA, Chiaki]

I have been using -gsplit-dwarf in my local build of comm-central thunderbifd for a while  with the modified CCACHE that supports split dwarf files.

It seems to work well, cutting down the compile and link step into 2 minutes range (under 64 bit linux, and I disabled audio-related extension for thunderbird)  except:

gdb can't grok the linked binary completely.

On gdb startup, I see a few messages that says something about "DWO" not understood, meaning that gdb can't grok the linked binary or libxul.so symbol or something.
From DebugFission wiki:
http://gcc.gnu.org/wiki/DebugFission

> Use the gold linker's --gdb-index option (-Wl,--gdb-index when linking with gcc 
> or g++) at link time to create the .gdb_index section that allows GDB to locate 
> and read the .dwo files as it needs them. 

Can someone enlighten me about where exactly this option should be specified?

Is there an easy way to specify this in mozconfig file?
Or, very crudely, should I simply specify this additional option using environment variable setting for CC and CXX as in
CC="cc-binary-progname ... -Wl,--gdb-index"
CXX="c++-binary-progname ... -Wl,--gdb-index"

Also, does anyone know if I need a newer gdb that has specific change for this
split dwarf? I searched GNU project web page and found that I seem to have
a copy of the latest official release. 

One other thing. I use valgrind for memory-related debugging. It is slow, but
under 64bits linux, it certainly feels to run somewhat faster than under 32bit linux. But my guess is that the added memory space plays the major role of improvement. Under 32bits linux, I am hitting virtual space limit and lots of paging. I am checking this since the weekend, but it may be that there could be a few issues regarding symbol search by valgrind when it tries to dump symbol-related information. (I can not use --gen-supression=all sometimes.)
The session crashes, and that is when I tried to use gdb to find that some symbols could not be grokked(?) I will report after using linker option.

TIA



TIA

TIA

Comment 9

[ISHIKAWA, Chiaki]

A few symptoms worth mentioning for developers who build mozilla software locally and
debug them locally.

I have been compiling thunderbird under Debian GNU/Linux (amd64) using
gcc-4.8.1 with -split-dwarf using the modified ccache that understands the split *.o and *.dwo files.

The link goes well, and it cuts down the link time quite a lot to a point that I can live with it.
(I mean, on my SLOW PC, build still takes 3-5 minutes or so after I change a single line in a CPP file, but it is much better than ~10 minutes or longer wait in the past. My use of 64 bit linux inside VMPlayer under Windows 6 64 bit of course doesn't help in terms of speed.)

GDB is another story (and valgrind *may* have some issues.)

Upon startup, GDB reports the following message when I attach it to a thunderbird process that stops in the
middle of |make mozmill| test suite because thunderbird crashes and |make mozmill| test harness offers 300 seconds of window in which I can attach GDB to the dying process to investigate stack and global variables and such, it turned useful sometimes. After 300 seconds, it kills the process completely and moves on to the next test target. [So this 300 second wait is only useful when I am around the computer and notices the print out from the |make mozmill| test harness that I can attach gdb within 300 seconds to the dying TB.]

- begin quote-
Attaching to process 4849
Reading symbols from /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...
warning: Skipping deprecated .gdb_index section in /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
Dwarf Error: CU at offset 0x0 references unknown DWO with ID 070f6622b7b47842 [in module
/TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
(no debugging symbols found)...done.

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from
/usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
done.
   [... lot of omission ... ]
- end quote ---

"deprecated .gdb_index" ?  I was under the impression that -split-dwarf uses .gdb_index section
to let gdb to obtain detailed info for variables, etc. I wonder if anyone can shed light on
the disturbing "deprecated .gdb_index".

Also, "Dwarf Error: CU at offset 0x0 references unknown DWO with ID 070f6622b7b47842 [in module
/TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
(no debugging symbols found)...done." this part seems to indicate that my version of GDB either failed to grok the split dwarf scheme, or that as someone mentions GNU gold linker may not produce proper
dwarf data for gdb to understand (!?)

"warning: Could not load shared library symbols for linux-vdso.so.1" is another hurdle under Debian GNU/LInux amd64 (64 bit architecture) distribution. *.so files used during startup does not seem to
come with debug symbols at all. (Other large libraries do come with separate symbol files, which we can install. But ld*.so files don't.)
I have filed a bug to Debian so that the symbols be made available in the future hopefully.

If anyone can figure out how to make GDB understand the fine details of binary created by
GNU gold linker using gcc-4.8's -split-dwarf and -Wl,--gdb-index, I would appreciate it.
[I specified -Wl,--gdb-index as CC="..." part on make command:
make CC=" ... -Wl,--gdb-index"

ccache handles it properly by passing this option to linker command line (I checked by looking at log file. Basically, ccache passes the link command line, as opposed to the compile-only processing of a single file with -c, to native compiler and so it works just fine.)

One other issue which I have not been able to investigate fully is the
use of -read-var-info=yes option to valgrind. 

Under 32bit linux, trying to read variable information
using -read-var-info runs out of 3GB memory space under Debian GNU/Linux. So I had to do without -read-var-info=yes, and as a result, 
the information printed about uninitialized variable or memory heap, etc. can only point to
the source line and possibly the numerical address of the memory, heap area, etc.

According to valgrind manual, with proper -read-var-info=yes option,
valgrind can print out the variable name  and other symbolic information
related to the improper memory usage issue. That is attractive.
However, I am afraid that, even with -read-var-info, valgrind does not seem to print out such symbolic
name information, under 64bit Debian GNU/Linux.
I suspect that valgrind doesn't grok split dwarf symbols (yet?).
So I am a little disappointed.

TIA

Comment 10

[ISHIKAWA, Chiaki]

(In reply to ISHIKAWA, Chiaki from comment #9)
> A few symptoms worth mentioning for developers who build mozilla software
> locally and
> debug them locally.
> 
> I have been compiling thunderbird under Debian GNU/Linux (amd64) using
> gcc-4.8.1 with -split-dwarf using the modified ccache that understands the
> split *.o and *.dwo files.
> 
> The link goes well, and it cuts down the link time quite a lot to a point
> that I can live with it.
> (I mean, on my SLOW PC, build still takes 3-5 minutes or so after I change a
> single line in a CPP file, but it is much better than ~10 minutes or longer
> wait in the past. My use of 64 bit linux inside VMPlayer under Windows 6 64
> bit of course doesn't help in terms of speed.)
> 
> GDB is another story (and valgrind *may* have some issues.)
> 
> Upon startup, GDB reports the following message when I attach it to a
> thunderbird process that stops in the
> middle of |make mozmill| test suite because thunderbird crashes and |make
> mozmill| test harness offers 300 seconds of window in which I can attach GDB
> to the dying process to investigate stack and global variables and such, it
> turned useful sometimes. After 300 seconds, it kills the process completely
> and moves on to the next test target. [So this 300 second wait is only
> useful when I am around the computer and notices the print out from the
> |make mozmill| test harness that I can attach gdb within 300 seconds to the
> dying TB.]
> 
> - begin quote-
> Attaching to process 4849
> Reading symbols from
> /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...
> warning: Skipping deprecated .gdb_index section in
> /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird.
> Do "set use-deprecated-index-sections on" before the file is read
> to use the section anyway.
> Dwarf Error: CU at offset 0x0 references unknown DWO with ID
> 070f6622b7b47842 [in module
> /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
> (no debugging symbols found)...done.
> 
> warning: Could not load shared library symbols for linux-vdso.so.1.
> Do you need "set solib-search-path" or "set sysroot"?
> Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols
> from
> /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
> done.
>    [... lot of omission ... ]
> - end quote ---
> 
> "deprecated .gdb_index" ?  I was under the impression that -split-dwarf uses
> .gdb_index section
> to let gdb to obtain detailed info for variables, etc. I wonder if anyone
> can shed light on
> the disturbing "deprecated .gdb_index".

deprecated .gdb_index means gdb has found a stale (outdated) .gdb_index file from previous sessions.
We can use
# to cope with -gsplit-dwarf
set use-deprecated-index-sections on

to shut up the complaint.

> 
> Also, "Dwarf Error: CU at offset 0x0 references unknown DWO with ID
> 070f6622b7b47842 [in module
> /TEST-MAIL-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
> (no debugging symbols found)...done." this part seems to indicate that my
> version of GDB either failed to grok the split dwarf scheme, or that as
> someone mentions GNU gold linker may not produce proper
> dwarf data for gdb to understand (!?)
> 
> "warning: Could not load shared library symbols for linux-vdso.so.1" is
> another hurdle under Debian GNU/LInux amd64 (64 bit architecture)
> distribution. *.so files used during startup does not seem to
> come with debug symbols at all. (Other large libraries do come with separate
> symbol files, which we can install. But ld*.so files don't.)
> I have filed a bug to Debian so that the symbols be made available in the
> future hopefully.
> 
> If anyone can figure out how to make GDB understand the fine details of
> binary created by
> GNU gold linker using gcc-4.8's -split-dwarf and -Wl,--gdb-index, I would
> appreciate it.

I have investigated a little more and post a message as follows to gdb mailing list.

But if anyone reading this knows how to debug thuderbird using gdb when -gsplit-dwarf
is used, please let me know.
Without some magic, gdb does not grok necessary debug information for libxul.so, etc.
and can not show the symbol value in the stack backtrace (the function names are visible more or less)
and source lines can not be listed.

The post to gdb mailing list is lengthy, but it has complete information, and so I am quoting it here.

TIA

QUESTION:

I have investigated a little further.

Now it has dawned on me that using "-gsplit-dwarf" binaries for
debugigng is not a simple plug-and-play in contrast to ordinary dwarf
binaries where debug information for a source module is in the single
.o file and linker consolidates the information into a single final
binary.

After looking at this error message line below, I have come to the
conclusion that gdb is complaining that
 - it has found a CU dwarf section (or whatever),
 - this CU section refers to a DWO section (or file) with ID
   77955146488e868b,
 - but it has no knowledge of such DWO section and its whereabout,
   thus "unknown DWO".

--- quote ---
Reading symbols from
/REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
offset 0x0 references
unknown DWO with ID 77955146488e868b [in module
/REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
(no debugging symbols found)...done.
--- end quote ---

So I think I need to tell that relevant .dwo files are in the object directory of thunderbird.

Now that debug information is split, I have to do this. OK, fair enough.

But how can I tell gdb that all the .dwo files and dynamically linked
.so files are under certain directories?  Do I have to tell gdb. (CI comment: Oops copy&paste error.)
Also, do I have to tell gdb in advance, all the *.dwo files that are referenced by thunderbird, and
how?

After reading through gdb documentation and not finding much about
split dwarf feature itself , I now realize that it looks to me that I have to

- create file location dwarf information for gdb to locate
  information in scattered *.dwo files,

 B ut again, how?

- One possibility: this is done by using objcopy by extracting certain debug
  information from ALL *.dwo files and merging into a certain dwarf
  data structure (list of known DWO sections with its unique IDs and
  the file pathname?),

- AND maybe physically attaching this merged dwarf information to
  |thunderbird| and |thunderbird-bin| binary.  Basically extracting all
  the sections of |thunderbird| and merge them with the list of known
  DWO sections with the IDs and file pathnames.) and creating a new
  binary.

- This recreated binary will be used so that gdb will notice all the
  indirect references to the debug information in all *.dwo files when
  |thunderbird| and |thunderbird-bin| is loaded?

Is this what one is supposed to do when -gsplit-dwarf is used
and gdb debugging is attempted?

Oh, I now realize another possibility of creating consolidated
symbolic information for gdb debugging.
maybe I can read all the *.dwo files one by one from within gdb
(of course, I will use gdb script to read them in one sweep),
and write out the symbolic information for next time by |save gdb-index|
This may be the way to go, but not sure

I appreciate any helpful tips and guidance.

TIA


cf. *.dwo files and *.so files created and used
by mozilla thunderbird

I have re-compiled the tree under /REF-COMM-CENTRAL/comm-central
using the ordinary dwarf debug information (without -gsplit-dwarf and -Wl,--gdb-index)
so I am referring to a different tree which was compiled using -gsplit-dwarf.
The top-most directory of the source file hierarchy is /COMM-CENTRAL/comm-central, and
the top-most directory of the object file hierarchy is /TEST-MAIL-OBJ/objdir-tb3.

The final thunderbird binary is in the following directory.
/TEST-MAIL-DIR/objdir-tb3 is the top of the object directory.

There are many *.dwo files scattered in the object tree.
(The structure of the object tree reflects the original source tree more or less.)
The accompanying *.o files are under the same tree.

Also, there are *.so files scattered in the object tree. They are
consolidated into several directories and the number of directories were smaller than in the case of
*.dwo files.


ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ ls -l mozilla/dist/bin/thunderbird*
-rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 12:19 mozilla/dist/bin/thunderbird*
-rwxr-xr-x 1 ishikawa ishikawa 267277 Sep 20 08:42 mozilla/dist/bin/thunderbird-bin*

[*.dwo files]

ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.dwo" -print
./ldap/xpcom/src/nsLDAPProtocolModule.dwo
    [..omission..]
./ldap/xpcom/src/nsLDAPServer.dwo
./db/mork/src/morkStore.dwo
./db/mork/src/morkPortTableCursor.dwo
    [...omission...]
./db/mork/src/morkWriter.dwo
./db/mork/build/nsMorkFactory.dwo
./mozilla/parser/htmlparser/src/nsHTMLEntities.dwo
./mozilla/parser/htmlparser/src/nsHTMLTags.dwo
    [...omission...]
./mozilla/parser/htmlparser/src/nsExpatDriver.dwo
./mozilla/parser/xml/src/nsSAXXMLReader.dwo
./mozilla/parser/xml/src/nsSAXLocator.dwo
./mozilla/parser/xml/src/nsSAXAttributes.dwo
./mozilla/parser/expat/lib/xmlparse.dwo
./mozilla/parser/expat/lib/xmltok.dwo
./mozilla/parser/expat/lib/xmlrole.dwo
./mozilla/parser/html/nsHtml5TreeBuilder.dwo
    [...omission...]
./mozilla/parser/html/nsHtml5Parser.dwo
./mozilla/xpcom/reflect/xptcall/src/xptcall.dwo
    [...omission...]
./mozilla/xpcom/reflect/xptinfo/src/xptiWorkingSet.dwo
./mozilla/xpcom/io/Base64.dwo
    [...omission...]
./mozilla/xpcom/io/nsAppFileLocationProvider.dwo

    [... lots of lines omitted.]
./mail/components/shell/DirectoryProvider.dwo

[*.so files]

ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$ find . -name "*.so" -print
./ldap/sdks/c-sdk/ldap/libraries/libldif/libldif60.so
./ldap/sdks/c-sdk/ldap/libraries/libprldap/libprldap60.so
./ldap/sdks/c-sdk/ldap/libraries/libldap/libldap60.so
./mozilla/xpcom/tests/component_no_aslr/libtestcompnoaslr.so
./mozilla/xpcom/tests/component/libtestcomponent.so
./mozilla/xpcom/tests/bug656331_component/libtest656331.so
./mozilla/dist/plugins/libnptest.so
./mozilla/dist/plugins/libnpsecondtest.so
./mozilla/dist/bin/libmozalloc.so    <--- this is the directory where
./mozilla/dist/bin/libnspr4.so        <--- the main binary is stored.
./mozilla/dist/bin/libfreebl3.so    <--- *.so files are symlinks to other files under the
./mozilla/dist/bin/libssl3.so        <--- top-most objdir-tb3
./mozilla/dist/bin/libnssdbm3.so
./mozilla/dist/bin/libplds4.so
./mozilla/dist/bin/libsmime3.so
./mozilla/dist/bin/libmozsqlite3.so
./mozilla/dist/bin/libnssutil3.so
./mozilla/dist/bin/libxul.so        <--- a symlink libxul.so -> ../../toolkit/library/libxul.so
./mozilla/dist/bin/libsoftokn3.so
./mozilla/dist/bin/libprldap60.so
./mozilla/dist/bin/libldap60.so
./mozilla/dist/bin/libnssckbi.so
./mozilla/dist/bin/libldif60.so
./mozilla/dist/bin/libnss3.so
./mozilla/dist/bin/libplc4.so
./mozilla/dist/bin/libnpsecondtest.so
./mozilla/dist/bin/components/libdbusservice.so
./mozilla/dist/bin/components/libmozgnome.so
./mozilla/dist/lib/libmozalloc.so
./mozilla/dist/lib/libnspr4.so
./mozilla/dist/lib/libfreebl3.so
./mozilla/dist/lib/libssl3.so
./mozilla/dist/lib/libnssdbm3.so
./mozilla/dist/lib/libplds4.so
./mozilla/dist/lib/libsmime3.so
./mozilla/dist/lib/libmozsqlite3.so
./mozilla/dist/lib/libnssutil3.so
./mozilla/dist/lib/libxul.so        <---  symlink to  libxul.so -> ../../toolkit/library/libxul.so
./mozilla/dist/lib/libsoftokn3.so
./mozilla/dist/lib/libprldap60.so
./mozilla/dist/lib/libldap60.so
./mozilla/dist/lib/libnssckbi.so
./mozilla/dist/lib/libldif60.so
./mozilla/dist/lib/libnss3.so
./mozilla/dist/lib/libplc4.so
./mozilla/dist/lib/libnpsecondtest.so
./mozilla/dist/sdk/lib/libmozalloc.so
./mozilla/dist/sdk/lib/libnspr4.so
./mozilla/dist/sdk/lib/libssl3.so
./mozilla/dist/sdk/lib/libplds4.so
./mozilla/dist/sdk/lib/libsmime3.so
./mozilla/dist/sdk/lib/libnssutil3.so
./mozilla/dist/sdk/lib/libxul.so    <--- libxul.so -> ../../../toolkit/library/libxul.so
./mozilla/dist/sdk/lib/libnss3.so
./mozilla/dist/sdk/lib/libplc4.so
./mozilla/dom/plugins/test/testplugin/secondplugin/libnpsecondtest.so
./mozilla/dom/plugins/test/testplugin/libnptest.so
./mozilla/security/nss/lib/util/libnssutil3.so
./mozilla/security/nss/lib/ckfw/builtins/libnssckbi.so
./mozilla/security/nss/lib/smime/libsmime3.so
./mozilla/security/nss/lib/softoken/legacydb/libnssdbm3.so
./mozilla/security/nss/lib/softoken/libsoftokn3.so
./mozilla/security/nss/lib/freebl/libfreebl3.so
./mozilla/security/nss/lib/nss/libnss3.so
./mozilla/security/nss/lib/ssl/libssl3.so
./mozilla/build/unix/elfhack/test-array.so
./mozilla/build/unix/elfhack/test-ctors.so
./mozilla/nsprpub/pr/src/libnspr4.so
./mozilla/nsprpub/lib/libc/src/libplc4.so
./mozilla/nsprpub/lib/ds/libplds4.so
./mozilla/memory/mozalloc/libmozalloc.so
./mozilla/toolkit/system/dbus/libdbusservice.so
./mozilla/toolkit/system/gnome/libmozgnome.so
./mozilla/toolkit/crashreporter/test/libtestcrasher.so
./mozilla/toolkit/library/libxul.so    <--- libary file : size is 283MiB

       This is acccompanied by libxul.so-gdb.py in the same directory
    """ GDB Python customization auto-loader for libxul """

    import os.path
    sys.path[0:0] = [os.path.join('/COMM-CENTRAL/comm-central/mozilla', 'js', 'src', 'gdb')]

    import mozilla.autoload
    mozilla.autoload.register(gdb.current_objfile())




./mozilla/toolkit/components/ctypes/tests/libjsctypes-test.so
./mozilla/db/sqlite3/src/libmozsqlite3.so
./mozilla/js/xpconnect/tests/components/native/libxpctest.so
./mozilla/_tests/testing/mochitest/chrome/libraries/libjsctypes-test.so
./mozilla/_tests/testing/mochitest/chrome/toolkit/components/toolkit/components/ctypes/tests/libjsctypes-test.so
./mozilla/_tests/mozmill-virtualenv/lib/python2.7/site-packages/simplejson/_speedups.so
./mozilla/_tests/mozmill/mozmillprofile/plugins/libnptest.so
./mozilla/_tests/mozmill/mozmillprofile/plugins/libnpsecondtest.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcompnoaslr.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtestcomponent.so
./mozilla/_tests/xpcshell/xpcom/tests/unit/libtest656331.so
./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit/libtestcrasher.so
./mozilla/_tests/xpcshell/toolkit/crashreporter/test/unit_ipc/libtestcrasher.so
./mozilla/_tests/xpcshell/toolkit/components/ctypes/tests/unit/libjsctypes-test.so
./mozilla/_tests/xpcshell/js/xpconnect/tests/components/native/libxpctest.so
ishikawa@vm-debian-amd64:/TEST-MAIL-DIR/objdir-tb3$


(2013/09/21 6:43), ISHIKAWA,chiaki wrote:
> I think the following problem is related to the reading of dynamically linked libxul.so library file.
> The problem affects the behavior of gdb in that it fails to access the source files.
> So it can not list the source lines.
>
> I checked the operation of gdb against statically linked small program and
> all is well.
> It can list the source files, and this is the first thing I immediately noticed.
>
> With the setup described in the previous post of mine,
> gdb can print the stack trace (sans the argument values), but
> it can't print source files. Since the argument values are not shown in the
> stack trace. Its value is also diminished.
>
> Help is appreciated.
> Yes, libxul.so is rather large. Thunderbird is a large program and so
> we may hit into an unknown bug here.
>
>
>
> (2013/09/20 14:24), ishikawa wrote:
>> Hi,
>>
>> I recently learned of "-gsplit-dwarf" of GCC 4.8 and -Wl,--gdb-index
>> that is passed to GNU ld.gold linker, and have tried to use the split
>> debug information scheme for local compilation and link of mozilla
>> thunderbird mail client.
>> ( http://gcc.gnu.org/wiki/DebugFission )
>>
>> The compiling and linking produced working thunderbird binary.  The
>> time required for whole build process, especially the time necessary
>> for creating a large libxul.so file diminished dramatically.  I would
>> like to thank all the developers who have made this possible.
>>
>> There is a fly in the ointment, though.  Unfortunately, somehow gdb is
>> not reading the symbol information from a created shared library,
>> libxul.so, correctly.  Below is the beginning of a gdb session of
>> trying to debug a crash of thunderbird: I attach gdb to a dying
>> thunderbird process.  Note the error message marked with **** in the
>> excerpted lines.
>>
>> This bug is discussed in the following bug entry and
>> the gdb session log is also uploaded at the following URLs.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=918234
>> https://bug918234.bugzilla.mozilla.org/attachment.cgi?id=807287
>>
>>
>> Except of gdb session
>>
>>
>> $ gdb ../../.././mozilla/dist/bin/thunderbird 29680
>> GNU gdb (GDB) 7.6 (Debian 7.6-5)
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>>
>> *****
>> ***** The next line states that such file or directory does not exist
>> ***** (in Japanese).
>> ***** Strange: the command line is suggested by mozilla's
>> ***** |make mozmill| test suite. I have never figured out what
>> ***** the relative path is meant to be.
>> ***** This may play an important role when -gsplit-dwarf is used.
>> ***** Anyway as you can see from the full session log in the above URL,
>> ***** libraries and such are correctly accessed and symbols are
>> ***** deciphered except for libxul.so and a few others.
>>
>> ../../.././mozilla/dist/bin/thunderbird: そのようなファイルやディレクトリは
>> ありません.
>> Attaching to process 29680
>>
>> **** PROBLEM! A fly in the ointment!
>> **** Next line indicates that there is some mismatch of the data
>> **** layout and gdb's read routine.
>>
>> Reading symbols from
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird...Dwarf Error: CU at
>> offset 0x0 references
>> unknown DWO with ID 77955146488e868b [in module
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>> (no debugging symbols found)...done.
>>
>>       **** I am afraid Debian GNU/Linux does not ship symbol information for
>>       **** ld*.so type files: these files seem to be shipped in stripped
>>       **** form.
>> warning: Could not load shared library symbols for linux-vdso.so.1.
>> Do you need "set solib-search-path" or "set sysroot"?
>> Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols
>> from
>> /usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.17.so...done.
>> done.
>> [New LWP 29712]
>> [New LWP 29708]
>> [New LWP 29693]
>> [New LWP 29692]
>> [New LWP 29687]
>> [New LWP 29686]
>>
>>        ... the rest omitted ...
>>
>>
>> I wonder if anyone can shed light on the error:
>> Dwarf Error: CU at offset 0x0 references unknown DWO with ID
>> 77955146488e868b [in module
>> /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird]
>> (no debugging symbols found)...done.
>>
>> I am using
>> gcc 4.8
>>       gcc-4.8 --version
>>       gcc-4.8 (Debian 4.8.1-2) 4.8.1
>>       Copyright (C) 2013 Free Software Foundation, Inc.
>>       This is free software; see the source for copying conditions.  There is NO
>>       warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> ld.gold
>>       ld    --version
>>       GNU gold (GNU Binutils for Debian 2.22) 1.11
>>       Copyright 2011 Free Software Foundation, Inc.
>>       This program is free software; you may redistribute it under the terms of
>>       the GNU General Public License version 3 or (at your option) a later
>> version.
>>       This program has absolutely no warranty.
>>       ishikawa@vm-debian-amd64:/COMM-CENTRAL/comm-central$
>>
>> gdb
>>       gdb --version
>>       GNU gdb (GDB) 7.6 (Debian 7.6-5)
>>       Copyright (C) 2013 Free Software Foundation, Inc.
>>       License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>>       This is free software: you are free to change and redistribute it.
>>       There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>       and "show warranty" for details.
>>       This GDB was configured as "x86_64-linux-gnu".
>>       For bug reporting instructions, please see:
>>       <http://www.gnu.org/software/gdb/bugs/>.
>>
>> (I have upgraded objcopy, etc. to support the split-dwarf scheme, also.
>> Otherwise, the compile and liking with the options mentioned above would fail.)
>>
>> If this reading failure on startup of gdb
>> is a known error (in which case, I would like it to be fixed
>> soon), or if there is a workaround, or if I am using gdb incorrectly,
>> I would like to hear about it.
>>
>> TIA
>>
>>
>
>
>

Comment 11

[ISHIKAWA, Chiaki]

Folks, 

After searching for a clue using Google search engine, I could find an obscure reference to
a utility called |dwp|. This is mentioned in a linux tool chain mail post:
http://www.cygwin.com/ml/binutils/2012-10/msg00365.html
and a RedHat document (there are a few similar documents):
https://access.redhat.com/site/documentation/en-US/Red_Hat_Developer_Toolset/2/html/User_Guide/sect-binutils-Resources.html

However, there is no on-line document about how to use |dwp| as far as I could tell.
Also, the stock GNU gold directory in GNU binutils does't have |dwp| in it.
Only CVS version of gold source tree at sourceware.org
has it.
ftp://sourceware.org/pub/binutils/snapshots/

I downloaded the latest CVS source and compiled gold. dwp utitily is also compiled and
installed in my default /usr/local/bin directory.

Running |dwp -help| printed out something like this.

ishikawa@vm-debian-amd64:~/Dropbox$ dwp --help
Usage: dwp [options] [file...]
  -h, --help               Print this help message
  -e EXE, --exec EXE       Get list of dwo files from EXE (defaults output to EXE.dwp)
  -o FILE, --output FILE   Set output dwp file name
  -v, --verbose            Verbose output
  -V, --version            Print version number

Now some of you may remember that I am using a hacked version of ccache that supports
-gsplit-dwarf linking: this version of ccache supports the two outputs .o and .dwo
and copies them into the cache and retrieves thenm when an identical compilation of .cpp file
is requested (or so I thought. And as far as that is concerned, I was right. But read on.)

Well, a surprise.
When I tested |dwp| to see if "-e thunderbird-bin" spews out something meaningful,
I got the following output.

ishikawa@vm-debian-amd64:~/Dropbox$ dwp -v -e /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird-bin 
/FF-NEW/cache-dir/b/0/9d439fe3e01509d1030519704724ea-1758240.o.tmp.vm-debian-amd64.dwo
dwp: error: cannot open /FF-NEW/cache-dir/b/0/9d439fe3e01509d1030519704724ea-1758240.o.tmp.vm-debian-amd64.dwo: No such file or directory
dwp: fatal error: /FF-NEW/cache-dir/b/0/9d439fe3e01509d1030519704724ea-1758240.o.tmp.vm-debian-amd64.dwo: can't open
ishikawa@vm-debian-amd64:~/Dropbox$ 


WHOA!?

But those of you who know the innards of ccache would immediately recognize the problem.

The strange file name

/FF-NEW/cache-dir/b/0/9d439fe3e01509d1030519704724ea-1758240.o.tmp.vm-debian-amd64.dwo

is something ccache created internally for a compilation step.
To compile C, and CPP source file, 
|ccache| runs C preprocessor (with the provided options) to produce an .i file.
The name of this .i file is based on a to-be-used cached file name under the
ccache's cache directory instead of the original pathname of, say, .cpp file.
(This is important for subsequent discussion.)

|ccache| then invokes the real compiler to produce .o (and .dwo file if -gsplit-dwarf is
provided) from this .i file.
B|cache| internally uses a cache file under the cache directory with
a filename that has a hash value as part of the pathname, and the above strange
looking .dwo file name is the result of ccache producing
/FF-NEW/cache-dir/b/0/9d439fe3e01509d1030519704724ea-1758240.o.tmp.vm-debian-amd64.dwo
at intermediate compilation steps.

I checked and it seems that the intermediate file name was produced during the compilation of
comm-central/mail/app/nsMailApp.cpp
(It figures. The file with main() in it is first encountered in a linked thunderbird-bin.)

The cached files are still found in ccache directory.

  /FF-NEW/cache-dir/b/0:
  合計 19524
  drwxr-xr-x  2 ishikawa ishikawa   16384  9月 26 14:14 .
  drwxr-xr-x 18 ishikawa ishikawa    4096  9月 26 14:17 ..
  -rw-r--r--  1 ishikawa ishikawa   58174  9月 26 13:08 00195740ea9c7b8c0102ce7a7259d0-3733303.d
  -rw-r--r--  1 ishikawa ishikawa 1363481  9月 26 13:08 00195740ea9c7b8c0102ce7a7259d0-3733303.dwo
  -rw-r--r--  1 ishikawa ishikawa 1958712  9月 26 13:08 00195740ea9c7b8c0102ce7a7259d0-3733303.o
                       ....
  -rw-r--r--  1 ishikawa ishikawa   32376  9月 26 14:13 9d439fe3e01509d1030519704724ea-1758240.d
  -rw-r--r--  1 ishikawa ishikawa   87297  9月 26 14:13 9d439fe3e01509d1030519704724ea-1758240.dwo
  -rw-r--r--  1 ishikawa ishikawa   82744  9月 26 14:13 9d439fe3e01509d1030519704724ea-1758240.o
                       ....

Well, I think I have a deeper problem than I initially thought.
Somehow, |gcc -gsplit-dwarf| seems to embed the source file pathname
in the debug section of created .dwo (or .o, or maybe both) file(s).
(Come to think of it, it is natural.)

If there is no |ccache| involvement, then it is fine. 
I believe |dwp| would have found the proper .dwo file immediately using the original pathname
gcc happily placed in .dwo/.o file.

With |ccache|, however, since gcc only knows the artificially created pathnames 
created at intermediate steps of |ccache|, this intermediate path name is embedded in 
the created binary (.dwo and/or/both .o). But they are
removed after successful compilation.
In the case of nsMailApp.cpp compilation, the remaining dwo file
9d439fe3e01509d1030519704724ea-1758240.dwo
are the files that should be referenced. 

dwp, and I am sure gdb also, gets confused by the reference to files that are no longer
there and could not find the symbol file from the binary, thunderbird-bin.

It looks then, my choice is

 - Use -gsplit-dwarf, but give up ccache. 
   (This might be a way to go. After all the build infrastructure seems to be improved each day and so I would need to compile only the exactly necessary files to be compiled instead of compiling a whole set of files that need not be compiled after a small change in the source tree.)

 - Find a way to use the original pathname for -gsplit-dwarf compilation with ccache so that the
   original source file and object file(?) would be placed in the produced binary.
   [Hmm... I see immediately some issues whereby object is produced in a different directory
    using symlinks or something. But as long as the source file path is uniquely determined eventually
    and is embedded in the binary, I think it is fine.]

So I think people not using hacked up ccache (of my own making), and
simply use GNU gold should be able to debug mozilla software
(and maybe an improved gdb, I am not still sure what the
current support of split dwarf files in gdb since I see some
mailings about improvement: e.g., http://sourceware-org.1504.n7.nabble.com/patch-Add-support-for-Fission-DWP-files-td211719.html  I am not entirely sure if the latest GNU gdb has all the necessary support.)

So for now, my testing with valgrind, gdb, etc. only picked up the available information
such as symbol names (cpp's mangled names that let program deduce the
argument types and number of arguments, etc.), but no symbol information related to source files.
It figures. I got no listing of source files and no values passed in the stack trace chain.
I suspect that the variable names are in .dwo files which were never to be discovered when
ccache intervenes between the original cpp source file and GCC, and GCC could 
only know the name of ccache-crafted intermediate file name and put it into the finally created binary.

Oh well, back to the drawing board for ccache that supports -gsplit-dwarf :-(

Oh maybe I could still use dwp to bundle all the .dwo files that are under the MOZ_OBJDIR.
Stay tuned.

TIA

Comment 12

[ISHIKAWA, Chiaki]

To use -gsplit-dwarf fully for debugging,
utility script such as
mozilla/tools/rb/fix-stack-linux.pl
needs to be taught how to handle object files (libxul.so) created
using -gsplit-dwarf and required linker script.

Using -gsplit-dwarf does speed up compilation/speedup very much and
I welcome the speed improvement.
But occassionally, I need to convert stack trace created by NS_ASSERTION()
and I am afraid that the current version of fix-stack-linux.pl 
(the perl modules used by it are the culprits I think) 
do not seem to understand the objects/libxul.so generated by -gsplit-dwarf
and so I have to recompile everything wihout -gsplit-dwarf.
(Yeah, I can use a separate mozconfig and compilation script to
create the two sets of binaries with/without -gsplit-dwarf. Oh well.)

Just a note to be a warning to would-be user of "-gsplit-dwarf".
Maybe I should file a bug about the required enhancement of fix-stack-linux.pl to support -gsplit-dwarf?

TIA

PS: fix-stack-linux.pl may be also confused by the
missing file as noted in the previous comment due to my use of modified ccache, but
it does not even print out any warning regarding it, so I assume that
it does not grok the new split format at all.?

Comment 14

[ISHIKAWA, Chiaki]

I am using CCACHE with the following CCACHE_CPP2 setting for now.

#
# for -gsplit-dwarf
# https://bugzilla.samba.org/show_bug.cgi?id=10005
#
export CCACHE_CPP2=yes

This solved the source file mapping issue from the pathname stored inside 
debug information.

But proper gdb debugging requires the use of a tool called |dwp| and 
proper setup of gdb startup commands, which I have not mastered very well yet.

In the meantime, if you are curious, please visit the URL of bugzilla.samba.org above.
There seemed to be a few issues I overlooked and I *may* be able
to get by without setting CCACHE_CPP2 to above value (that setting defeats the purpose of
avoiding extra processing in the first run of compilation. It runs CPP twice: one as an independent compilation, and then as part of ordinary C (or C++) compiler pass.

I now learn a new option to gcc and clang: gcc's -fdebug-map-prefix and clang's -fdebug-compilation-dir
from a latest post to ccache bugzilla at samba.org which I may be able to use to tell the compiler to generate debug information which explicitly adds the source directory
where the source code ought to be searched. But not sure how to make it work in a tree-like
hierarchy of comm-central, etc. [Not sure if I have to tweak this -f... option setting for each
file, or can get away by just pointing at the top-directory of the source tree, etc. I think I need to experiment a bit.]

Anyway, more eyeballs will certainly help make a progress.

TIA

Comment 15

[ISHIKAWA, Chiaki]

In 
https://bugzilla.mozilla.org/show_bug.cgi?id=985828#c2

Mike Hommey said:
>Note, you're filing this for mac, on mac, clang does something similar to debug fission by default.
(where |you| refers to an earlier poster, Ehsan Akhgari, in bug 985828.)

I am curious: does debugging with GDB with such debug-fission-like objects/binary on Mac? 
Or does it require special processing of object files? [Maybe Apple did a better integration of its development/debug tool chain than the free-for-all open source tool available in the wild.]

TIA

Comment 16

[(not currently active) Ted Mielczarek]

By default Apple's toolchain doesn't link DWARF into the final binary, it leaves it in the .o files. Apple's GDB knows how to track down the .o files and read the debug info from there. Most other tools do not, and require you to run the "dsymutil" tool which links the debug info together. The default debugger on OS X nowadays is LLDB, and I don't know if it does the same thing but I assume so.

Comment 17

[ISHIKAWA, Chiaki]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #16)
> By default Apple's toolchain doesn't link DWARF into the final binary, it
> leaves it in the .o files. Apple's GDB knows how to track down the .o files
> and read the debug info from there. Most other tools do not, and require you
> to run the "dsymutil" tool which links the debug info together. The default
> debugger on OS X nowadays is LLDB, and I don't know if it does the same
> thing but I assume so.

Thank you.

> Apple's GDB knows how to track down the .o files

This is the part I am struggling without much success (due to scarcity of documentation and
lack of time. A very small-scale compiling maybe a few files in a flat directory and
non-flat directory structure, linking and debugging with gdb
work with |dwp| based on the scant information I have. But
when I try it with thunderbird binary, somehow something failed.
Oh, use of |dwp| basically re-builds the whole .dwo information files into a gigantic
single file for reference, and this defeats the original purpose or
reducing the linking time and consumed space. Building such a file
is only required for someone who needs debugging (and, in local
testing case, it can be done
in the background once the final binary is created.)
Gdb can look for source files with the file path recorded in .dwo
names, and we need proper setting of source file path and that is where I think 
I am failing, but could not dig up the smoking gun yet.
 
I will re-check
ccache: making sure improper meddling of path names is not done.
gcc: proper use of -fdebug-map-prefix during linking (might help)
gdb: figuring out exact command(s)  for proper source file search

Sometimes I feel just putting extra fprintf() is easier :-)

Comment 18

[ISHIKAWA, Chiaki]

Hi, I fixed hopefully the final remaining issue in my modified version of
ccache.
Now I can debug thunderbird-bin binary using gdb (!)

Used Tools:
     gcc --version
     gcc (Debian 4.8.2-16) 4.8.2

     ld (is a shell script to invoke ld.gold)
     ld --version
     GNU gold (GNU Binutils for Debian 2.24) 1.11

    ccache to support -gsplit-dwarf is in the following as before.
    https://bitbucket.org/zephyrus00jp/ccache-gsplit-dwarf-support 
 
1. Now I no longer need to define CCACHE_CPP2 (= yes) during
   compilation. 

2. But, please don't forget to specify the debug-file-directory to
   make gdb search for .dwo files in various directories.
  
For this, I modified my gdb startup file, .gdbinit,
to contain the following two commands
source commandfile1
source commandfile2

commandfile1 is used to specify the directories to search for .dwo
files (this is because only the last component of the full pathname of
.dwo file is embedded in object file due to the way thunderbird source
files are compiled [by visiting each source directory and then
compiling the source without giving the full pathname.).

commandfile1 is the pathname of a
file that contains a command to set |debug-file-directory| to gdb.
E.g.:
set debug-file-directory MOZ_OBJ/db/mork/build:MOZ_OBJ/db/mork/src:...:MOZ_OBJ/mozilla/xpfe/components/windowds

In the real file, MOZ_OBJ is actually the directory path of my MOZ_OBJ.

I created the list of directories as follows.
Basically I searched for the directories where .dwo files are stored by
the following command:
find MOZ_OBJ -name "*.dwo" -print | xargs dirname | sort | uniq
This prints out the directory list.

Then I created the commandfile1 to set debug-file-directory to tell
gdb to look for *.dwo files in |debug-file-directory| with the
following command:
set debug-file-directory dir1:dir2:dir3:...:dirN
(I manually concatenated the directory list with intervening ":".)

I also inserted 
source commandfile2
in .gdbinit
where commandfile2 contains the list of the following commands.
      path dir1
      path dir2
      path dir3
      ...
      path dirN

where dir1, dir2, dir3, ..., dirN are the same as are used for
commandfile1.
path command is used to tell the gdb where to look for the source
files.

(The way TB is compiled, only the basename part of the full pathname
is passed to the compiler, and so I think we must tell gdb where to
look for the source files as well as .dwo files. Oh well, I may be wrong on the
necessity of commandfile2, i.e., |path| command part. But |set
debug-file-directory| is absolutely necessary.)

With these set up,
I could run gdb ....path.../thunderbird-bin
and could set a breakpoint to a function that is not 
loaded at the initial load time, and
get gdb prompt when the execution reaches the breakpoint (!)

TODO/FIXME:
1. I don't know what to do for the case of clang. Anyone?

There are issues with support utilities. Some I can think of.

2. Does run-time backtrace printing function work with this Dwarf Fission scheme?
   (Actually built-in one as well as routines used elsewhere. See 3 below.)

3. There is a script to decode the stack dump (as is used
   by NS_ASSERT(), ASSERTION(), etc. 
   The script is, in comm-central, 
   ${MOZ_SRCDIR}/mozilla/tools/rb/fix-linux-stack.pl

   I wonder if this tool is usable with Dwarf Fission scheme.
   I doubt it. There is now NO way to pass the
   search directories for .dwo file lookup. Hmm...
 
SUCCESSFUL RUN:

Below is the successful debug session.
My TB is a full DEBUG build of TB and so
I culled the debug messages manually somewhat from the recording below.

Before running, I set a breakpoint on a function which was
not loaded initially (presumably in an .so somewhere)
> break nsMsgLocalMailFolder::DeleteMessages

And then after the TB startup, I delete a message from a folder and
then this breakpoint is encountered and I get gdb prompt!
And the stack is visible, too!

I still need to figure out the issue behind the following
messages, but I think it is related to my system setup, and
not intrinsic to the modified cache, etc.
>warning: the debug information found in "/lib64/ld-2.18.so" does not match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).

>warning: Could not load shared library symbols for linux-vdso.so.1.
>Do you need "set solib-search-path" or "set sysroot"?

Also, I noticed that listing backtrace using "where" takes 
much, much longer than in the case of
the ordinary debug file (non-split case).
The listing stops at #20, and at #32 for a while and I thought
gdb got hung until I see the listing finally.
Once the whole stack is printed, I can re-issue "where" and
get an instantaneous listing.
I suspect this is due to the added File I/O to open debug section,
search for .dwo file, and then parse it, etc.

Wait, there are many arguments on the stacktrace which  are optimized
out.  Well, I am using 
ac_add_options --enable-optimize="-g -O2 -freorder-blocks"
to run the full DEBUG version of TB under valgrind,
so this -O2 must be responsible for the optimized-out variables.

But any issues/bug reports related to this modified ccache will
be welcome.

Here is the real session (edited to remove many warning lines, etc.)
Tested on Debian GNU/Linux 64-bit version.


ishikawa@vm-debian-amd64:~/repos/binutils-2.24.51/gold$ gdb /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird-bin 
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird-bin...done.
(gdb) break nsMsgLocalMailFolder::DeleteMessages
Function "nsMsgLocalMailFolder::DeleteMessages" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 1 (nsMsgLocalMailFolder::DeleteMessages) pending.
(gdb) run
Starting program: /REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/thunderbird-bin 
warning: the debug information found in "/lib64/ld-2.18.so" does not match "/lib64/ld-linux-x86-64.so.2" (CRC mismatch).

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loading JavaScript value pretty-printers; see js/src/gdb/README.
If they cause trouble, type: disable pretty-printer .* SpiderMonkey
[19789] WARNING: '!compMgr', file /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/glue/nsComponentManagerUtils.cpp, line 59
...
[19789] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/crashreporter/nsExceptionHandler.cpp, line 2107
[19789] WARNING: '!compMgr', file /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/glue/nsComponentManagerUtils.cpp, line 59
...
[19789] WARNING: Couldn't get the user appdata directory. Crash events may not be produced.: file /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/crashreporter/nsExceptionHandler.cpp, line 2107
[New Thread 0x7fffe2fc3700 (LWP 19798)]
[New Thread 0x7fffe2749700 (LWP 19799)]
[New Thread 0x7fffe1eec700 (LWP 19800)]
[New Thread 0x7fffe169b700 (LWP 19801)]
[New Thread 0x7fffe0aff700 (LWP 19802)]
[New Thread 0x7fffd3fff700 (LWP 19803)]
[New Thread 0x7fffd32f4700 (LWP 19804)]
[New Thread 0x7fffe0caa700 (LWP 19805)]
[New Thread 0x7fffd20ff700 (LWP 19806)]
[New Thread 0x7fffae2cb700 (LWP 19808)]
[New Thread 0x7fffae0ca700 (LWP 19809)]
[New Thread 0x7fffd2aa3700 (LWP 19810)]
[New Thread 0x7fffd26ff700 (LWP 19811)]
[New Thread 0x7fffd24ff700 (LWP 19812)]
[New Thread 0x7fffd22ff700 (LWP 19813)]
[New Thread 0x7fffad6ff700 (LWP 19814)]
[New Thread 0x7fff8ffff700 (LWP 19815)]
JavaScript strict warning: chrome://global/content/bindings/tree.xml, line 50: reference to undefined property this.treeBoxObject.columns
...
[19789] WARNING: NS_ENSURE_TRUE(NS_SUCCEEDED(rv) && subjPrincipal) failed: file /REF-COMM-CENTRAL/comm-central/mozilla/docshell/base/nsDocShell.cpp, line 8682
...
[19789] WARNING: Failed to open external DTD: publicId "-//W3C//DTD SVG 1.1//EN" systemId "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" base "file:///REF-COMM-CENTRAL/comm-central/mail/themes/linux/mail/tabs/selected-start.svg" URL "resource://gre/res/dtd/svg11.dtd": file /REF-COMM-CENTRAL/comm-central/mozilla/parser/htmlparser/src/nsExpatDriver.cpp, line 696
...
[New Thread 0x7fff83fff700 (LWP 19817)]
[New Thread 0x7fff837ae700 (LWP 19818)]
[19789] WARNING: NS_ENSURE_TRUE(globalObject && globalObject->GetGlobalJSObject()) failed: file /REF-COMM-CENTRAL/comm-central/mozilla/content/html/document/src/nsHTMLContentSink.cpp, line 739
++DOMWINDOW == 19 (0x31ac030) [pid = 19789] [serial = 19] [outer = 0x1a57b30]
[19789] WARNING: Subdocument container has no frame: file /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsDocumentViewer.cpp, line 2419
...
[19789] WARNING: NS_ENSURE_TRUE(globalObject && globalObject->GetGlobalJSObject()) failed: file /REF-COMM-CENTRAL/comm-central/mozilla/content/html/document/src/nsHTMLContentSink.cpp, line 739
++DOMWINDOW == 24 (0x3329610) [pid = 19789] [serial = 24] [outer = 0x2236990]
[19789] WARNING: NS_ENSURE_TRUE(globalObject && globalObject->GetGlobalJSObject()) failed: file /REF-COMM-CENTRAL/comm-central/mozilla/content/html/document/src/nsHTMLContentSink.cpp, line 739
JavaScript strict warning: chrome://messenger/content/tabmail.xml, line 352: reference to undefined property aTabType.panelId
[New Thread 0x7fff827ff700 (LWP 19819)]
[New Thread 0x7fff81fae700 (LWP 19820)]
[New Thread 0x7fff813ff700 (LWP 19821)]
[19789] WARNING: Could not get disk status from nsIDiskSpaceWatcher: file /REF-COMM-CENTRAL/comm-central/mozilla/uriloader/prefetch/nsOfflineCacheUpdateService.cpp, line 325
[New Thread 0x7fff80bae700 (LWP 19822)]
[New Thread 0x7fff67fff700 (LWP 19823)]
[New Thread 0x7fff677ae700 (LWP 19824)]
[New Thread 0x7fff66f5d700 (LWP 19825)]
[New Thread 0x7fff6670c700 (LWP 19826)]
[New Thread 0x7fff65ab2700 (LWP 19827)]
[New Thread 0x7fff65261700 (LWP 19828)]
++DOMWINDOW == 31 (0x2a23730) [pid = 19789] [serial = 31] [outer = 0x1a78a00]
[New Thread 0x7fffd227e700 (LWP 19829)]
[New Thread 0x7fff64a10700 (LWP 19830)]
[New Thread 0x7fff4bfff700 (LWP 19831)]
[Thread 0x7fffd20ff700 (LWP 19806) exited]
[19789] WARNING: We should have hit the document element...: file /REF-COMM-CENTRAL/comm-central/mozilla/layout/xul/nsBoxObject.cpp, line 178
[19789] WARNING: We should have hit the document element...: file /REF-COMM-CENTRAL/comm-central/mozilla/layout/xul/nsBoxObject.cpp, line 178

Breakpoint 1, nsMsgLocalMailFolder::DeleteMessages (this=<optimized out>, 
    messages=<optimized out>, msgWindow=<optimized out>, 
    deleteStorage=<optimized out>, isMove=<optimized out>, 
    listener=<optimized out>, allowUndo=true)
    at /REF-COMM-CENTRAL/comm-central/mailnews/local/src/nsLocalMailFolder.cpp:1152
1152	{
(gdb)  where
#0  nsMsgLocalMailFolder::DeleteMessages (this=<optimized out>, 
    messages=<optimized out>, msgWindow=<optimized out>, 
    deleteStorage=<optimized out>, isMove=<optimized out>, 
    listener=<optimized out>, allowUndo=true)
    at /REF-COMM-CENTRAL/comm-central/mailnews/local/src/nsLocalMailFolder.cpp:1152
#1  0x00007ffff42c3f9e in nsMsgDBView::DeleteMessages (this=<optimized out>, 
    window=<optimized out>, indices=<optimized out>, 
    numIndices=<optimized out>, deleteStorage=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgDBView.cpp:3093
#2  0x00007ffff42cd453 in nsMsgDBView::ApplyCommandToIndices (
    this=<optimized out>, command=<optimized out>, indices=<optimized out>, 
    numIndices=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgDBView.cpp:2799
#3  0x00007ffff42d197d in nsMsgDBView::DoCommand (this=<optimized out>, 
    command=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgDBView.cpp:2468
#4  0x00007ffff1e6c88e in NS_InvokeByIndex (that=<optimized out>, 
    methodIndex=<optimized out>, paramCount=<optimized out>, 
    params=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_x86_64_unix.cpp:164
#5  0x00007ffff2e70628 in Invoke (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:2436
#6  Call (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1752
#7  XPCWrappedNative::CallMethod (ccx=..., mode=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1719
#8  0x00007ffff2e7c2ea in XPC_WN_CallMethod (cx=<optimized out>, 
    argc=<optimized out>, vp=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1285
#9  0x00007ffff51042c0 in js::CallJSNative (cx=<optimized out>, 
    native=<optimized out>, args=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jscntxtinlines.h:239
#10 0x00007ffff50f2c7b in js::Invoke (cx=<optimized out>, args=..., 
    construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:476
#11 0x00007ffff50f499e in Interpret (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:2614
#12 0x00007ffff51021f8 in js::RunScript (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:423
#13 0x00007ffff50f2df2 in js::Invoke (cx=<optimized out>, args=..., 
    construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:495
#14 0x00007ffff4f9ce2e in js_fun_call (cx=<optimized out>, 
    argc=<optimized out>, vp=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsfun.cpp:938
#15 0x00007ffff51042c0 in js::CallJSNative (cx=<optimized out>, 
    native=<optimized out>, args=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jscntxtinlines.h:239
#16 0x00007ffff50f2c7b in js::Invoke (cx=<optimized out>, args=..., 
    construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:476
#17 0x00007ffff50f499e in Interpret (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:2614
#18 0x00007ffff51021f8 in js::RunScript (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:423
#19 0x00007ffff50f2df2 in js::Invoke (cx=<optimized out>, args=..., 
    construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:495
#20 0x00007ffff51033f4 in js::Invoke (cx=<optimized out>, thisv=..., 
    fval=..., argc=<optimized out>, argv=<optimized out>, rval=JSVAL_VOID)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:532
#21 0x00007ffff4f55c89 in JS::Call (cx=<optimized out>, thisv=..., fval=..., 
    args=..., rval=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4947
#22 0x00007ffff294dddd in mozilla::dom::EventHandlerNonNull::Call (
    this=<optimized out>, cx=<optimized out>, aThisVal=..., event=..., 
    aRv=...)
    at /REF-OBJ-DIR/objdir-tb3/mozilla/dom/bindings/EventHandlerBinding.cpp:36
#23 0x00007ffff30931e8 in Call<nsISupports*> (
    aExceptionHandling=<optimized out>, aRv=..., event=..., 
    thisObjPtr=<optimized out>, this=<optimized out>)
    at ../../dist/include/mozilla/dom/EventHandlerBinding.h:62
#24 nsJSEventListener::HandleEvent (this=<optimized out>, 
    aEvent=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsJSEventListener.cpp:235
#25 0x00007ffff3089977 in nsEventListenerManager::HandleEventSubType (
    this=<optimized out>, aListenerStruct=<optimized out>, 
    aDOMEvent=<optimized out>, aCurrentTarget=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventListenerManager.cpp:964
#26 0x00007ffff3089c36 in nsEventListenerManager::HandleEventInternal (
    this=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aDOMEvent=<optimized out>, 
    aCurrentTarget=<optimized out>, aEventStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventListenerManager.cpp:1024
#27 0x00007ffff308410b in HandleEvent (aEventStatus=<optimized out>, 
    aCurrentTarget=<optimized out>, aDOMEvent=<optimized out>, 
    aEvent=<optimized out>, aPresContext=<optimized out>, 
    this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventListenerManager.h:328
#28 nsEventTargetChainItem::HandleEvent (this=<optimized out>, aVisitor=..., 
    aCd=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:194
#29 0x00007ffff3082771 in nsEventTargetChainItem::HandleEventTargetChain (
    aChain=..., aVisitor=..., aCallback=<optimized out>, aCd=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:282
#30 0x00007ffff308396f in nsEventDispatcher::Dispatch (
    aTarget=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aDOMEvent=<optimized out>, 
    aEventStatus=<optimized out>, aCallback=<optimized out>, 
    aTargets=aTargets@entry=0x0)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:595
#31 0x00007ffff3083e35 in nsEventDispatcher::DispatchDOMEvent (
    aTarget=<optimized out>, aEvent=<optimized out>, 
    aDOMEvent=<optimized out>, aPresContext=<optimized out>, 
    aEventStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:659
#32 0x00007ffff3a7d494 in PresShell::HandleDOMEventWithTarget (
    this=<optimized out>, aTargetContent=<optimized out>, 
    aEvent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:7418
#33 0x00007ffff337b43e in nsContentUtils::DispatchXULCommand (
    aTarget=<optimized out>, aTrusted=<optimized out>, 
    aSourceEvent=<optimized out>, aShell=<optimized out>, 
    aCtrl=<optimized out>, aAlt=<optimized out>, aShift=aShift@entry=false, 
    aMeta=aMeta@entry=false)
    at /REF-COMM-CENTRAL/comm-central/mozilla/content/base/src/nsContentUtils.cpp:5630
#34 0x00007ffff3cc6660 in nsButtonBoxFrame::DoMouseClick (
    this=<optimized out>, aEvent=<optimized out>, aTrustEvent=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/xul/nsButtonBoxFrame.cpp:152
#35 0x00007ffff3cc6291 in nsButtonBoxFrame::HandleEvent (
    this=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aEventStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/xul/nsButtonBoxFrame.cpp:115
#36 0x00007ffff3a8d20e in nsPresShellEventCB::HandleEvent (
    this=<optimized out>, aVisitor=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:489
#37 0x00007ffff30827c3 in nsEventTargetChainItem::HandleEventTargetChain (
    aChain=..., aVisitor=..., aCallback=<optimized out>, aCd=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:324
#38 0x00007ffff308396f in nsEventDispatcher::Dispatch (
    aTarget=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aDOMEvent=<optimized out>, 
    aEventStatus=<optimized out>, aCallback=<optimized out>, 
    aTargets=aTargets@entry=0x0)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventDispatcher.cpp:595
#39 0x00007ffff3a84897 in PresShell::HandleEventInternal (
    this=<optimized out>, aEvent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:7261
#40 0x00007ffff3a8741c in PresShell::HandleEventWithTarget (
    this=<optimized out>, aEvent=<optimized out>, aFrame=<optimized out>, 
    aContent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:7016
#41 0x00007ffff303f30d in nsEventStateManager::CheckForAndDispatchClick (
    this=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventStateManager.cpp:4810
#42 0x00007ffff3043cfb in nsEventStateManager::PostHandleEvent (
    this=<optimized out>, aPresContext=<optimized out>, 
    aEvent=<optimized out>, aTargetFrame=<optimized out>, 
    aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/events/nsEventStateManager.cpp:3411
#43 0x00007ffff3a84905 in PresShell::HandleEventInternal (
    this=<optimized out>, aEvent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:7273
#44 0x00007ffff3a85415 in PresShell::HandlePositionedEvent (
    this=<optimized out>, aTargetFrame=<optimized out>, 
    aEvent=<optimized out>, aEventStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:6990
#45 0x00007ffff3a86fd4 in PresShell::HandleEvent (this=<optimized out>, 
    aFrame=<optimized out>, aEvent=<optimized out>, 
    aDontRetargetEvents=<optimized out>, aEventStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/layout/base/nsPresShell.cpp:6793
#46 0x00007ffff332d981 in nsViewManager::DispatchEvent (this=<optimized out>, 
    aEvent=<optimized out>, aView=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/view/src/nsViewManager.cpp:758
#47 0x00007ffff33293c8 in nsView::HandleEvent (this=<optimized out>, 
    aEvent=<optimized out>, aUseAttachedEvents=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/view/src/nsView.cpp:1084
#48 0x00007ffff2d506b0 in nsWindow::DispatchEvent (this=<optimized out>, 
    aEvent=<optimized out>, aStatus=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/gtk/nsWindow.cpp:466
#49 0x00007ffff2d565d2 in nsWindow::OnButtonReleaseEvent (
    this=<optimized out>, aEvent=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/gtk/nsWindow.cpp:2808
#50 0x00007ffff2d5666f in button_release_event_cb (widget=<optimized out>, 
    event=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/gtk/nsWindow.cpp:5296
#51 0x00007ffff001b695 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
#52 0x00007ffff0838138 in g_closure_invoke ()
   from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#53 0x00007ffff0849aad in ?? ()
   from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#54 0x00007ffff0851469 in g_signal_emit_valist ()
   from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#55 0x00007ffff0851a52 in g_signal_emit ()
   from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#56 0x00007ffff012a954 in ?? ()
   from /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
#57 0x00007ffff0019e44 in gtk_propagate_event ()
   from /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
#58 0x00007ffff001a1fb in gtk_main_do_event ()
   from /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
#59 0x00007fffef01ab2c in ?? ()
   from /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#60 0x00007ffff056f526 in g_main_context_dispatch ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#61 0x00007ffff056f878 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#62 0x00007ffff056f91c in g_main_context_iteration ()
   from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#63 0x00007ffff2d4814f in nsAppShell::ProcessNextNativeEvent (
    this=<optimized out>, mayWait=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/gtk/nsAppShell.cpp:138
#64 0x00007ffff2d984bd in nsBaseAppShell::DoProcessNextNativeEvent (
    this=<optimized out>, mayWait=<optimized out>, 
    recursionDepth=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/xpwidgets/nsBaseAppShell.cpp:140
#65 0x00007ffff2d985eb in nsBaseAppShell::OnProcessNextEvent (
    this=<optimized out>, thr=<optimized out>, mayWait=<optimized out>, 
    recursionDepth=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/xpwidgets/nsBaseAppShell.cpp:298
#66 0x00007ffff1e620d4 in nsThread::ProcessNextEvent (this=<optimized out>, 
    mayWait=<optimized out>, result=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/threads/nsThread.cpp:616
#67 0x00007ffff1dc9f7d in NS_ProcessNextEvent (thread=<optimized out>, 
    mayWait=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/glue/nsThreadUtils.cpp:263
#68 0x00007ffff219c35f in mozilla::ipc::MessagePump::Run (
    this=<optimized out>, aDelegate=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/glue/MessagePump.cpp:136
#69 0x00007ffff2159d80 in MessageLoop::RunInternal (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:226
#70 0x00007ffff2159dd8 in RunHandler (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:219
#71 MessageLoop::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:193
#72 0x00007ffff2d977e9 in nsBaseAppShell::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/xpwidgets/nsBaseAppShell.cpp:164
#73 0x00007ffff40a7c53 in nsAppStartup::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/components/startup/nsAppStartup.cpp:276
#74 0x00007ffff3fda6a8 in XREMain::XRE_mainRun (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4008
#75 0x00007ffff3fdc137 in XREMain::XRE_main (this=<optimized out>, 
    argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4075
#76 0x00007ffff3fdc4a4 in XRE_main (argc=<optimized out>, 
    argv=<optimized out>, aAppData=<optimized out>, aFlags=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4285
#77 0x000000000040371d in do_main (argv=0x7fffffff8888, argc=1, 
    exePath=0x7fffffff7770 "/REF-OBJ-DIR/objdir-tb3/mozilla/dist/bin/")
    at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:117
#78 main (argc=1, argv=0x7fffffff8888)
    at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:223
(gdb) 


[end of memo]

Comment 19

[ISHIKAWA, Chiaki]

I forgot to mention.

(a-1)
-Wl,--gdb-index: per comment 5

I specify "-Wl,--gdb-index" to CC and CXX environment variable
(before configure, etc.)
To wit, this is in my script:
(I am not sure, but ASAN option below seems to cause the failure of
TB compilation. So it is not specified for now.)
#
SPLITDWARF="-Wl,--gdb-index"
# SPLITDWARF=""
ASAN=-fsanitize=address
ASAN=
export CC="/usr/bin/gcc-4.8  $ASAN -fno-builtin-strlen $SPLITDWARF -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1"
export CXX="/usr/bin/g++-4.8 $ASAN  -fno-builtin-strlen $SPLITDWARF -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1"

-fno-builtin-strlen is given here since the strlen in the
latest Debian library seems to have an issue with valgrind's idea of memory check.
(It uses 4 or 8 byte load, and somtimes the last portion of 4 or 8 byte load falls
outside the allocated area, and then valgrind/memcheck raises the illegal access.)

(2) .gdb_index from comment 10

>deprecated .gdb_index means gdb has found a stale (outdated) .gdb_index file from previous sessions.
>We can use
># to cope with -gsplit-dwarf
> set use-deprecated-index-sections on

Come to think of it, whether this is good is a question, but
I put this in .gdbinit.

To wit, the following lines are at the
beginning of .gdbinit file:
--- begin quote ---
# to cope with -gsplit-dwarf
set use-deprecated-index-sections on

# .gdbinit file for debugging Mozilla
#
# Originally from http://web.mit.edu/bzbarsky/www/gdbinit
#
#
# Path for looking for .dwo files.
source /REF-COMM-CENTRAL/work-dir/d.dirs
# Path for source files. Maybe we don't need this? 
source /REF-COMM-CENTRAL/work-dir/t.dirs
--- end quote ---

I think the explicit path for .dwo files 
given by source .../d.dirs sets up the
internal structure for search correctly (.gdb_index) initially,
and unless
TB's file directory structure changes, on the subsequent 
we should be OK (even if .gdb_index may be "stale".)

Any comment from people familiar with gdb/ld/gcc Dwarf Fission are welcome!

TIA

Comment 20

[ISHIKAWA, Chiaki]

For completeness's sake, here is how I specify -gsplit-dwarf in my local mozconfig.

# LINK speed OPTIMIZATION
ac_add_options --enable-debug-symbols=-gsplit-dwarf

Comment 21

[ISHIKAWA, Chiaki]

After trying new ccache and gdb on a different PC (32-bit linux PC)
I realized  that there ares more subtle points: you need to 
put in a few more commands in .gdbinit to get
the |bt|, |where|, |list| command in gdb to work.

Without the settings explained below,
you may not see the symbols and source information very well..

For example, I did not I see the symbols for argument in libxul.so on a PC without
the proper .gdbinit setting.

E.g.:
   from /new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/libxul.so
#7  0xb2baca80 in nsInputStreamPump::OnInputStreamReady(nsIAsyncInputStream*)
    ()
   from /new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/libxul.so
#8  0xb2abb879 in nsInputStreamReadyEvent::Run() ()
   from /new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/libxul.so
#9  0xb2ae0746 in nsThread::ProcessNextEvent(bool, bool*) ()


Analysis:

[1] Should I issue something to reset .gdb_index
because I see the following warning at gdb startup?

warning: Skipping deprecated .gdb_index section in
/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/thunderbird-bin.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.

So I put 
"set use-deprecated-index-sections on"
in my .gdbinit and restart gdb.

[2] More warning.

After setting |use-deprecated-index-sections| to |on|, I got
the following warning during gdb startup.

warning: File
"/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/toolkit/library/libxul.so-gdb.py"
auto-loading has been declined by your `auto-load safe-path' set to
"$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path
/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/toolkit/library/libxul.so-gdb.py
line to your configuration file "/home/mtest2/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/mtest2/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

So I add the described |add-auto-load-safe-path| command to .gdbinit,
and restarted gdb.

Now I get only the following extra warning, which I am not sure how to
solve it. It does not seem to impact gdb operation much anyway.

E.g.:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/thunderbird-bin...done.
(gdb) run
Starting program:
/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/thunderbird-bin
warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Loading JavaScript value pretty-printers; see js/src/gdb/README.
If they cause trouble, type: disable pretty-printer .* SpiderMonkey

[Summary]

Put "set use-deprecated-index-sections on" to your .gdbinit.

This seems to be necessary, and also put 

|add-auto-load-safe-path YOUR_MOZ_OBJ/mozilla/toolkit/library/libxul.so-gdb.py|

in .gdbinit, too. (The above path may be different for your local PC and the target program, TB, FF, etc.)

The above seem to be important to make sure that the symbols in
libxul.so can be printed in the stack trace (and the source lines for
the functions in it) correctly.

After the above changes (and the |set debug-file-directory ...| explained earlier) in .gdbinit, 
I get the following which I wanted to see:

E.g.: Issuing |where| after a breakpiont is reached.
(Before the modification to .gdbinit, the function name 
was shown, but argument symbols and line informatin line were missing. This was also true when
breakpoint is reached and function information is printed.)

E.g.:
   ...
New Thread 0x9e2f9b40 (LWP 421)]
++DOMWINDOW == 23 (0x9b26708) [pid = 358] [serial = 35] [outer = 0x8d78f48]
GetDiskSpaceAvailable returned: 3384508416 bytesw

Breakpoint 1, nsMsgBrkMBoxStore::GetNewMsgOutputStream (this=0x9d6b670,
    aFolder=aFolder@entry=0x9d5bc2c, aNewMsgHdr=0x8e23950,
    aReusable=aReusable@entry=0xbfffe39f, aResult=aResult@entry=0x8e238dc)
    at
/new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mailnews/local/src/nsMsgBrkMBoxStore.cpp:614
614	{
(gdb) where
#0  nsMsgBrkMBoxStore::GetNewMsgOutputStream (this=0x9d6b670,
    aFolder=aFolder@entry=0x9d5bc2c, aNewMsgHdr=0x8e23950,
    aReusable=aReusable@entry=0xbfffe39f, aResult=aResult@entry=0x8e238dc)
    at
/new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mailnews/local/src/nsMsgBrkMBoxStore.cpp:614
#1  0xb5634c2b in nsMsgLocalMailFolder::InitCopyMsgHdrAndFileStream (
    this=this@entry=0x9d5bc08)
    at
/new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mailnews/local/src/nsLocalMailFolder.cpp:1963
#2  0xb56408b8 in nsMsgLocalMailFolder::BeginCopy (this=0x9d5bc08, message=
    0x8e21ea8)
    at
/new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mailnews/local/src/nsLocalMailFolder.cpp:1979
#3  0xb52d66c0 in nsCopyMessageStreamListener::OnStartRequest (
    this=0x943da98, request=0x87af24c, ctxt=0x8bb3f28)
    at
/new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mailnews/base/src/nsCopyMessageStreamListener.cpp:101

	[ #4 ... #19 are omitted ]

#20 0x0804af6a in do_main (argv=0xbffffc74, argc=1,
    exePath=0xbfffebc0
"/new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/")
    at /new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mail/app/nsMailApp.cpp:111
#21 main (argc=1, argv=0xbffffc74)
    at /new-hd1/extra/ishikawa/TB-3HG/NEW-COMMSRC/mail/app/nsMailApp.cpp:200
(gdb)
(gdb) list
609	NS_IMETHODIMP
610	nsMsgBrkMBoxStore::GetNewMsgOutputStream(nsIMsgFolder *aFolder,
611	                                         nsIMsgDBHdr **aNewMsgHdr,
612	                                         bool *aReusable,
613	                                         nsIOutputStream **aResult)
614	{
615	  NS_ENSURE_ARG_POINTER(aFolder);
616	  NS_ENSURE_ARG_POINTER(aNewMsgHdr);
617	  NS_ENSURE_ARG_POINTER(aReusable);
618	  NS_ENSURE_ARG_POINTER(aResult);
(gdb) quit
A debugging session is active.

	Inferior 1 [process 358] will be killed.

Quit anyway? (y or n) y


I hope the above information is helpful for those trying to debug
mozilla software (in this case TB) using -gsplit-dwarf, ld.gold (GNU
gold linker) and new enough binutils.

TIA

PS: I compared the content of .gdbinit on my main development PC and
found that .gdbinit contains extra command as

add-auto-load-safe-path MOZ_SRC/mozilla/.gdbinit

MOZ_SRC is actually the real path for my local mozilla source.
(This is the case of debugging TB in comm-central. Maybe for firefox
debugging, you need to specify MOZ_SRC/.gdbinit ?)

Also, my .gdbinit contained
the content taken from
# .gdbinit file for debugging Mozilla
#
# Originally from http://web.mit.edu/bzbarsky/www/gdbinit
#

Hope this helps.

Comment 22

[ISHIKAWA, Chiaki]

Agah, more mozilla-specific complications, very subtle indeed.

On a PC (Debian GNU/Linux 32-bit), I noticed ccache did not use
cache very well. Where I thought hits would have been produced, there were
cache misses.

Now, I realize the problem.
Depending on the setup in mozconfig file, I think,
[and maybe the use of |mach| command ?]
a following option in the C compiler command line is produced:
e.g., -DGRE_BUILDID=20140327174446

It looks like a timestamp of a sort.
Since ccache uses the command line arguments as well as the preprocessed C source file
to create the hash key, if the value of GRE_BUILDID is different at the invocation of
compiler even if the preprocessed source file is the same, so is the hash and
there is no cache hit.

(Also, there seems to be feature called "direct hit" check which seem to
use only the timestamps of involved files (to preprcess the source file), and
the command line argument. If the source file and involved files have not changed, 
and the compiler command arguments are the same, then we don't even need to create
preprocessed file, and pick up the cached object immediately.
This obviously won't work if there is a changing GRE_BUILDID define on the command line.)

So we have to be careful in setting up the
mozconfig for local compilation to make use of ccache effectively.

I am not sure which mozconfig setting controls the appearance of -DGRE_BUILDID=YYYYMMHHmmss
on the command line. (Or can it be an environment variable setting? Does the use of |mach|
affect this?)

Anyway, this is something one wants to be aware of.
(Such an argument to C compiler won't make ccache useful on
shared compilation server.)

TIA

Comment 23

[Robert Kaiser]

(In reply to ISHIKAWA, Chiaki from comment #22)
> Since ccache uses the command line arguments as well as the preprocessed C
> source file
> to create the hash key, if the value of GRE_BUILDID is different at the
> invocation of
> compiler even if the preprocessed source file is the same, so is the hash and
> there is no cache hit.

I think this has nothing directly to do with the bug report here, but it's a very interesting find. Could you file a separate bug on this ccache+mozilla inefficiency?

Comment 24

[ISHIKAWA, Chiaki]

(In reply to Robert Kaiser (:[e-mail]) from comment #23)
> (In reply to ISHIKAWA, Chiaki from comment #22)
> > Since ccache uses the command line arguments as well as the preprocessed C
> > source file
> > to create the hash key, if the value of GRE_BUILDID is different at the
> > invocation of
> > compiler even if the preprocessed source file is the same, so is the hash and
> > there is no cache hit.
> 
> I think this has nothing directly to do with the bug report here, 

You are right!
I was so hung up that a bug of modified ccache prohibited the use of
-gsplit-dwarf and debugging using gdb until lately (a few days ago),
and so I blindly reported the find about ccache in this bugzilla.
Blush.

> but it's a
> very interesting find. Could you file a separate bug on this ccache+mozilla
> inefficiency?

After checking to make sure there is no outstanding ccache bug entry related to this issue, 
I will file a separate bug.
(Maybe we need a master entry for the usage of ccache with mozilla software if there is not one already.)

TIA

Comment 25

[ISHIKAWA, Chiaki]

Filed
Bug 988976 - BEWARE: ccache does not work efficiently with -DGRE_BUILDID=YYYYMMDDhhmmss compiler option

I have a feeling maybe I have a patch that touches configuration file (configuration data)
that possibly caused a reconfigure to run when I pop/push/pop/push hg patch queues to
refresh the tree by fetching latest comm-central (pop)and the merge them (push), and remove now empty patches because the have landed (pop), and finally merge the patches to continue debugging/testing (push).
It is unlikely that each compilation command uses a different timestamp, and may be the
reconfigured/recreated script holds such timestamp of the reconfigure date.

Still, in this case, it is instructive to know that a pop/push of patch queue may trigger reconfigure which, in turn, produces different GRE_BUILDID that may affect ccache operation negatively very much.

People who do local build for debugging/testing ought to know this.
Fast CPU of today may have hidden this issue under the rug.

But why this happens shall be discussed in the new bug.

Thank you.

Comment 26

[Tom Tromey :tromey]

(In reply to ISHIKAWA, Chiaki from comment #21)

Hi.  A couple notes on this.

First, I want to encourage trying -gsplit-dwarf.  I think in conjunction
with the gdb index it can speed up both linking and gdb startup -- a nice
combination.

> [1] Should I issue something to reset .gdb_index
> because I see the following warning at gdb startup?
> 
> warning: Skipping deprecated .gdb_index section in
> /new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/thunderbird-bin.
> Do "set use-deprecated-index-sections on" before the file is read
> to use the section anyway.
> 
> So I put 
> "set use-deprecated-index-sections on"
> in my .gdbinit and restart gdb.

This warning occurs because gdb expects the very latest version of
the .gdb_index section (currently version 8) but gold only knows how
to emit an earlier version (version 7).

I don't recall exactly why this is, but telling gdb to use the deprecated
sections anyway is fine.


> warning: Could not load shared library symbols for linux-gate.so.1.
> Do you need "set solib-search-path" or "set sysroot"?

It's safe to ignore this.  I vaguely recall that this was fixed either
in glibc or gdb, but I don't remember now.  The gdb bug is still open:
https://sourceware.org/bugzilla/show_bug.cgi?id=14466

Comment 27

[ISHIKAWA, Chiaki]

(In reply to Tom Tromey :tromey from comment #26)
> (In reply to ISHIKAWA, Chiaki from comment #21)
> 
> Hi.  A couple notes on this.
> 
> First, I want to encourage trying -gsplit-dwarf.  I think in conjunction
> with the gdb index it can speed up both linking and gdb startup -- a nice
> combination.

Yes, I wholeheartedly agree on this.

Gdb starts up quickly [at least, it feels so!], and
the link time using GNU ld.gold with -gsplit-dwarf has been
wonderful on my somewhat slow (in modern standard) machine.
If someone needs a ccache that understands -split-dwarf [the standard/official version
does not grok it, and invoke the real compiler always.], please let me know.
(from comment 18
    ccache to support -gsplit-dwarf is in the following as before.
    https://bitbucket.org/zephyrus00jp/ccache-gsplit-dwarf-support 

> > [1] Should I issue something to reset .gdb_index
> > because I see the following warning at gdb startup?
> > 
> > warning: Skipping deprecated .gdb_index section in
> > /new-hd1/extra/ishikawa/TB-3HG/objdir-tb3/mozilla/dist/bin/thunderbird-bin.
> > Do "set use-deprecated-index-sections on" before the file is read
> > to use the section anyway.
> > 
> > So I put 
> > "set use-deprecated-index-sections on"
> > in my .gdbinit and restart gdb.
> 
> This warning occurs because gdb expects the very latest version of
> the .gdb_index section (currently version 8) but gold only knows how
> to emit an earlier version (version 7).
> 
> I don't recall exactly why this is, but telling gdb to use the deprecated
> sections anyway is fine.
> 
> 
> > warning: Could not load shared library symbols for linux-gate.so.1.
> > Do you need "set solib-search-path" or "set sysroot"?
> 
> It's safe to ignore this.  I vaguely recall that this was fixed either
> in glibc or gdb, but I don't remember now.  The gdb bug is still open:
> https://sourceware.org/bugzilla/show_bug.cgi?id=14466

It seems then we can ignore this for the moment.

Having to pay to binary utilities is a pain during mozilla development.
I encountered an ld.gold bug that was fixed earlier this year, but
it was triggered by a GCC regression. Oh well, sometimes the debugging mozilla requires
the tracing all the way to GCC's mis-generating a certain section of object file :-(

(In reply to Justin Lebar (not reading bugmail) from comment #3)
> Thanks so much for writing ccache patches for this.  I have some experience
> hacking on ccache, so feel free to ask if you need help getting these
> patches landed.

I think this patched ccache in
https://bitbucket.org/zephyrus00jp/ccache-gsplit-dwarf-support 
is usable now.

It was based on a current released source (and contains maybe a bit from the next release).
It ran local test of its own |make test| successfully this spring.
And like the comments in this thread, I can run gdb successfully with the objects from GCC/G++/modified ccache combination successfully for the last several months.
(I forgot, fix-linux-stack script still does not seem to understand the
binary created by -gsplit-dwarf. I think it is more of a problem of one of the binutils programs
that is used by fix-linux-stack script.) 

However, the source code of ccache has been diverging from the original ccache source
on which the "-gsplit-dwarf" support was created in the last half year or so, and so
I think it would be good if we can start merge work before the base source diverges too widely.
Any idea?

TIA

Comment 28

[Tom Tromey :tromey]

I tried this out a little bit today.

It didn't help at all, much to my surprise.
Build times are about the same, and gdb "attach" times are too.
The latter was quite surprising to me, I thought that the index
would make more difference.

I did check to make sure that the gdb index is there, and I did
set the option to use it (despite its having the wrong version number).

Comment 29

[ISHIKAWA, Chiaki]

(In reply to Tom Tromey :tromey from comment #28)
> I tried this out a little bit today.
> 
> It didn't help at all, much to my surprise.
> Build times are about the same,

If your CPU is fast, and you have plenty of memory to start with,
build times would be about the same, I think.
If CPU is not that fast (CPU from the past generation), and 
memory is tight (i.e. 32bit OS),
using -gsplit-dwarf and ld.gold can make a large difference during link time and
thus shorter build time, and lesser memory usage (ld.gold).
If I did not use ld.gold and -gsplit-dwarf, the linking of DEBUG VERSION of TB could not be done in 32-bit space. It failed. 
With the combination of -gsplit-dwarf and ld.gold, it can be done in 32-bit space.
(It could be done until this summer. Due to many short bugs lately, C-C tree was disarray, and I could not compile DEBUG VERSION of C-C TB under 32-bit OS on a slow CPU OS.)

> and gdb "attach" times are too.
> The latter was quite surprising to me, I thought that the index
> would make more difference.

It is.
There was a visible difference on my PC.

When "-gsplit-dwarf" was not used, loading TB into gdb was very slow.

After -gsplit-dwarf is used, initial loading TB into gdb has become fast.

Instead, traceback or listing now takes much longer. It takes time before
something is printed. Once that happens, it can be repeated fast enough. 
I think this is because
gdb now needs to access many debug information files (.dwo) and search them in many directories to obtain symbol information, instead of accessing gigantic debug info sections in linked binaries.

> I did check to make sure that the gdb index is there, and I did
> set the option to use it (despite its having the wrong version number).

This requires a little investigation.
I wonder which version of 
C compiler and linker
were used. 
OS, and CPU.
Any special options passed to C compiler or linker?


Is your tree new version that contains
the patch from fixed
Bug 1081682 - ccache "-with-ccache=/usr/bin/ccache" is not used properly any more in build (both C-C and M-C) 

My environment

(I compiled ld myself.)
ld --version
GNU gold (GNU Binutils 2.24.51.20140610) 1.11

gdb --version
GNU gdb (Debian 7.7.1+dfsg-3) 7.7.1

gcc --version
gcc (Debian 4.9.1-15) 4.9.1

uname -a
Linux ip030 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 GNU/Linux
(that is, 64-bit Debian GNU/Linux)

--- a few issues

Now that said, since the end of September,
I have run into a difficulty to run gdb successfully after a change of
environment. I am running a newly installed OS image (from Debian
GNU/Linux Wheezy to GNU/Linux Jessie.)

I am not sure what is wrong.

- the change of binary layout (object directory) C-C TB did not help.
  I had to change the source/object  file search path setup.

  The first time around, my attempt to pass the source file
  paths, all the different paths that need to be told to gdb
  backfired.  At the time of |exec|, "argument list became too long"
  and gdb could not run TB.

  I think certain environment variable containing the path must have
  been passed to TB or something.  Come to think of it, probably "set
  debug-file-directory ....very long path list...." to tell gdb where
  to look for .dwo would be enough for gdb to figure out where .dwo
  files are and from there to find source paths.  So I had to remove
  path /REF-OBJ-DIR/objdir-tb3
  path .... many variations ....
  path .........................
  specified in a file sourced during .gdbinit

  Then gdb could run TB

Now, gdb successfuly runs. (As I write this, I realize that there
could be a system-wide setting of argument length to exec. I need to
investigate.)

But, although gdb runs TB, sometimes
it cannot print traceback after TB crashed during |make mozmil|
and I try to attach the image (this could well be TB's problem: it may
have really trashed memory and corrupt the stack. Or some symbols from system libraries could not be found...),

One more thing I noticed: it could be that the latest gcc 4.9 is very eager with
-O2 and gdb does not print variable values anymore saying that
they are optimized out. I think I need to use -O or something instead.
(I wonder if there is a bug in the gdb that comes with experimental
release of Debian, though.)

A run example:

$ gdb /REF-OBJ-DIR/objdir-tb3/dist/bin/thunderbird
GNU gdb (Debian 7.7.1+dfsg-3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /REF-OBJ-DIR/objdir-tb3/dist/bin/thunderbird...done.
(gdb) break main
Breakpoint 1 at 0x4031c0: file /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp, line 318.
(gdb) break nsMsgAccountManager::LoadAccounts
Function "nsMsgAccountManager::LoadAccounts" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (nsMsgAccountManager::LoadAccounts) pending.
(gdb) run
Starting program: /REF-OBJ-DIR/objdir-tb3/dist/bin/thunderbird 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main (argc=1, argv=0x7fffffffda08) at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:318
318	{
(gdb) where
#0  main (argc=1, argv=0x7fffffffda08) at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:318

(gdb) list
313	
314	  return rv;
315	}
316	
317	int main(int argc, char* argv[])
318	{
319	#ifdef XP_MACOSX
320	  TriggerQuirks();
321	#endif
322	
(gdb) cont
Continuing.
Loading JavaScript value pretty-printers; see js/src/gdb/README.
If they cause trouble, type: disable pretty-printer .* SpiderMonkey
[New Thread 0x7fffe7c4f700 (LWP 3010)]
[New Thread 0x7fffe73d5700 (LWP 3011)]
[New Thread 0x7fffe6b78700 (LWP 3016)]
[New Thread 0x7fffe6af7700 (LWP 3017)]
[New Thread 0x7fffe6a76700 (LWP 3018)]
[New Thread 0x7fffe69f5700 (LWP 3019)]
[New Thread 0x7fffe6974700 (LWP 3020)]
[New Thread 0x7fffe68f3700 (LWP 3021)]
[New Thread 0x7fffe6872700 (LWP 3022)]
[New Thread 0x7fffe67f1700 (LWP 3023)]
[New Thread 0x7fffe56ff700 (LWP 3024)]
[New Thread 0x7fffe4bff700 (LWP 3025)]
[New Thread 0x7fffb7a35700 (LWP 3026)]
[New Thread 0x7fffe4e8b700 (LWP 3027)]
JavaScript warning: resource://gre/modules/Preferences.jsm, line 381: mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create

	   [... many debug output ...]
	   
Breakpoint 2, nsMsgAccountManager::LoadAccounts (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgAccountManager.cpp:1236
1236	  if (m_accountsLoaded) {
(gdb) where
#0  nsMsgAccountManager::LoadAccounts (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgAccountManager.cpp:1236
#1  0x00007fffeff24d5b in nsMsgAccountManager::GetAllIdentities (this=<optimized out>, _retval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mailnews/base/src/nsMsgAccountManager.cpp:1136
#2  0x00007ffff04686b3 in NS_InvokeByIndex (that=<optimized out>, methodIndex=<optimized out>, paramCount=<optimized out>, 
    params=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:164
#3  0x00007ffff0d9f699 in Invoke (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:2394
#4  Call (this=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1746
#5  XPCWrappedNative::CallMethod (ccx=..., mode=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1713
#6  0x00007ffff0dad616 in GetAttribute (ccx=...) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/xpcprivate.h:2140
#7  XPC_WN_GetterSetter (cx=<optimized out>, argc=<optimized out>, vp=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1281
#8  0x00007ffff4040a14 in js::CallJSNative (cx=<optimized out>, native=<optimized out>, args=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jscntxtinlines.h:231
#9  0x00007ffff403e410 in js::Invoke (cx=<optimized out>, args=..., construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:482
#10 0x00007ffff403eb15 in js::Invoke (cx=<optimized out>, thisv=..., fval=..., argc=<optimized out>, argv=<optimized out>, 
    rval=JSVAL_VOID) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:538
#11 0x00007ffff403ec8a in js::InvokeGetterOrSetter (cx=<optimized out>, obj=<optimized out>, 
    fval=$jsval((JSObject *) 0x7fffb44a0a00 [object Function "allIdentities"]), argc=<optimized out>, argv=<optimized out>, 
    rval=...) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:611
#12 0x00007ffff405f187 in get (vp=..., pobj=<optimized out>, obj=<optimized out>, receiver=..., cx=<optimized out>, 
    this=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Shape-inl.h:44
#13 NativeGetInline<(js::AllowGC)1> (cx=<optimized out>, obj=..., receiver=..., pobj=..., shape=..., vp=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/NativeObject.cpp:1641
#14 0x00007ffff405f50f in GetPropertyHelperInline<(js::AllowGC)1> (cx=<optimized out>, obj=..., receiver=..., id=..., 
    vp=JSVAL_VOID) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/NativeObject.cpp:1889
#15 0x00007ffff405f9aa in js::baseops::GetProperty (cx=<optimized out>, obj=..., receiver=..., id=..., vp=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/NativeObject.cpp:1899
#16 0x00007ffff3bf439e in JSObject::getGeneric (cx=<optimized out>, obj=..., receiver=..., id=..., vp=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/NativeObject.h:1396
#17 0x00007ffff4032c71 in GetPropertyOperation (vp=..., lval=..., pc=<optimized out>, script=..., fp=<optimized out>, 
    cx=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:253
#18 Interpret (cx=0x6f9c90, state=...) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:2373
#19 0x00007ffff403e1c7 in js::RunScript (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:432
#20 0x00007ffff403fce8 in js::ExecuteKernel (cx=<optimized out>, script=..., scopeChainArg=..., thisv=..., 
    type=<optimized out>, evalInFrame=..., result=0x0) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:638
#21 0x00007ffff404027f in js::Execute (cx=<optimized out>, script=..., scopeChainArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:675
#22 0x00007ffff3e9ae1a in ExecuteScript (cx=<optimized out>, obj=..., scriptArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4727
#23 0x00007ffff3e9af5c in JS_ExecuteScriptVersion (cx=<optimized out>, obj=..., script=..., version=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4766
#24 0x00007ffff0d1b9b9 in mozJSComponentLoader::ObjectForLocation (this=<optimized out>, aInfo=..., 
    aComponentFile=<optimized out>, aObject=..., aTableScript=..., aLocation=<optimized out>, aPropagateExceptions=true, 
    aException=JSVAL_VOID) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:931
#25 0x00007ffff0d1fa67 in mozJSComponentLoader::ImportInto (this=<optimized out>, aLocation=..., targetObj=..., 
    callercx=<optimized out>, vp=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:1150
#26 0x00007ffff0d1fdfb in mozJSComponentLoader::Import (this=<optimized out>, registryLocation=..., targetValArg=..., 
    cx=<optimized out>, optionalArgc=<optimized out>, retval=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:1037
#27 0x00007ffff0d3b13b in nsXPCComponents_Utils::Import (this=<optimized out>, registryLocation=..., targetObj=..., 
    cx=<optimized out>, optionalArgc=<optimized out>, retval=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCComponents.cpp:2750
#28 0x00007ffff04686b3 in NS_InvokeByIndex (that=<optimized out>, methodIndex=<optimized out>, paramCount=<optimized out>, 
    params=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:164
#29 0x00007ffff0d9f699 in Invoke (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:2394
#30 Call (this=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1746
#31 XPCWrappedNative::CallMethod (ccx=..., mode=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1713
#32 0x00007ffff0dad152 in XPC_WN_CallMethod (cx=<optimized out>, argc=<optimized out>, vp=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1245
#33 0x00007ffff4040a14 in js::CallJSNative (cx=<optimized out>, native=<optimized out>, args=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jscntxtinlines.h:231
#34 0x00007ffff403e410 in js::Invoke (cx=<optimized out>, args=..., construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:482
#35 0x00007ffff403960b in Interpret (cx=0x6f9c90, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:2537
#36 0x00007ffff403e1c7 in js::RunScript (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:432
#37 0x00007ffff403fce8 in js::ExecuteKernel (cx=<optimized out>, script=..., scopeChainArg=..., thisv=..., 
    type=<optimized out>, evalInFrame=..., result=0x0) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:638
#38 0x00007ffff404027f in js::Execute (cx=<optimized out>, script=..., scopeChainArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:675
#39 0x00007ffff3e9ae1a in ExecuteScript (cx=<optimized out>, obj=..., scriptArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4727
#40 0x00007ffff3e9af5c in JS_ExecuteScriptVersion (cx=<optimized out>, obj=..., script=..., version=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4766
#41 0x00007ffff0d1b9b9 in mozJSComponentLoader::ObjectForLocation (this=<optimized out>, aInfo=..., 
    aComponentFile=<optimized out>, aObject=..., aTableScript=..., aLocation=<optimized out>, aPropagateExceptions=true, 
    aException=JSVAL_VOID) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:931
#42 0x00007ffff0d1fa67 in mozJSComponentLoader::ImportInto (this=<optimized out>, aLocation=..., targetObj=..., 
    callercx=<optimized out>, vp=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:1150
#43 0x00007ffff0d1fdfb in mozJSComponentLoader::Import (this=<optimized out>, registryLocation=..., targetValArg=..., 
    cx=<optimized out>, optionalArgc=<optimized out>, retval=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/loader/mozJSComponentLoader.cpp:1037
#44 0x00007ffff0d3b13b in nsXPCComponents_Utils::Import (this=<optimized out>, registryLocation=..., targetObj=..., 
    cx=<optimized out>, optionalArgc=<optimized out>, retval=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCComponents.cpp:2750
#45 0x00007ffff04686b3 in NS_InvokeByIndex (that=<optimized out>, methodIndex=<optimized out>, paramCount=<optimized out>, 
    params=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/reflect/xptcall/md/unix/xptcinvoke_x86_64_unix.cpp:164
#46 0x00007ffff0d9f699 in Invoke (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:2394
#47 Call (this=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1746
#48 XPCWrappedNative::CallMethod (ccx=..., mode=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNative.cpp:1713
#49 0x00007ffff0dad152 in XPC_WN_CallMethod (cx=<optimized out>, argc=<optimized out>, vp=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1245
#50 0x00007ffff4040a14 in js::CallJSNative (cx=<optimized out>, native=<optimized out>, args=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jscntxtinlines.h:231
#51 0x00007ffff403e410 in js::Invoke (cx=<optimized out>, args=..., construct=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:482
#52 0x00007ffff403960b in Interpret (cx=0x1322370, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:2537
#53 0x00007ffff403e1c7 in js::RunScript (cx=<optimized out>, state=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:432
#54 0x00007ffff403fce8 in js::ExecuteKernel (cx=<optimized out>, script=..., scopeChainArg=..., thisv=..., 
    type=<optimized out>, evalInFrame=..., result=0x0) at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:638
#55 0x00007ffff404027f in js::Execute (cx=<optimized out>, script=..., scopeChainArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/vm/Interpreter.cpp:675
#56 0x00007ffff3e9ae1a in ExecuteScript (cx=<optimized out>, obj=..., scriptArg=..., rval=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4727
#57 0x00007ffff3ea0bab in JS::CloneAndExecuteScript (cx=<optimized out>, 
    obj=(JSObject * const) 0x7fffb6328060 [object Window] delegate, scriptArg=...)
    at /REF-COMM-CENTRAL/comm-central/mozilla/js/src/jsapi.cpp:4753
#58 0x00007ffff277257d in mozilla::dom::XULDocument::ExecuteScript (this=<optimized out>, aScript=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/xul/XULDocument.cpp:3596
#59 0x00007ffff277fd74 in mozilla::dom::XULDocument::OnScriptCompileComplete (this=<optimized out>, aScript=<optimized out>, 
    aStatus=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/dom/xul/XULDocument.cpp:3485
#60 0x00007ffff2795832 in NotifyOffThreadScriptCompletedRunnable::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/dom/xul/nsXULElement.cpp:2742
#61 0x00007ffff045c920 in nsThread::ProcessNextEvent (this=<optimized out>, aMayWait=<optimized out>, aResult=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/threads/nsThread.cpp:830
#62 0x00007ffff048d59d in NS_ProcessNextEvent (aThread=<optimized out>, aMayWait=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/xpcom/glue/nsThreadUtils.cpp:265
#63 0x00007ffff08bd813 in mozilla::ipc::MessagePump::Run (this=<optimized out>, aDelegate=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/glue/MessagePump.cpp:99
#64 0x00007ffff087950c in MessageLoop::RunInternal (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:233
#65 0x00007ffff08795a8 in RunHandler (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:226
#66 MessageLoop::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/ipc/chromium/src/base/message_loop.cc:200
#67 0x00007ffff2830de9 in nsBaseAppShell::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/widget/nsBaseAppShell.cpp:164
#68 0x00007ffff33c9973 in nsAppStartup::Run (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/components/startup/nsAppStartup.cpp:281
#69 0x00007ffff344d957 in XREMain::XRE_mainRun (this=<optimized out>)
    at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4098
#70 0x00007ffff344ed48 in XREMain::XRE_main (this=<optimized out>, argc=<optimized out>, argv=<optimized out>, 
    aAppData=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4174
#71 0x00007ffff344f0d1 in XRE_main (argc=<optimized out>, argv=<optimized out>, aAppData=<optimized out>, 
    aFlags=<optimized out>) at /REF-COMM-CENTRAL/comm-central/mozilla/toolkit/xre/nsAppRunner.cpp:4394
#72 0x000000000040394a in do_main (argc=argc@entry=1, argv=argv@entry=0x7fffffffda08, xreDirectory=0x4232a0)
    at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:195
#73 0x0000000000403211 in main (argc=1, argv=0x7fffffffda08) at /REF-COMM-CENTRAL/comm-central/mail/app/nsMailApp.cpp:380
(gdb) list
1231	nsMsgAccountManager::LoadAccounts()
1232	{
1233	  nsresult rv;
1234	
1235	  // for now safeguard multiple calls to this function
1236	  if (m_accountsLoaded) {
1237	#if DEBUG
1238	    NS_WARNING("LoadAccounts called multiple times. returning NS_OK");
1239	#endif
1240	    return NS_OK;
(gdb) quit


So except for the failure to print optimized-out variables, and the failure to dump
traceback after TB crashes during |make mozmill|, it seems to work more or less.

TIA

Comment 30

[Tom Tromey :tromey]

(In reply to ISHIKAWA, Chiaki from comment #29)

> With the combination of -gsplit-dwarf and ld.gold, it can be done in 32-bit
> space.

Yeah.  I have a "big" machine.

> > and gdb "attach" times are too.
> > The latter was quite surprising to me, I thought that the index
> > would make more difference.
> 
> It is.
> There was a visible difference on my PC.

Well, what is odd here is that, for me, -gsplit-dwarf plus the index
does not help gdb at all.

However, if I use the ordinary -g and then make an index later
(using gdb-add-index), then this notably helps gdb -- the initial "attach"
time goes from ~15 seconds to ~2 seconds.

So, either I did something wrong making the index in the split case,
or there is something bad about the split+index case.

> This requires a little investigation.
> I wonder which version of 
> C compiler and linker
> were used. 
> OS, and CPU.
> Any special options passed to C compiler or linker?

I'm using x86-64 Fedora 20.
I have the default linker set to gold.
I used -gsplit-dwarf and -Wl,--gdb-index.

pokyo. gcc --version
gcc (GCC) 4.8.3 20140911 (Red Hat 4.8.3-7)

pokyo. ld --version
GNU gold (version 2.23.2) 1.11


> Is your tree new version that contains
> the patch from fixed
> Bug 1081682 - ccache "-with-ccache=/usr/bin/ccache" is not used properly any
> more in build (both C-C and M-C) 

I think I am not using ccache.

> One more thing I noticed: it could be that the latest gcc 4.9 is very eager
> with
> -O2 and gdb does not print variable values anymore saying that
> they are optimized out. I think I need to use -O or something instead.
> (I wonder if there is a bug in the gdb that comes with experimental
> release of Debian, though.)

It's always hard to say about this kind of thing in the abstract.
You have to dig into the DWARF to see for sure.
E.g., there was a recent VTA bug fix in gcc which could affect debuginfo generation
in this case.  And while "-g -O2" has improved a lot in recent years,
it will never be perfect as there are diminishing returns to that kind of work.

Comment 31

[(not currently active) Ted Mielczarek]

Chromium is using this by default since last year:
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/8tKb9-Wr8gk

Comment 32

[ISHIKAWA, Chiaki]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #31)
> Chromium is using this by default since last year:
> https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/8tKb9-
> Wr8gk

This is interesting.
I even noticed that the article mentioned a few remaining issues:

> * Issue 439320: linux_use_debug_fission causes "optimized out" error trying to print 
> variables in gdb -- https://code.google.com/p/chromium/issues/detail?id=439320

This is exactly what happened to me several months ago. I had to downgrade optimization to -O0, but I don't think the earlier gcc/gdb combination needed -O0 for debugging: I could access many variables, etc. with gdb. So I suspect a very subtle issue
of GCC/GDB and the GDB's debug information generation (when -gsplit-dwarf is used and not.). But I digress.

Also, in the comment 
https://code.google.com/p/chromium/issues/detail?id=352046#c52

>Since -gsplit-dwarf breaks some extremely useful build tools (eg ccache and icecc), it >deserves its own gyp variable.  This setting replaces the previous hack, setting 
> binutils_version=0.

At least, ccache patch to handle -gsplit-dwarf is posted at 
https://bugzilla.samba.org/show_bug.cgi?id=10005
and it works beautifully (I have been using it more than a year)
and I hope others also test it.
I  hope that the patch will make into the main ccache git repository before summer.
But maybe I am too optimistic.

Someone's positive test result would encourage the merging of the patch to the main ccache git repository.

TIA

Comment 33

[ISHIKAWA, Chiaki]

Great news.

Finally, "-gsplit-dwarf" support is officially in ccache's git repository.
https://bugzilla.samba.org/show_bug.cgi?id=10005#c50

I would encourage people who perform local build and
debugging under linux to to try out the latest ccache (fetched from ccache git repository) and report any bugs that may have crept during the merge of ccache support.

How to obtain git repository version of ccache:
https://ccache.samba.org/repo.html

% git clone git://git.samba.org/ccache.git

Or

% git clone git://git.samba.org/ccache.git your-ccache-directory-name

Comment 34

[ISHIKAWA, Chiaki]

I forgot:
After fetching the git repository, you have to

./autogen.sh

The above step is crucial. The shell script above will run
autoheader
autoconf

and so you need to have the above tools installed.

Then

./configure
make

and then copy ccache to /usr/bin/ccahe or whatever your linux distribution would put the  official ccache.
If you have installed the linux's distribution of official ccache, you may want to
do something like (modulo different directory)
assuming you are inside the ccache source directory

mv /usr/bin/ccache /usr/bin/ccache.distro
mv ./ccache /usr/bin/ccache

TIA