42527 - linux not builds not always getting good symbolic info in talkback database

Status: VERIFIED FIXED

(Core Graveyard :: Talkback Client, defect, P2)

Comment 1

[Shiva Thirumazhusai]

We(I, leaf and granrose) have identified the problem. leaf and granrose are 
working on the fix. 

It looks like the symbols from mozilla builds are stripped before copied on to 
CLIMATE(symbols repository). 

Assigning to Granrose.

Comment 2

[Shiva Thirumazhusai]

cc-ing leger and huang.

Comment 3

[Daniel (Leaf) Nunes]

Jon has a fix for automation theorized. Just need to implement.

Comment 4

[leger]

Putting on [dogfood+] radar.

Comment 5

[lchiang]

adding myself to cc: list. Once this is fixed, I need to know to retest a bug 
for alec.

Comment 6

[Jon Granrose]

I've finished hacking the automation and am doing a test run now.  With any luck 
it will work, and I can turn it on tomorrow so we will have symbols again 
starting with tomorrow's 8pm automatic build.

I have also turned on the linux nscp6 installer as well.

Comment 7

[Jon Granrose]

test run worked more or less.  you can try the linux bits at
ftp://sweetlou/products/client/seamonkey/unix/linux/2.2/x86/2000-06-14-16-M17/
and see if they give you a stack trace in talkback that you can use.

I changed the automation to save the various files with the symbols, and it
saved most of them.  However, the following files in the symbol directory had no
symbols: 

libcmt.so: no symbols
libgtksuperwin.so: no symbols
libgtkxtbin.so: no symbols
libjpeg.so: no symbols
libmozjs.so: no symbols
libnspr4.so: no symbols
libplc4.so: no symbols
libplds4.so: no symbols
libprotocol.so: no symbols
libXpcs.so: no symbols
libXprt.so: no symbols
libXptl.so: no symbols
libzlib.so: no symbols
mozilla-bin.elf: no symbols

I'm looking into this further, especially the mozilla-bin.elf.

cc'ing wtc for input on what it takes to get nspr symbols.

Comment 8

[Jon Granrose]

oh, and I'm doing another test run now.  if all goes well, I should be able to
turn this on for all future linux builds (barring respins, next scheduled build
is 8pm tonight).

Comment 9

[Wan-Teh Chang]

I don't have time to look into this soon.
You want an optimized build with debug
symbols, right?  This probably just
requires that -g and -O2 be used together.
Then mozilla/configure needs to tell nspr
to build itself this way.

Comment 10

[Jon Granrose]

ok, test builds completed successfully and automation is delivering symbols 
again.  looking at the symbols uploaded last night, the only files missing 
symbols are the NSPR libraries:

libnspr4.so: no symbols
libplc4.so: no symbols
libplds4.so: no symbols

I'm tempted to resolve/fix this bug, but cc'ing cls for any input on how easy it 
would be to get NSPR reporting symbols for talkback reports based on wtc's 
comments above.

Comment 11

[cls]

wtc's suggestion is not as simple as it sounds.  Currently, we do not pass any
CC or CFLAGS settings from mozilla's configure to the classic NSPR build (see
bug #34759).  NSPR's classic build sets its own optimization flags.  
The easiset way to override those settings would be to use
--enable-nspr-autoconf and set CFLAGS & CXXFLAGS when you configure mozilla.

Comment 12

[Jon Granrose]

Ok, thanks for the into.  In that case, I'm punting.  Resolved/fixed.

If it turns out we need NSPR symbols in talkback at some point in the future, we 
can reopen this bug, or file a new one to address the problem.

Comment 13

[leger]

lchiang, you can verify now :-)

Comment 14

[Karen Huang]

Following is what I got for today's Linux morning commercial build crash when
exit from the browser:

Stack Trace

nsCAimDataSource::Initialize() 
nsCAimDataSource::FinalConstruct() 
nsCAimDataSource::Create() 
nsCCoolGlue::AimDataSource() 
NotifyChildrenOfStateChange() 
nsCAimSession::Initialize() 
nsCAimSession::FinalConstruct() 
nsCAimSession::Create() 
nsCCoolGlue::AimSession() 
nsCAimAdminManager::Uninitialize() 
nsCAimAdminManager::~nsCAimAdminManager() 
Release() 
nsCOMPtr_base::assign_with_AddRef() 
nsCCoolGlue::Uninitialize() 
nsCCoolGlue::~nsCCoolGlue() 
Release() 
nsCOMPtr_base::assign_with_AddRef() 
nsCIMManager::Uninitialize() 
nsCIMManager::~nsCIMManager() 
nsCIMManager::Release() 
DeleteEntry() 
_hashEnumerateRemove__FP11PLHashEntryiPv() 
libplds4.so + 0x1410 (0x4015c410) 
nsHashtable::Reset() 
nsObjectHashtable::Reset() 
nsObjectHashtable::~nsObjectHashtable() 
nsServiceManagerImpl::~nsServiceManagerImpl() 
nsServiceManagerImpl::Release() 
nsServiceManager::ShutdownGlobalServiceManager() 
NS_ShutdownXPCOM() 
main() 
libc.so.6 + 0x17cb3 (0x40241cb3) 

Comment 15

[lchiang]

I'd have to get a crash to have this verified!  :-)
But, it is working per Karen's stack trace.  Also, the other day, Esther got a 
Linux stack trace with some good info it for alec to track down.

Comment 16

[Karen Huang]

Can somebody confirm whether the talkback report for Linux is fixed completely 
or not?
I compare the talkback report from debug build & commercial build, it seems that 
the talk back report from commercial build is not displaying enough 
information....can somebody confirm for reopening this bug or explain why there 
are differences between these two talk back reports for the same crash: 

Stack Trace from Linux commercial build:

nsEventStateManager::PreHandleEvent() 
PresShell::HandleEventInternal() 
PresShell::HandleEvent() 
nsView::HandleEvent() 
nsView::HandleEvent() 
nsView::HandleEvent() 
nsViewManager2::DispatchEvent() 
HandleEvent() 
nsWidget::DispatchEvent() 
nsWidget::DispatchWindowEvent() 
nsWidget::DispatchFocus() 
nsWindow::SetFocus() 
GlobalWindowImpl::Focus() 
nsWebShellWindow::HandleEvent() 
nsWidget::DispatchEvent() 
handle_mozarea_focus_in() 
libgtk-1.2.so.0 + 0xf4229 (0x40714229) 
libgtk-1.2.so.0 + 0xb965d (0x406d965d) 
libgtk-1.2.so.0 + 0xb8ab2 (0x406d8ab2) 
libgtk-1.2.so.0 + 0xb6c05 (0x406d6c05) 
libgtk-1.2.so.0 + 0xeb9d8 (0x4070b9d8) 
libgtk-1.2.so.0 + 0xf3d1b (0x40713d1b) 
libgtk-1.2.so.0 + 0xf4485 (0x40714485) 
libgtk-1.2.so.0 + 0xb8aeb (0x406d8aeb) 
libgtk-1.2.so.0 + 0xb6c05 (0x406d6c05) 
libgtk-1.2.so.0 + 0xf1138 (0x40711138) 
libgtk-1.2.so.0 + 0xec4f2 (0x4070c4f2) 
libgtk-1.2.so.0 + 0xf465d (0x4071465d) 
libgtk-1.2.so.0 + 0xb8aeb (0x406d8aeb) 
libgtk-1.2.so.0 + 0xb6c05 (0x406d6c05) 
libgtk-1.2.so.0 + 0xec2e8 (0x4070c2e8) 
handle_toplevel_focus_in() 
libgtk-1.2.so.0 + 0xf4229 (0x40714229) 
libgtk-1.2.so.0 + 0xb965d (0x406d965d) 
libgtk-1.2.so.0 + 0xb8ab2 (0x406d8ab2) 
libgtk-1.2.so.0 + 0xb6c05 (0x406d6c05) 
libgtk-1.2.so.0 + 0xeb9d8 (0x4070b9d8) 
libgtk-1.2.so.0 + 0x8be1a (0x406abe1a) 
handle_gdk_event() 
libgdk-1.2.so.0 + 0x170fb (0x407540fb) 
libglib-1.2.so.0 + 0xfa86 (0x4077ea86) 
libglib-1.2.so.0 + 0x10041 (0x4077f041) 
libglib-1.2.so.0 + 0x101e1 (0x4077f1e1) 
libgtk-1.2.so.0 + 0x8b7a9 (0x406ab7a9) 
nsAppShell::Run() 
nsAppShellService::Run() 
main1() 
main() 
libc.so.6 + 0x17cb3 (0x40241cb3) 

stack trace for debug build

#0  0x415c9ed2 in nsEventStateManager::PreHandleEvent (this=0x8951bc0, 
aPresContext=0x8827fc0, aEvent=0xbfffe668, aTargetFrame=0x894ffac, 
aStatus=0xbfffe594, aView=0x8876b10) at nsEventStateManager.cpp:456
#1  0x4162d195 in PresShell::HandleEventInternal (this=0x88a8458, 
aEvent=0xbfffe668, aView=0x8876b10, aStatus=0xbfffe594) at nsPresShell.cpp:3888
#2  0x4162ce59 in PresShell::HandleEvent (this=0x88a8458, aView=0x8876b10, 
aEvent=0xbfffe668, aEventStatus=0xbfffe594, aHandled=@0xbfffe538) at 
nsPresShell.cpp:3829
#3  0x41d35377 in nsView::HandleEvent (this=0x8876b10, event=0xbfffe668, 
aEventFlags=8, aStatus=0xbfffe594, aHandled=@0xbfffe538) at nsView.cpp:769
#4  0x41d35300 in nsView::HandleEvent (this=0x88767a0, event=0xbfffe668, 
aEventFlags=8, aStatus=0xbfffe594, aHandled=@0xbfffe538) at nsView.cpp:753
#5  0x41d35300 in nsView::HandleEvent (this=0x86e4bc8, event=0xbfffe668, 
aEventFlags=28, aStatus=0xbfffe594, aHandled=@0xbfffe538) at nsView.cpp:753
#6  0x41d48553 in nsViewManager2::DispatchEvent (this=0x86e4ac0, 
aEvent=0xbfffe668, aStatus=0xbfffe594) at nsViewManager2.cpp:1387
#7  0x41d32db4 in HandleEvent (aEvent=0xbfffe668) at nsView.cpp:68

The top of the stack trace for the assertion is:

#0  nsDebug::Assertion (aStr=0x805f1e0 "You can't dereference a NULL nsCOMPtr 
with operator->().", aExpr=0x805f1c9 "mRawPtr != 0", aFile=0x805f1ab 
"../../dist/include/nsCOMPtr.h", aLine=649) at nsDebug.cpp:184
#1  0x4012067d in nsDebug::PreCondition (aStr=0x805f1e0 "You can't dereference a 
NULL nsCOMPtr with operator->().", aExpr=0x805f1c9 "mRawPtr != 0", 
aFile=0x805f1ab "../../dist/include/nsCOMPtr.h", aLine=649) at nsDebug.cpp:342
#2  0x805cfba in nsCOMPtr<nsIDOMWindow>::operator-> (this=0xbfffe274) at 
../../dist/include/nsCOMPtr.h:649
#3  0x415c9eb5 in nsEventStateManager::PreHandleEvent (this=0x8947848, 
aPresContext=0x87f3298, aEvent=0xbfffe688, aTargetFrame=0x8943e74, 
aStatus=0xbfffe5b4, aView=0x8945e30) at nsEventStateManager.cpp:456
#4  0x4162d195 in PresShell::HandleEventInternal (this=0x87ff688, 
aEvent=0xbfffe688, aView=0x8945e30, aStatus=0xbfffe5b4) at nsPresShell.cpp:3888
#5  0x4162ce59 in PresShell::HandleEvent (this=0x87ff688, aView=0x8945e30, 
aEvent=0xbfffe688, aEventStatus=0xbfffe5b4, aHandled=@0xbfffe558) at 
nsPresShell.cpp:3829
#6  0x41d35377 in nsView::HandleEvent (this=0x8945e30, event=0xbfffe688, 
aEventFlags=8, aStatus=0xbfffe5b4, aHandled=@0xbfffe558) at nsView.cpp:769
#7  0x41d35300 in nsView::HandleEvent (this=0x880ab18, event=0xbfffe688, 
aEventFlags=8, aStatus=0xbfffe5b4, aHandled=@0xbfffe558) at nsView.cpp:753
#8  0x41d35300 in nsView::HandleEvent (this=0x8946898, event=0xbfffe688, 
aEventFlags=28, aStatus=0xbfffe5b4, aHandled=@0xbfffe558) at nsView.cpp:753
#9  0x41d48553 in nsViewManager2::DispatchEvent (this=0x8946790, 
aEvent=0xbfffe688, aStatus=0xbfffe5b4) at nsViewManager2.cpp:1387
#10 0x41d32db4 in HandleEvent (aEvent=0xbfffe688) at nsView.cpp:68

Comment 17

[Jon Granrose]

that's to be expected as the verification builds are not debug builds, so you 
won't get all the debug symbols.  the thing to do is compare win32 stack traces 
with linux stack traces and see if they have the same level of detail.

to the best of my knowledge this bug is resolved for all things but NSPR which 
we aren't going to tackle at this point.

Comment 18

[chris hofmann]

Shiva,  can you check to see if we have ever had line numbers?
mcafee might recall if we are up against some OS limitation...

Comment 19

[Shiva Thirumazhusai]

Communicator 4.5,4.6x,4.7x all of them for LinuxIntel provides only the
the stack trace. NO source file/line no information available. 

Comment 20

[Shiva Thirumazhusai]

Communicator 4.5,4.6x,4.7x all of them for SunSparc provides only the
the stack trace. NO source file/line no information available. 

Comment 21

[Alec Flett]

these stack traces look good, but can we get the libgtk symbols too? they 
shouldn't be hard to get - they aren't part of seamonkey, but can we tell 
talkback to include libglib/libgtk to treat these like system libraries, and 
extract the symbols for them too?

Comment 22

[Daniel (Leaf) Nunes]

Hmm. Is just tossing /usr/lib/libgtk-1.2.so.0.5.0, or whatever version is on the
build system, onto the talkback server enough?

Comment 23

[Shiva Thirumazhusai]

ccing wu.liu from epeople (formerly fullcircle, nowonder) for more information.