450485 - thunderbird trunk nightly w32 buildsymbols broken on comm-central so no useful breakpad report - symbols missing at http://symbols.mozilla.org/thunderbird

Status: VERIFIED FIXED

(MailNews Core :: Build Config, defect)

Description

[Wayne Mery (:wsmwk)]

I crashed today and was unable to load symbol store. ... thunderbird trunk nightly w32 buildsymbols broken on comm-central broken - missing at http://symbols.mozilla.org/thunderbird 

per Ted, also seamonkey, possibly only w32 ... "look at the build log, see if anything funny shows up under the "buildsymbols" 

<ted>	if you know the buildid, you can check
<ted>	there's always a .txt file that lists the symbols
<ted>	http://symbols.mozilla.org/thunderbird/thunderbird-3.0b1pre-WINNT-20080813031200-symbols.txt
<ted>	there totally isn't a thunderbird.pdb in there
<wsmwk>	ted: that's the one
<ted>	have a look at the build log, see if anything funny shows up under the "buildsymbols" step?
<wsmwk>	ted: would there be a related reason why breakpad didn't trigger from my crash? crash threw me into MS dialog
<ted>	not afaik
<ted>	you know, i think buildsymbols might be kind of broken for you
<ted>	since the move to comm-central
<wsmwk>	could be :)
<ted>	you should file a bug on that
<ted>	i think it only scans the moz objdir for pdb files
<ted>	so it won't ever find thunderbird.pdb

Comment 1

[Joe Sabash [:JoeS1]]

(In reply to comment #0)

> <wsmwk> ted: would there be a related reason why breakpad didn't trigger from
> my crash? crash threw me into MS dialog
> <ted>   not afaik

I don't think breakpad is working at all in current trunk.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a2pre) Gecko/20080813031200 Shredder/3.0b1pre ID:20080813031200

I have a few reproducable crashes, if I can confirm, I'll open a new bug.

Comment 2

[Joe Sabash [:JoeS1]]

Opened bug 450519 for the lack of breakpad reports

Comment 3

[Philippe M. Chiasson (:gozer)]

It's busted for at least 2 different reasons. Slightly unrelated, but on Linux, dump_syms crashes on various things, like thunderbird-bin. When this happens, symbols are simply skipped for that particular object. See Bug 445090.

On Win32, it's a very simple problem, the symbol dumping code needs to find .pdb files everywhere within the objdir, and unfortunately, it now runs within objdir/mozilla and isn't seeing all .pdb files. 

The offending line is this one:

http://mxr.mozilla.org/comm-central/source/mozilla/Makefile.in#157

And for our usage, replacing '.' with '..' would work, but the real solution is to use $OBJDIR or something similar that will work both with mozilla-central alone, and with the comm-central/mozilla-central nesting setup of Thunderbird.

A possible temporary fix would be to run

$> make buildsymbols MAKE_SYM_STORE_PATH=..

Until the buildsystem is fixed.

I've applied that change temporarly on the nightly buildbot master, lets see what happens to tonight symbols...

Comment 4

[Robert Kaiser]

(In reply to comment #3)
> $> make buildsymbols MAKE_SYM_STORE_PATH=..

The correct solution might be to create a makefile target in comm-central for this and call the mozilla/ makefile with this variable override.

Comment 5

[Philippe M. Chiasson (:gozer)]

Attachment: [checked in] comm-central buildsymbols fixup

Something like this would do the trick, like recommended in comment #4.

However, doesn't that mean that everything that builds a project like comm-central does, with a mozilla-central checkout one directory down have to replicate this code ?

Feels too me like mozilla-central's make file shouldn't use '.' in specifying where to find binaries, but rather use a more precise make variable that we could overried more easily. MAKE_SYM_STORE_PATH feels like internal details that might change in the future.

Comment 6

[Robert Kaiser]

(In reply to comment #5)
> However, doesn't that mean that everything that builds a project like
> comm-central does, with a mozilla-central checkout one directory down have to
> replicate this code ?

True, if you call this "code" at all ;-)
I don't expect too many projects building up something like comm-central though.

I'd add a comment there pointing to this bug to tell why this was done, but else it looks good to me - if it actually works.

Comment 7

[Philippe M. Chiasson (:gozer)]

Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

I've tested it on my own build, and it worked as expected, really only making a difference on win32.

Comment 8

[Robert Kaiser]

Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

Looks good to me. I neither have win32 nor a build with crashreporter enabled around, so I can't actually test, but I trust what you said and it looks good from code inspection.

Comment 9

[Philippe M. Chiasson (:gozer)]

Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

http://hg.mozilla.org/comm-central/index.cgi/rev/138

changeset:   138:418432362df6
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Mon Aug 18 15:27:39 2008 -0400

Comment 10

[Wayne Mery (:wsmwk)]

I don't have a crash to test, but looks good for 20080819 build.

http://symbols.mozilla.org/thunderbird/thunderbird-3.0b1pre-WINNT-20080819031251-symbols.txt
includes the lines
thunderbird.pdb/A405399E4AB4459B8D6992C9DCEAC8061/thunderbird.sym
thunderbird.pdb/A405399E4AB4459B8D6992C9DCEAC8061/thunderbird.pdb

whether the symbols are in fact available remains to be seen

Comment 11

[Philippe M. Chiasson (:gozer)]

See comment #10. Somebody with Windows and a reproducible way to crash Thunderbird should be able to verify whether the debugging symbols for thunderbird.exe are pulled in correctly, as it used to.

Comment 12

[Wayne Mery (:wsmwk)]

Sorry, I still can't confirm.  

With this bug 450519 fixed - No breakpad reports since comm-central move - crash-stats should be getting w32 and linux crashes again. But after 4 hours I still have no response to view a crash I induced at 8am this morning, so I can't confirm.  http://crash-stats.mozilla.com/report/pending/512cc60f-74f8-11dd-bb58-001a4bd43ed6

Comment 13

[Wayne Mery (:wsmwk)]

symbols should be there, but whether they are working is another matter.  Due to  no response from crash-stats we must await fixing of bug 422908

BTW, Ted's crashme extension can be used to force crash in Thunderbird, when breakpad is enabled for the build :)

crashme @ https://wiki.mozilla.org/QA/Firefox3/TestPlan/Breakpad#References

Comment 14

[Wayne Mery (:wsmwk)]

symbols found for http://crash-stats.mozilla.com/report/pending/512cc60f-74f8-11dd-bb58-001a4bd43ed6 (some hours later)

FIXED

Comment 15

[Philippe M. Chiasson (:gozer)]

This turns out to be the wrong thing to do for OS X Universal builds.

In that case, MAKE_SYM_STORE_PATH should be set to dist/universal, as can be seen here: http://mxr.mozilla.org/mozilla-central/source/Makefile.in#164

The fix to comm-central/Makefile.in should probably look a little like this:

+MAKE_SYM_STORE_PATH := ..
+ifeq ($(OS_ARCH),Darwin)
+ifdef UNIVERSAL_BINARY
+    MAKE_SYM_STORE_PATH := dist/universal
+endif
+endif
+
 # http://bugzilla.mozilla.org/show_bug.cgi?id=450485
 buildsymbols ::
-       $(MAKE) -C mozilla $@ MAKE_SYM_STORE_PATH=..
+       $(MAKE) -C mozilla $@ MAKE_SYM_STORE_PATH=$(MAKE_SYM_STORE_PATH)


I haven't tested this, and I am not sure all these variables are available at that point, OS_ARCH, UNIVERSAL_BINARY, but that's the idea of the fix.

Comment 16

[Philippe M. Chiasson (:gozer)]

Attachment: process universal binaries when generating buildsymbols on OS X [Checkin: Comment 19]

Finally gotten around to writing/testing a proper patch.

This patch limits symbols extraction to the final universal binaries.

Comment 17

[Robert Kaiser]

Comment on attachment 344484 [details] [diff] [review]
process universal binaries when generating buildsymbols on OS X
[Checkin: Comment 19]

r=me by code inspection, I take it that you tested this, and it looks reasonable.

Comment 18

[Philippe M. Chiasson (:gozer)]

(In reply to comment #17)
> (From update of attachment 344484 [details] [diff] [review])
> r=me by code inspection, I take it that you tested this,

Yes, both on OSX Universal and Linux without breakage

> and it looks reasonable.

Thanks!

Comment 19

[Serge Gautherie (:sgautherie)]

Comment on attachment 344484 [details] [diff] [review]
process universal binaries when generating buildsymbols on OS X
[Checkin: Comment 19]

http://hg.mozilla.org/comm-central/rev/5709136d4b9c

Comment 20

[Philippe M. Chiasson (:gozer)]

The patch in attachment 344484 [details] [diff] [review] is responsible for bug 464247. The sub-make will not inherit the value of MAKE_SYM_STORE_PATH and will be back to using mozilla's defaults.

The proper fix is to explicitely pass this into mozilla's sub-make invocation.

The only case where this would have worked is if MAKE_SYM_STORE_PATH was set in the environemnt or passed by the make command-line.

Comment 21

[Philippe M. Chiasson (:gozer)]

Attachment: Pass MAKE_SYM_STORE_PATH down to mozilla's buildsymbols target

Explicitely passing down the value for MAKE_SYM_STORE_PATH we've decided on down to mozilla's buildsymbols target is required.

Comment 22

[Robert Kaiser]

Comment on attachment 348790 [details] [diff] [review]
Pass MAKE_SYM_STORE_PATH down to mozilla's buildsymbols target

sounds logical to me, I wondered myself if that was the problem when looking at the last patch. r=me by code inspection.

Comment 23

[Philippe M. Chiasson (:gozer)]

What must have hapenned is that I've either tested this with MAKE_SYM_STORE_PATH in my environment, or invoked make directly with it, possibly. Otherwise, not entirely sure why it would have worked quite right, even on linux.

Comment 24

[Philippe M. Chiasson (:gozer)]

http://hg.mozilla.org/comm-central/rev/be77b2aa036b

changeset:   1143:be77b2aa036b
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Tue Nov 18 14:01:02 2008 -0500
summary:     Bug 450485. Make sure the value of MAKE_SYM_STORE_PATH is passed to mozilla's buildsymbols target. r=KaiRo

Comment 25

[Wayne Mery (:wsmwk)]

yay!  we have w32 symbols.  I haven't checked Mac.
thanks gozer

crashme extension results

bp-13f2c778-b6cf-475b-bada-ef2620081119 build 20081118031447
crashme.dll	nsCrasher::Crash	nsCrasher.cpp:21
xpcom_core.dll	NS_InvokeByIndex_P	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:101
thunderbird.exe	thunderbird.exe@0x9aab0	

bp-2cd19882-840c-4fba-a8ac-739720081119 buildid 20081119032441
crashme.dll	nsCrasher::Crash	nsCrasher.cpp:21
xpcom_core.dll	NS_InvokeByIndex_P	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:101
thunderbird.exe	XPCWrappedNative::CallMethod	js/src/xpconnect/src/xpcwrappednative.cpp:2422
thunderbird.exe	XPC_WN_CallMethod	js/src/xpconnect/src/xpcwrappednativejsops.cpp:1477