thunderbird trunk nightly w32 buildsymbols broken on comm-central so no useful breakpad report - symbols missing at http://symbols.mozilla.org/thunderbird

VERIFIED FIXED in mozilla1.9.1b2

Status

VERIFIED FIXED
11 years ago
10 years ago

People

(Reporter: wsmwk, Assigned: gozer)

Tracking

Trunk
mozilla1.9.1b2
Bug Flags:
in-testsuite -

Firefox Tracking Flags

(Not tracked)

Details

(URL)

Attachments

(3 attachments)

(Reporter)

Description

11 years ago
I crashed today and was unable to load symbol store. ... thunderbird trunk nightly w32 buildsymbols broken on comm-central broken - missing at http://symbols.mozilla.org/thunderbird 

per Ted, also seamonkey, possibly only w32 ... "look at the build log, see if anything funny shows up under the "buildsymbols" 

<ted>	if you know the buildid, you can check
<ted>	there's always a .txt file that lists the symbols
<ted>	http://symbols.mozilla.org/thunderbird/thunderbird-3.0b1pre-WINNT-20080813031200-symbols.txt
<ted>	there totally isn't a thunderbird.pdb in there
<wsmwk>	ted: that's the one
<ted>	have a look at the build log, see if anything funny shows up under the "buildsymbols" step?
<wsmwk>	ted: would there be a related reason why breakpad didn't trigger from my crash? crash threw me into MS dialog
<ted>	not afaik
<ted>	you know, i think buildsymbols might be kind of broken for you
<ted>	since the move to comm-central
<wsmwk>	could be :)
<ted>	you should file a bug on that
<ted>	i think it only scans the moz objdir for pdb files
<ted>	so it won't ever find thunderbird.pdb
(In reply to comment #0)

> <wsmwk> ted: would there be a related reason why breakpad didn't trigger from
> my crash? crash threw me into MS dialog
> <ted>   not afaik

I don't think breakpad is working at all in current trunk.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a2pre) Gecko/20080813031200 Shredder/3.0b1pre ID:20080813031200

I have a few reproducable crashes, if I can confirm, I'll open a new bug.
Opened bug 450519 for the lack of breakpad reports
(Assignee)

Comment 3

11 years ago
It's busted for at least 2 different reasons. Slightly unrelated, but on Linux, dump_syms crashes on various things, like thunderbird-bin. When this happens, symbols are simply skipped for that particular object. See Bug 445090.

On Win32, it's a very simple problem, the symbol dumping code needs to find .pdb files everywhere within the objdir, and unfortunately, it now runs within objdir/mozilla and isn't seeing all .pdb files. 

The offending line is this one:

http://mxr.mozilla.org/comm-central/source/mozilla/Makefile.in#157

And for our usage, replacing '.' with '..' would work, but the real solution is to use $OBJDIR or something similar that will work both with mozilla-central alone, and with the comm-central/mozilla-central nesting setup of Thunderbird.

A possible temporary fix would be to run

$> make buildsymbols MAKE_SYM_STORE_PATH=..

Until the buildsystem is fixed.

I've applied that change temporarly on the nightly buildbot master, lets see what happens to tonight symbols...

Comment 4

11 years ago
(In reply to comment #3)
> $> make buildsymbols MAKE_SYM_STORE_PATH=..

The correct solution might be to create a makefile target in comm-central for this and call the mozilla/ makefile with this variable override.
(Assignee)

Comment 5

11 years ago
Created attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

Something like this would do the trick, like recommended in comment #4.

However, doesn't that mean that everything that builds a project like comm-central does, with a mozilla-central checkout one directory down have to replicate this code ?

Feels too me like mozilla-central's make file shouldn't use '.' in specifying where to find binaries, but rather use a more precise make variable that we could overried more easily. MAKE_SYM_STORE_PATH feels like internal details that might change in the future.

Comment 6

11 years ago
(In reply to comment #5)
> However, doesn't that mean that everything that builds a project like
> comm-central does, with a mozilla-central checkout one directory down have to
> replicate this code ?

True, if you call this "code" at all ;-)
I don't expect too many projects building up something like comm-central though.

I'd add a comment there pointing to this bug to tell why this was done, but else it looks good to me - if it actually works.
(Assignee)

Comment 7

11 years ago
Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

I've tested it on my own build, and it worked as expected, really only making a difference on win32.
Attachment #334217 - Flags: review?(kairo)

Comment 8

11 years ago
Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

Looks good to me. I neither have win32 nor a build with crashreporter enabled around, so I can't actually test, but I trust what you said and it looks good from code inspection.
Attachment #334217 - Flags: review?(kairo) → review+
(Assignee)

Comment 9

11 years ago
Comment on attachment 334217 [details] [diff] [review]
[checked in] comm-central buildsymbols fixup

http://hg.mozilla.org/comm-central/index.cgi/rev/138

changeset:   138:418432362df6
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Mon Aug 18 15:27:39 2008 -0400
Attachment #334217 - Attachment description: comm-central buildsymbols fixup → [checked in] comm-central buildsymbols fixup
(Reporter)

Comment 10

11 years ago
I don't have a crash to test, but looks good for 20080819 build.

http://symbols.mozilla.org/thunderbird/thunderbird-3.0b1pre-WINNT-20080819031251-symbols.txt
includes the lines
thunderbird.pdb/A405399E4AB4459B8D6992C9DCEAC8061/thunderbird.sym
thunderbird.pdb/A405399E4AB4459B8D6992C9DCEAC8061/thunderbird.pdb

whether the symbols are in fact available remains to be seen
(Assignee)

Comment 11

11 years ago
See comment #10. Somebody with Windows and a reproducible way to crash Thunderbird should be able to verify whether the debugging symbols for thunderbird.exe are pulled in correctly, as it used to.
Keywords: qawanted
(Reporter)

Comment 12

11 years ago
Sorry, I still can't confirm.  

With this bug 450519 fixed - No breakpad reports since comm-central move - crash-stats should be getting w32 and linux crashes again. But after 4 hours I still have no response to view a crash I induced at 8am this morning, so I can't confirm.  http://crash-stats.mozilla.com/report/pending/512cc60f-74f8-11dd-bb58-001a4bd43ed6
(Reporter)

Comment 13

11 years ago
symbols should be there, but whether they are working is another matter.  Due to  no response from crash-stats we must await fixing of bug 422908

BTW, Ted's crashme extension can be used to force crash in Thunderbird, when breakpad is enabled for the build :)

crashme @ https://wiki.mozilla.org/QA/Firefox3/TestPlan/Breakpad#References
Depends on: 422908
Keywords: qawanted
Summary: thunderbird trunk nightly w32 buildsymbols broken on comm-central broken - missing at http://symbols.mozilla.org/thunderbird → thunderbird trunk nightly w32 buildsymbols broken on comm-central so no useful breakpad report - symbols missing at http://symbols.mozilla.org/thunderbird
(Reporter)

Comment 14

11 years ago
symbols found for http://crash-stats.mozilla.com/report/pending/512cc60f-74f8-11dd-bb58-001a4bd43ed6 (some hours later)

FIXED
No longer depends on: 422908
(Reporter)

Updated

11 years ago
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
(Assignee)

Comment 15

10 years ago
This turns out to be the wrong thing to do for OS X Universal builds.

In that case, MAKE_SYM_STORE_PATH should be set to dist/universal, as can be seen here: http://mxr.mozilla.org/mozilla-central/source/Makefile.in#164

The fix to comm-central/Makefile.in should probably look a little like this:

+MAKE_SYM_STORE_PATH := ..
+ifeq ($(OS_ARCH),Darwin)
+ifdef UNIVERSAL_BINARY
+    MAKE_SYM_STORE_PATH := dist/universal
+endif
+endif
+
 # http://bugzilla.mozilla.org/show_bug.cgi?id=450485
 buildsymbols ::
-       $(MAKE) -C mozilla $@ MAKE_SYM_STORE_PATH=..
+       $(MAKE) -C mozilla $@ MAKE_SYM_STORE_PATH=$(MAKE_SYM_STORE_PATH)


I haven't tested this, and I am not sure all these variables are available at that point, OS_ARCH, UNIVERSAL_BINARY, but that's the idea of the fix.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 16

10 years ago
Created attachment 344484 [details] [diff] [review]
process universal binaries when generating buildsymbols on OS X
[Checkin: Comment 19]

Finally gotten around to writing/testing a proper patch.

This patch limits symbols extraction to the final universal binaries.
Attachment #344484 - Flags: review?(kairo)

Comment 17

10 years ago
Comment on attachment 344484 [details] [diff] [review]
process universal binaries when generating buildsymbols on OS X
[Checkin: Comment 19]

r=me by code inspection, I take it that you tested this, and it looks reasonable.
Attachment #344484 - Flags: review?(kairo) → review+
(Assignee)

Comment 18

10 years ago
(In reply to comment #17)
> (From update of attachment 344484 [details] [diff] [review])
> r=me by code inspection, I take it that you tested this,

Yes, both on OSX Universal and Linux without breakage

> and it looks reasonable.

Thanks!
Keywords: checkin-needed
Comment on attachment 344484 [details] [diff] [review]
process universal binaries when generating buildsymbols on OS X
[Checkin: Comment 19]

http://hg.mozilla.org/comm-central/rev/5709136d4b9c
Attachment #344484 - Attachment description: process universal binaries when generating buildsymbols on OS X → process universal binaries when generating buildsymbols on OS X [Checkin: Comment 19]
Assignee: nobody → gozer
Status: REOPENED → RESOLVED
Last Resolved: 11 years ago10 years ago
Component: Build Config → Build Config
Flags: in-testsuite-
Keywords: checkin-needed
OS: Windows Vista → All
Product: Thunderbird → MailNews Core
QA Contact: build-config → build-config
Hardware: PC → All
Resolution: --- → FIXED
Target Milestone: --- → mozilla1.9.1b2
(Reporter)

Updated

10 years ago
Depends on: 464247
(Assignee)

Comment 20

10 years ago
The patch in attachment 344484 [details] [diff] [review] is responsible for bug 464247. The sub-make will not inherit the value of MAKE_SYM_STORE_PATH and will be back to using mozilla's defaults.

The proper fix is to explicitely pass this into mozilla's sub-make invocation.

The only case where this would have worked is if MAKE_SYM_STORE_PATH was set in the environemnt or passed by the make command-line.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Assignee)

Comment 21

10 years ago
Created attachment 348790 [details] [diff] [review]
Pass MAKE_SYM_STORE_PATH down to mozilla's buildsymbols target

Explicitely passing down the value for MAKE_SYM_STORE_PATH we've decided on down to mozilla's buildsymbols target is required.
(Assignee)

Updated

10 years ago
Attachment #348790 - Flags: review?(kairo)

Comment 22

10 years ago
Comment on attachment 348790 [details] [diff] [review]
Pass MAKE_SYM_STORE_PATH down to mozilla's buildsymbols target

sounds logical to me, I wondered myself if that was the problem when looking at the last patch. r=me by code inspection.
Attachment #348790 - Flags: review?(kairo) → review+
(Assignee)

Comment 23

10 years ago
What must have hapenned is that I've either tested this with MAKE_SYM_STORE_PATH in my environment, or invoked make directly with it, possibly. Otherwise, not entirely sure why it would have worked quite right, even on linux.
(Assignee)

Comment 24

10 years ago
http://hg.mozilla.org/comm-central/rev/be77b2aa036b

changeset:   1143:be77b2aa036b
tag:         tip
user:        Philippe M. Chiasson <gozer@mozillamessaging.com>
date:        Tue Nov 18 14:01:02 2008 -0500
summary:     Bug 450485. Make sure the value of MAKE_SYM_STORE_PATH is passed to mozilla's buildsymbols target. r=KaiRo
Status: REOPENED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED
(Reporter)

Comment 25

10 years ago
yay!  we have w32 symbols.  I haven't checked Mac.
thanks gozer

crashme extension results

bp-13f2c778-b6cf-475b-bada-ef2620081119 build 20081118031447
crashme.dll	nsCrasher::Crash	nsCrasher.cpp:21
xpcom_core.dll	NS_InvokeByIndex_P	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:101
thunderbird.exe	thunderbird.exe@0x9aab0	

bp-2cd19882-840c-4fba-a8ac-739720081119 buildid 20081119032441
crashme.dll	nsCrasher::Crash	nsCrasher.cpp:21
xpcom_core.dll	NS_InvokeByIndex_P	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:101
thunderbird.exe	XPCWrappedNative::CallMethod	js/src/xpconnect/src/xpcwrappednative.cpp:2422
thunderbird.exe	XPC_WN_CallMethod	js/src/xpconnect/src/xpcwrappednativejsops.cpp:1477
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.