476518 - Sunbird debug symbols for crash reports seem to be missing

Status: RESOLVED FIXED

(Calendar :: Build Config, defect)

Description

[Stefan Sitter [:ssitter]]

Today I experienced some crashes when shutting down Thunderbird and Sunbird. I submitted crash reports for both applications.

The Thunderbird ones contain a useful stacktrace:
http://crash-stats.mozilla.com/report/index/f0e5021c-ce60-4c10-80a4-2719a2090202
http://crash-stats.mozilla.com/report/index/02bc8c17-11f3-4dea-860e-0bda82090202

The Sunbird ones not:
http://crash-stats.mozilla.com/report/index/dbbc3e1a-9ac1-452f-8b31-58e702090202
http://crash-stats.mozilla.com/report/index/9c772afc-85f3-422e-a246-c420c2090202

Looking at http://crash-stats.mozilla.com/ there are more Sunbird crash reports for the last day but none seems to contain useful information either.

Is there a possibility to check if the debug symbols for crash reports are correctly uploaded to the crash server?

Comment 1

[Philippe M. Chiasson (:gozer)]

Hrm, looks like the current buildbot configuration has the symbol generation disabled for nightly Sunbird. I can turn them on and see what happens.

Comment 2

[Philippe M. Chiasson (:gozer)]

Attachment: Enable buildsymbols and upload for Sunbird

This is the required buildbot config change to get debug symbols generated and uploaded for Sunbird.

Comment 3

[Philippe M. Chiasson (:gozer)]

Note that if we are turning this on, it would be a great opportunity to consider switching the Macs over to the new debug symbol generation (stabs) away from the buggy (dwarf-2).

Comment 4

[Tony Mechelynck [:tonymec]]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1b3pre) Gecko/20090201 Calendar/1.0pre

bp-d976c843-cf78-43f9-8971-798a32090202

I see symbols for libxpcom_core.so but not for the top and bottom of the stack where Breakpad only reports "sunbird-bin@" plus some hex addresses

Comment 5

[Philippe M. Chiasson (:gozer)]

(In reply to comment #4)
>
> I see symbols for libxpcom_core.so but not for the top and bottom of the stack

That's probably because luckly, a libxpcom_core.so with the same checksum was built as a result of a Thunderbird build, and crash-stats looks up symbols infomrmation by library checksum.

I've just enabled buildsymbols generation for the Linux and Windows Sunbird nightly builds.

pushing to http://hg.mozilla.org/build/buildbot-configs
searching for changes
changeset:   720:339b2e8b3b74
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Mon Feb 02 22:09:37 2009 -0500
summary:     Bug 476518. Enable nightly buildsymbols for Sunbird Linux+Windows. r=me

For Mac, I'd prefer if we could agree on the dwarf/stubs debugging symbol change before enabling them.

Comment 6

[Stefan Sitter [:ssitter]]

(In reply to comment #5)
> For Mac, I'd prefer if we could agree on the dwarf/stubs debugging symbol
> symbol change before enabling them.

Sunbird already seems to use stabs:
https://hg.mozilla.org/build/buildbot-configs/file/339b2e8b3b74/thunderbird/macosx/mozconfig-sunbird#l20

Comment 7

[Philippe M. Chiasson (:gozer)]

that's because I got them reversed. stabs is the old way, dwarf-2 is the new way, see :

https://hg.mozilla.org/build/buildbot-configs/file/339b2e8b3b74/thunderbird/macosx/mozconfig

Oh, and it's stabs, not stubs, jeez.

Sorry about the confusion.

Comment 8

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

So what is needed to enable dwarf-2 ? I'm all for it.

Comment 9

[Philippe M. Chiasson (:gozer)]

Attachment: [checked in] Enable dwarf-2 buildsymbols for Calendar on OSX

This is all that's needed, the rest of the build tools know how to adapt.

Thunderbird has switched over to it a while back, and it's great. Works 100% of the time and is much, much easier on CPU

Comment 10

[Philippe M. Chiasson (:gozer)]

Attachment: Disable stabs cleanup on osx

Once calendar switched over, we can get rid of a slow, painfull `find . [...]` to delete leftovers files from stabs builds.

Comment 11

[Philippe M. Chiasson (:gozer)]

Note, worked fine for Linux and Win32 yesterday night:

<http://symbols.mozilla.org/sunbird/sunbird-1.0pre-Linux-20090203050314-symbols.txt>
<http://symbols.mozilla.org/sunbird/sunbird-1.0pre-WINNT-20090203035902-symbols.txt>

Comment 12

[Philippe M. Chiasson (:gozer)]

Thanks to Standard8's ever watchful eyes, I did forget to mention that this requires the latest version of Apple's XCode 2.5

Comment 13

[Philipp Kewisch [:Fallen] [:📆][:🧩]]

Comment on attachment 360317 [details] [diff] [review]
[checked in] Enable dwarf-2 buildsymbols for Calendar on OSX


>+<<<<<<< local
>+BRANCHES['comm-central-sunbird']['platforms']['macosx']['upload_symbols'] = True
>+=======
> BRANCHES['comm-central-sunbird']['platforms']['macosx']['upload_symbols'] = False
>+>>>>>>> other
This looks like a misguided merge?


r=philipp with comments considered.

Comment 14

[Philippe M. Chiasson (:gozer)]

(In reply to comment #12)
> Thanks to Standard8's ever watchful eyes, I did forget to mention that this
> requires the latest version of Apple's XCode 2.5

The upgrade to XCode 2.5 was painless, and worked just fine for me. I could easily upgrade cb-xserve03 (downloaded xcode 2.5 to it already)

Comment 15

[Philippe M. Chiasson (:gozer)]

upgraded cb-xserve03 to XCode 2.5, restarted build slave

Comment 16

[Philippe M. Chiasson (:gozer)]

Comment on attachment 360317 [details] [diff] [review]
[checked in] Enable dwarf-2 buildsymbols for Calendar on OSX

comparing with ssh://hg.mozilla.org/build/buildbot-configs
searching for changes
changeset:   722:9e0527a80b00
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Tue Feb 03 14:32:48 2009 -0500
summary:     Bug 360317. Switch calendar osx builds to -gdwarf-2 for buildsymbols. r=philipp

Comment 17

[Philippe M. Chiasson (:gozer)]

Grr, apologies, referenced the wrong bug ## in the hg log message

Comment 18

[Philippe M. Chiasson (:gozer)]

Last night's Mac nightlies compiled with -gdwarf-2 without problems, so all that's left is to enable the buildsymbols step.

Pushed that last remaining change. 

changeset:   723:a79a8147cd59
tag:         tip
user:        Philippe M. Chiasson <[e-mail]>
date:        Wed Feb 04 09:57:47 2009 -0500
summary:     Bug 476518. Enable buildsymbols generation on Sunbird/Mac. r=me