379396 - Enable Breakpad Support In Thunderbird

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[Scott MacGregor]

We'll be moving our crash reporting tools from talkback to breakpad in the 1.9 time frame. 

Here are some thoughts I have about adding this feature to Thunderbird. It looks like almost all of the airbag client code is in mozilla\toolkit so we may not have to do much!

* Bug 368206 --> this patch contains app specific build foo for break pad on the mac that we'll want to port to thunderbird.
* Packaging changes to packages-static to package the break pad client
* Make sure MOZ_AIRBAG is defined for thunderbird.
* Make sure we're building toolkit\airbag for Thunderbird
* I suspect there are some tinderbox configuration changes we'll have to make for the Thunderbird build machines as well so we do the right thing with symbols.

I have a feeling Ted might be able to link us to more bugs for adding breakpad support to Firefox that we'll want to look into porting to Thunderbird. cc'ing Ted :).

Comment 1

[(not currently active) Ted Mielczarek]

Scott: I think (although I haven't tested it) that it should Just Work.  I have tested Breakpad in Firefox and XulRunner, and it's been designed to work in any XR app, since it installs the handler in XRE_Main.  In fact, the code is built by default on Win32, so your trunk TB nightlies should have it!  We might need to tweak the packaging stuff to get the crashreporter.exe/ini in your packages, but other than that it should mostly work.  You can try this with a Thunderbird nightly (or your own build) on Win32 by setting MOZ_AIRBAG=1 and crashing.  :)  If you're using a nightly, make sure that crashreporter.exe/ini exist in the app dir.

Comment 2

[Scott MacGregor]

Attachment: [fix checked in] packaging changes to support breakpad in tbird

Add some packaging foo for the breakpad client in Thunderbird.

Comment 3

[Scott MacGregor]

Comment on attachment 263391 [details] [diff] [review]
[fix checked in] packaging changes to support breakpad in tbird

Forgot to set a reviewer..

Thanks for the good news Ted!

Comment 4

[Benjamin Smedberg]

The changes in browser/app from bug 368206 really probably ought to be in static-config.mk or static-rules.mk so that all apps pick them up automatically. Other than that, it's just the packaging changes.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 263391 [details] [diff] [review]
[fix checked in] packaging changes to support breakpad in tbird

>+; [airbag]

change that to [crash reporter], r=me

Comment 6

[Scott MacGregor]

Comment on attachment 263391 [details] [diff] [review]
[fix checked in] packaging changes to support breakpad in tbird

this patch has been checked in with Ted's suggested wording change.

Comment 7

[Scott MacGregor]

Attachment: more packaging changes to stay in sync with Firefox

This ports Benjamin's patch in Bug #381099. This matches Firefox by only using installing talkback 50% of the time on Windows. And it removes the talkback files on the Mac.

When landing this patch, I believe we also need to turn off talkback on the Mac for Thunderbird. I need to do some digging to see if we need to tweak our windows build machine.

Comment 8

[Scott MacGregor]

Some notes to myself, see bsmedberg's change to stop building talkback on the Mac:

http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=tinder-config.pl&branch=&root=/cvsroot&subdir=/mozilla/tools/tinderbox-configs/firefox/macosx&command=DIFF_FRAMESET&rev1=1.16&rev2=1.17

See Ted's patch to start pushing symbols on the Mac nightly for Firefox: Bug 379117.

See Ted's patch for pushing symbols on the Windows nightly Firefox build: Bug 376063.

Comment 9

[Scott MacGregor]

Hi Ted, do I need to do some extra stuff (like get a new ssh key for the tb build) to get a server user name instead of ffxbld?

Also, what's the extra f in front of fxbld (and for symbols_ffx) stand for? I fxbld --> firefox build I assume. 

+$ENV{SYMBOL_SERVER_USER}   = 'ffxbld';
+$ENV{SYMBOL_SERVER_PATH}   = '/mnt/netapp/breakpad/symbols_ffx/';
+$ENV{SYMBOL_SERVER_SSH_KEY}   = "$ENV{HOME}/.ssh/ffxbld_dsa";

Thanks!

Comment 10

[(not currently active) Ted Mielczarek]

Scott,

"ffxbld" is a new user, essentially equivalent to "cltbld".  Presumably there's an equivalent user for Thunderbird, but none of the build guys are around to ask right now.  CCing coop for info.

"symbols_ffx" is just what Aravind named that dir.  :)  We'll have to get one created for Thunderbird symbols.  We will need to make sure that there's an ssh key available to upload the symbols.

But, all that being said, I think we should hold off on this for a bit.  We're still working through deployment issues with Socorro, and the reporting bits of it are still running on staging VMs.  I think we should wait until things calm down and we're more confident in our infrastructure before we add another product to the mix.  Can we reexamine this prior to alpha 6?

Your patches look good for when we're ready, except you missed a bit that bsmedberg didn't include in his patch (but checked in):
http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=nsBrowserApp.cpp&branch=&root=/cvsroot&subdir=/mozilla/browser/app&command=DIFF_FRAMESET&rev1=1.35&rev2=1.36

Can you file a separate bug about uploading debug symbols from Thunderbird tinderboxen?  We may be able to get that done before we worry about actually enabling crash reporting in Thunderbird.

Comment 11

[Scott MacGregor]

Thanks for the quick reply Ted.

Yeah, I agree it makes sense to hold off until things get more settled with Socorro. I'll try to keep adding changes we'll need to make to this bug so I don't forget them when we decide we're ready to go.

I will files bugs for uploading debug symbols as well. 

Comment 12

[Chris Cooper [:coop] (he/him)]

The equivalent key for Thunderbird is tbirdbld, but it will need to be installed on the machines where it is needed in a manner similar to ffxbld. Let me know when this is needed and I'll get it done.

Comment 13

[Scott MacGregor]

Because Ted, Coop and Aravind rock, we have debug symbols being updated to the symbols server from the Thunderbird tinderbox machines.

On a separate note, I was able to submit a crash report using a optimized build I made by hand, and then search for the crash report on crash-stats.mozilla.com:
http://crash-stats.mozilla.com/report/index/036a276e-2b53-11dc-a9f2-001a4bd46e84

Although I don't see symbols for that stack trace, not sure if this is because it was a build I made.

I think the next steps are:

* Get review and land attachment 266663 [details] [diff] [review] which reduces talkback to 50%
* Add the Enabled=1 flag for official builds to Thunderbird's application.ini file
* Get Thunderbird added as a product to: http://crash-stats.mozilla.com/ to make it easier to look up crash reports

Comment 14

[(not currently active) Ted Mielczarek]

Yeah, we only have symbols for official nightlies, so those are the only ones that will have symbolic stack traces on crash-stats.  We do need to get bug 386618 comment 8 taken care of to get the processor pointing at those symbols though.  bsmedberg should be able to add Thunderbird as a product, I believe he has an admin login to crash-stats.

In addition, in bug 386837, we got the symbol server working for Thunderbird as well, so you should be able to follow the directions at http://developer.mozilla.org/en/docs/Using_the_Mozilla_symbol_server with recent (as of July 3rd or so) Thunderbird builds.  Just use http://symbols.mozilla.org/thunderbird as the symbol server URL.

Comment 15

[Benjamin Smedberg]

Scott, I wouldn't worry about doing 50% unless you're worried about breakpad missing crashes. I'd just switch over completely and remove talkback.

FWIW, we don't add products in general to the query page, we add particular versions e.g. "Thunderbird/3.0a2pre" and that gets automatically made unique. I go through every couple of days and add new versions as they appear in reports.

Comment 16

[Scott MacGregor]

Attachment: remove talkback & turn on breakpad

Thanks for the info Benjamin.

This patch removes talkback and enables breakpad for Thunderbird. 

Yippee!

Comment 17

[Benjamin Smedberg]

Comment on attachment 271538 [details] [diff] [review]
remove talkback & turn on breakpad

>Index: mail/installer/Makefile.in

> MOZ_OPTIONAL_PKG_LIST = \
>-	talkback \
> 	$(NULL)

Remove this variable altogether.

Comment 18

[Scott MacGregor]

Attachment: tinderbox config changes to stop shipping talkback and use objdirs

Now that Thunderbird is using breakpad, we can stop shipping talkback and switch to objdirs. This change should look very similar to what bsmedberg did for the tinderconfig files for Firefox.

For some reason it looks like linux wasn't shipping with talkback on the trunk already...I tweaked it to make it consistent with the rest of the tinderconfig files for thunderbird & firefox.

Comment 19

[J. Paul Reed [:preed]]

Comment on attachment 271962 [details] [diff] [review]
tinderbox config changes to stop shipping talkback and use objdirs

Looks good!

Comment 20

[Phil Ringnalda (:philor)]

Since Talkback is starting to look like it might be at fault in bug 387111, you may well get a twofer out of this.

Comment 21

[Scott MacGregor]

These last two patches have been checked in. Thanks for all the help everyone.