Closed Bug 721160 Opened 12 years ago Closed 12 years ago

[10.5] Breakpad broken on OS X 10.5.8 in current trunk and aurora nightlies

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
blocker

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla13
Tracking Flags:
Tracking Status
firefox12 --- verified
firefox13 --- verified

People

(Reporter: marcia, Assigned: smichaud)

References

Details

(Keywords: crash, reproducible, Whiteboard: [qa!] STR in comment #10)

Attachments

(2 files)

Crash reporter crashing
2.80 KB, text/html
Details
Fix
1.86 KB, patch
ted
: review+
akeybl
: approval-mozilla-aurora+
Details | Diff | Splinter Review 
Seen while running the latest trunk on 10.5.8.

STR:
1. Load ftp://ftp.mozilla.org/pub/firefox/nightly/10.0b6-candidates/build1/mac/en-US/
2. Click on the Mac .dmg file in that directory
3. I get the attached screenshot
Unable to reproduce the crash using Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:10.0) Gecko/20100101 Firefox/10.0 on the same machine (latest beta).
From the Mac console app:

1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly ACR (Component): component init 
1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly *** nsHTMLElement::FinishDecoderSetup() mDecoder=0x23401560 stream=0x24d80000 src=http://videos-cdn.mozilla.net/serv/qa/qa-360p-mp4.theora.ogv 
1/25/12 11:56:53 AM [0x0-0x51051].org.mozilla.nightly *** nsHTMLElement::FinishDecoderSetup() mDecoder=0x23401560 stream=0x24d80000 src=http://videos-cdn.mozilla.net/serv/qa/qa-360p-mp4.theora.ogv 
1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly dyld: Library not loaded: /usr/lib/libcrypto.0.9.8.dylib 
1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly   Referenced from: /Users/mozilla/Desktop/trunk/Nightly.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter 
1/25/12 11:57:09 AM [0x0-0x51051].org.mozilla.nightly   Reason: image not found
Comment on attachment 591574 [details]
Crash reporter crashing

The key bit here is:
Library not loaded: /usr/lib/libcrypto.0.9.8.dylib
Reason: image not found
So we're linking the crashreporter client to libcrypto:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/Makefile.in#89

but I don't think it actually uses it directly. It just uses Cocoa classes to do the HTTP POST, I assume they'll load libcrypto to do SSL if need be.

The Mac-specific client bits:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/crashreporter_osx.mm
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/common/mac/HTTPMultipartUpload.m
I'd guess the broken builds are built with the 10.6 SDK, not the 10.5 SDK.

Let me see if I can confirm my hunch (by among other things finding the regression range).

Here's Ted's Crashme extension to test with:
http://ted.mielczarek.org/mozilla/crashme.html
I think that was bug 674655, but I don't see the changesets listed there.
Also your comment in bug 674655 comment 22 seems quite prescient now :)
Here's the regression range:

firefox-2012-01-19-03-11-05-mozilla-central
firefox-2012-01-20-03-11-25-mozilla-central

> I think that was bug 674655

Actually it was bug 715397:

Bug 715397 - ability to build m-c on 10.7 (lion). r=ted.
author	Rafael Ávila de Espíndola <respindola@mozilla.com>
	Wed Jan 18 15:35:28 2012 -0500 (at Wed Jan 18 15:35:28 2012 -0500)

http://hg.mozilla.org/mozilla-central/rev/429503206f26
Blocks: 715397
> Also your comment in bug 674655 comment 22 seems quite prescient now :)

Yes, indeed.

I suppose it's a good sign that this bug is the only major breakage discovered since the patch for bug 715397 was landed 3+ weeks ago.  Or is it? :-)

I think it's a major disaster for Breakpad to be broken on OS X 10.5, so I'm labeling this a "blocker".
Severity: critical → blocker
Summary: Crash Reporter Crashes when loading link → [10.5] Breakpad broken on OS X 10.5.8 in current trunk nightlies
STR (for OS X 10.5.8):

1) Run a recent	mozilla-central nightly (2012-01-20 or later):

2) If you haven't already, install Ted Mielczarek's Crashme extension:
   http://ted.mielczarek.org/mozilla/crashme.html

3) Choose one of the crash options from Tools : Crash me! -- for
   example Null pointer deref!

   An Apple crash report will appear, with (more or less) the
   following contents:

Process:         crashreporter [1877]
Path:            /Users/smichaud/Desktop/FirefoxNightly 2012-02-13.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter
Identifier:      crashreporter
Version:         ??? (???)
Code Type:       X86 (Native)
Parent Process:  firefox [1863]

Interval Since Last Report:          84 sec
Crashes Since Last Report:           1
Per-App Interval Since Last Report:  0 sec
Per-App Crashes Since Last Report:   1

Date/Time:       2012-02-13 10:26:40.583 -0600
OS Version:      Mac OS X 10.5.8 (9L30)
Report Version:  6
Anonymous UUID:  C9512736-EC85-4CD1-B209-AB5833DEE8E2

Exception Type:  EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000002, 0x0000000000000000
Crashed Thread:  0

Dyld Error Message:
  Library not loaded: /usr/lib/libcrypto.0.9.8.dylib
  Referenced from: /Users/smichaud/Desktop/FirefoxNightly 2012-02-13.app/Contents/MacOS/crashreporter.app/Contents/MacOS/crashreporter
  Reason: image not found

The OS X 10.5.8 version of libcrypto.dylib is 0.9.7.
Whiteboard: STR in comment #10
> I suppose it's a good sign that this bug is the only major breakage
> discovered since the patch for bug 715397 was landed 3+ weeks ago.
> Or is it? :-)

What I meant is does anyone know of other kinds of breakage?
> So we're linking the crashreporter client to libcrypto:
> http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/client/Makefile.in#89
>
> but I don't think it actually uses it directly.

So the obvious thing to try is to get rid of that line.  I'll do that and see what happens.
Does the crashreporter client build if we drop the -lcrypto from the Makefile? If so that might be the simplest fix.
> Does the crashreporter client build if we drop the -lcrypto from the Makefile?

Yes.

I haven't yet gotten Breakpad to work in my builds, but that's probably because I'm not building correctly.  I'll keep trying.  I'll also do some tryserver builds.

I've found another explicit link to libcryto.dylib for building dump_syms on the Mac.  That isn't needed either (dump_syms builds fine without it).

Shortly I'll post a patch and ask you to review it.
Attached patch Fix — — Splinter Review 
Stop explicitly linking libcrypto on the Mac, to avoid breakage on OS X 10.5 when building with the 10.6 SDK.
Assignee: nobody → smichaud
Attachment #596724 - Flags: review?(ted.mielczarek)
This bug also exists on the aurora branch -- it's reproducible with today's aurora nightly.
status-firefox12: --- → affected
status-firefox13: --- → affected
tracking-firefox12: --- → ?
Target Milestone: --- → mozilla12
Summary: [10.5] Breakpad broken on OS X 10.5.8 in current trunk nightlies → [10.5] Breakpad broken on OS X 10.5.8 in current trunk and aurora nightlies
Comment on attachment 596724 [details] [diff] [review]
Fix

Review of attachment 596724 [details] [diff] [review]:
-----------------------------------------------------------------

I think the dump_syms one is leftover from before we fixed bug 599475, which replaced use of libcrypto with Breakpad's internal MD5 implementation.
Attachment #596724 - Flags: review?(ted.mielczarek) → review+
> I haven't yet gotten Breakpad to work in my builds, but that's
> probably because I'm not building correctly.  I'll keep trying.

Breakpad now works fine	(with my patch) on OS X 10.5.8, in a build
made with the following mozconfig (which is the same as, or at least
very close to, the mozconfig used for Intel-universal nightly builds):

export CFLAGS="-gdwarf-2"
export CXXFLAGS="-gdwarf-2"
. $topsrcdir/browser/config/mozconfig
. $topsrcdir/build/macosx/universal/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-firefox-nightly
mk_add_options MOZ_MAKE_FLAGS=-j4
mk_add_options AUTOCONF=autoconf213
ac_add_options --enable-application=browser
ac_add_options --enable-tests
ac_add_options --enable-codesighs
ac_add_options --disable-install-strip
ac_add_options --enable-js-diagnostics
ac_add_options --with-macbundlename-prefix=Firefox
# For NSS symbols
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols="-gdwarf-2"
# Needed to enable breakpad in application.ini
export MOZILLA_OFFICIAL=1
https://hg.mozilla.org/mozilla-central/rev/8630cfc50df9
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: mozilla12 → mozilla13
We should absolutely take this on aurora once we've verified that it fixes the problem in nightly.
This patch didn't make it into today's nightly.  I'll test with tomorrow's, then (presuming it works as expected) I'll request approval for aurora.
Comment on attachment 596724 [details] [diff] [review]
Fix

Today's mozilla-central nightly works fine on OS X 10.5.8.  You get the expected results from the STR in comment #10 -- Breakpad functions normally.

On that strength I'm requesting approval to land on the aurora branch.

This is a trivial patch, with almost zero risk, that fixes a very serious problem.
Attachment #596724 - Flags: approval-mozilla-aurora?
Comment on attachment 596724 [details] [diff] [review]
Fix

[Triage Comment]
Low risk, prevents a blind spot in crash data. Approving for Aurora 12.
Attachment #596724 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Whiteboard: STR in comment #10 → [qa+] STR in comment #10
I have followed the steps from comment10 and no Apple crash report appears, only the Firefox crash reporter. 
It's enough to mark this as Verified on:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:12.0) Gecko/20100101 Firefox/12.0 beta 3 ?
Thanks
Yes, if the Firefox crash reporter appears then the bug is fixed.
Considering comment26 and also comment27, setting the resolution to Verified Fixed on Firefox 12 beta.
status-firefox12: fixed → verified
I have verified this using the steps from comment10 and no Mac crash reporter appeared.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; rv:13.0) Gecko/20100101 Firefox/13.0 beta 2

Setting the resolution to Verified Fixed
Status: RESOLVED → VERIFIED
status-firefox13: fixed → verified
Whiteboard: [qa+] STR in comment #10 → [qa!] STR in comment #10
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: