Closed Bug 778364 Opened 13 years ago Closed 13 years ago

MacOS and Linux crash signatures no longer have C++ parameter information

Categories

(Socorro :: General, task)

Product:
Socorro
Component:
General
Platform:
All
macOS
Type:
task
Priority:
Not set
Severity:
critical

Tracking

(firefox17 affected, firefox18 affected, firefox19 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox17 --- affected
firefox18 --- affected
firefox19 --- fixed

People

(Reporter: marcia, Assigned: ted)

Details

(Keywords: hang)

Crash Data

Attachments

(1 file)

wip patch
1.31 KB, patch
Details | Diff | Splinter Review
This bug was filed from the Socorro interface and is report bp-1a4ac3de-8004-4179-9412-94b882120726 . ============================================================= Seen while looking at the Mac specific crash report. These hangs are on 10.8 OS only: https://crash-stats.mozilla.com/report/list?signature=hang%20|%20libsystem_kernel.dylib@0x12122 Frame Module Signature Source 0 libsystem_kernel.dylib libsystem_kernel.dylib@0x12122 1 libsystem_c.dylib libsystem_c.dylib@0x19ddc 2 XUL google_breakpad::ExceptionHandler::WriteMinidump toolkit/crashreporter/google-breakpad/src/client/mac/handler/exception_handler.cc:298 3 XUL CrashReporter::CreatePairedMinidumps toolkit/crashreporter/nsExceptionHandler.cpp:2378 4 XUL mozilla::plugins::PluginModuleParent::ShouldContinueFromReplyTimeout CrashReporterParent.h:120 5 XUL mozilla::plugins::PPluginModuleParent::OnReplyTimeout obj-firefox/x86_64/ipc/ipdl/PPluginModuleParent.cpp:1388 6 XUL mozilla::ipc::SyncChannel::ShouldContinueFromTimeout ipc/glue/SyncChannel.cpp:232 7 XUL mozilla::ipc::RPCChannel::Call ipc/glue/RPCChannel.cpp:179 8 XUL mozilla::plugins::PPluginInstanceParent::CallNPP_Destroy obj-firefox/x86_64/ipc/ipdl/PPluginInstanceParent.cpp:845 9 XUL mozilla::plugins::PluginInstanceParent::Destroy dom/plugins/ipc/PluginInstanceParent.cpp:145 10 XUL mozilla::plugins::PluginModuleParent::NPP_Destroy dom/plugins/ipc/PluginModuleParent.cpp:409 11 XUL nsNPAPIPluginInstance::Stop dom/plugins/base/nsNPAPIPluginInstance.cpp:213 12 XUL nsPluginHost::StopPluginInstance dom/plugins/base/nsPluginHost.cpp:3142 13 XUL nsObjectLoadingContent::DoStopPlugin content/base/src/nsObjectLoadingContent.cpp:2124 14 XUL nsObjectLoadingContent::StopPluginInstance content/base/src/nsObjectLoadingContent.cpp:2155 15 XUL nsObjectLoadingContent::NotifyOwnerDocumentActivityChanged content/base/src/nsObjectLoadingContent.cpp:738 16 XUL NotifyActivityChanged content/base/src/nsDocument.cpp:3715 17 XUL EnumerateFreezables content/base/src/nsDocument.cpp:7934 18 XUL PL_DHashTableEnumerate obj-firefox/x86_64/xpcom/build/pldhash.cpp:715 19 XUL nsDocument::RemovedFromDocShell nsTHashtable.h:237 20 XUL DocumentViewerImpl::Close layout/base/nsDocumentViewer.cpp:1431 21 XUL nsDocShell::SetupNewViewer docshell/base/nsDocShell.cpp:7785 22 XUL nsDocShell::Embed docshell/base/nsDocShell.cpp:5887 23 XUL nsDocShell::CreateContentViewer docshell/base/nsDocShell.cpp:7595 24 XUL nsDSURIContentListener::DoContent docshell/base/nsDSURIContentListener.cpp:133 25 XUL nsDocumentOpenInfo::TryContentListener uriloader/base/nsURILoader.cpp:677 26 XUL nsDocumentOpenInfo::DispatchContent uriloader/base/nsURILoader.cpp:374 27 XUL nsDocumentOpenInfo::OnStartRequest uriloader/base/nsURILoader.cpp:262 28 XUL nsHttpChannel::CallOnStartRequest netwerk/protocol/http/nsHttpChannel.cpp:767 29 XUL nsHttpChannel::ContinueProcessNormal netwerk/protocol/http/nsHttpChannel.cpp:1273 30 XUL nsHttpChannel::ProcessNormal netwerk/protocol/http/nsHttpChannel.cpp:1208 31 XUL nsHttpChannel::ProcessResponse netwerk/protocol/http/nsHttpChannel.cpp:1105 32 XUL nsHttpChannel::OnStartRequest netwerk/protocol/http/nsHttpChannel.cpp:4308 33 XUL nsInputStreamPump::OnStateStart netwerk/base/src/nsInputStreamPump.cpp:416 34 XUL nsInputStreamPump::OnInputStreamReady netwerk/base/src/nsInputStreamPump.cpp:367 35 XUL nsInputStreamReadyEvent::Run xpcom/io/nsStreamUtils.cpp:81 36 XUL nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:624 37 XUL NS_ProcessPendingEvents_P obj-firefox/x86_64/xpcom/build/nsThreadUtils.cpp:163 38 XUL nsBaseAppShell::NativeEventCallback widget/xpwidgets/nsBaseAppShell.cpp:97 39 XUL nsAppShell::ProcessGeckoEvents widget/cocoa/nsAppShell.mm:410 40 CoreFoundation CoreFoundation@0x12840 41 CoreFoundation CoreFoundation@0x12164 42 CoreFoundation CoreFoundation@0x354e4
Some URLs: Total Count URL 2 http://cnettv.cnet.com/8301-33245_53-57480153-288.html?tag=FD.epic 2 http://games.latimes.com/games/daily-crossword/daily-crossword.aspx 2 http://www.punchbowl.com/events/d88f0ee9ec63d5bd367e/designs?show_designs=true#/ 1 http://www.hbogo.com/#series/browse&assetID=GOROSTGP35829?seriesID=GORO1D600?ass 1 http://watch.slingbox.com/watch/sling_player 1 http://www.kilians.com/attachments/File/AMCalendar_2012-13_Version_1_20_6_12.pdf 1 http://www.ted.com/talks/malte_spitz_your_phone_company_is_watching.html?utm_sou 1 https://flash.poker.svenskaspel.se/flashcgi/flashcgi.exe?ProductId=6001 1 http://www.bing.com/explore/social/?FORM=L8SP55 1 http://live.video.sina.com.cn/room/zhejiang 1 http://socialitelife.com/photos/cute-overload-ryan-gosling-a-baby-photos/on-loca 1 http://www.google.com/finance/portfolio 1 https://apps.facebook.com/playtheville/?fb_source=bookmark_apps&ref=bookmarks&co 1 http://www.is-ag.com/ 1 http://saitek1.camstreams.com/ 1 http://tap2-cdn.rubiconproject.com/partner/scripts/rubicon/emily.html?rtb_ext=1& 1 http://www.cnn.com/video/?_=062049&sessionToken=0TOUwxuz52RPHQ6g70avl57V5#/video 1 https://jody-miller-s-imac--2--lhxazvehdv.app04-12.logmein.com/main.html 1 http://www.hulu.com/ 1 http://www.yahoo.com/ 1 http://nascarfanpage.wix.com/nascar-international-fan#!vstc2=page-3/vstc13=sprin 1 https://test.bankid.com/en/testbankidcom/PersonalTrack/Auth-not-displayed/ 1 http://watch.slingbox.com/sling_player 1 http://prezi.com/3kxqz8_kmlfb/edit/#28
This is the parent-side report which isn't walking the stack back to mozilla::plugins::PPluginInstanceParent::CallNPP_Destroy (in this particular case). I'm not sure why the signature generation isn't correct. I'm going to morph the bug here to fix the signature generation to get this right. Lars can you look at this?
Component: Plug-ins → General
Product: Core → Socorro
Version: 15 Branch → unspecified
what do you expect the signature to be for this crash? I'm assuming that you're wanting the "mozilla::ipc::RPCChannel::Call" in the stack to have triggered the signature sentinel rule. However it is not an exact match for the existing rule. This is stack is a showing it as "C" call with no parameters. Our current signature Sentinel rule wants a "C++" signature with parameters: "mozilla::ipc::RPCChannel::Call" vs. "mozilla::ipc::RPCChannel::Call(IPC::Message*, IPC::Message*)" Would you like to have a new SignatureSentinel rule created to cover this case?
This is the top crash in chofmann's Mac report today: https://crash-analysis.mozilla.com/chofmann/20120809/top-mac-crashes-all.txt
@bsmedberg, was the change from Bug 778162 supposed to fix the signature problem in this bug? If so, it did not result in a change to the signatures here for the reasons that I outlined in Comment #3
I know this bug was morphed, but meanwhile I was able to reproduce this crash signature, which I think based on the steps maps back to the UnityPlayer crashes.
ok, *every* C++ frame in https://crash-stats.mozilla.com/report/index/1a4ac3de-8004-4179-9412-94b882120726 has only a C signature. I looked through all the FF17 crash reports, and it appears that we are getting C-style signatures in all cases. This makes me suspect that the switch to clang has changed the signatures. This is not a desirable change because it means we can't aggregate the mac signatures with windows/linux signatures which do have full C++ information. Also, it doesn't appear that there are any hang reports from MacOS < 10.8. This indicates to me that we haven't been getting any hang reports from macOS < 10.8. But I think Ted already knew that...
Summary: [10.8] hang | libsystem_kernel.dylib@0x12122 → MacOS Crash signatures no longer have C++ parameter information (from the clang switch?)
Blocks: 733905
Parameter types have been missing from Linux and OSX since the new DWARF reader went in. Here's a patch to get them back that was close to being finished: http://breakpad.appspot.com/67001/
clang is probably not at fault.
The parameter-less signatures were not caused by clang, they are a result of using DWARF for unwinding, which has been true on Linux and Mac for more than a year now. Options: * strip windows parameters so everything matches * add DWARF parameters back in, http://breakpad.appspot.com/67001/ has code which would need to be finished and landed
No longer blocks: 733905
Summary: MacOS Crash signatures no longer have C++ parameter information (from the clang switch?) → MacOS and Linux crash signatures no longer have C++ parameter information
Another alternative would be to use the DW_AT_linkage_name attributes and demangle them to get function names. These are present on Linux: <2><573>: Abbrev Number: 27 (DW_TAG_subprogram) <574> DW_AT_name : (indirect string, offset: 0x6d3b9): operator== <57a> DW_AT_linkage_name: (indirect string, offset: 0x6d4be): _ZNK4jsideqES_ <57e> DW_AT_type : <0x5b4> ... If they're present on OSX too (I don't know), then that might be a quick fix. I avoided this initially because it's not a required attribute, it takes up a lot of space, and the GCC/GDB people have been grumbling about getting rid of it for years. But if it's there now, we could use it in the mean time.
I can try to take a look. I will check the OS X binary to see if DW_AT_linkage_name is present. Any steps I can use to reproduce this?
Looks like on OS X we have DW_AT_MIPS_linkage_name.
Attached patch wip patch — — Splinter Review
I will try to finish this patch and send it for review tomorrow.
Assignee: nobody → respindola
Status: NEW → ASSIGNED
Would be good to get some traction on this as there are two signatures in the top 5 in Mac Firefox beta crash stats. Thanks.
(In reply to Rafael Ávila de Espíndola (:espindola) from comment #14) > Created attachment 662679 [details] [diff] [review] > wip patch > > I will try to finish this patch and send it for review tomorrow. Any progress here? Tracking for FF17 to make sure that these crashes are actionable in the near future.
tracking-firefox17: ? → +
> > I will try to finish this patch and send it for review tomorrow. > > Any progress here? Tracking for FF17 to make sure that these crashes are > actionable in the near future. It went for review upstream and I asked legal about the copyright paperwork with google. I was away last week. I am currently catching up on email.
Landed upstream: http://code.google.com/p/google-breakpad/source/detail?r=1059 We can update our in-tree Breakpad to r1059 to pick this up.
Prepared a patch with the Breakpad update, pushed to try: https://tbpl.mozilla.org/?tree=Try&rev=dd036ba41e2b
Assigning to ted since he is doing the breakpad update in m-c.
Assignee: respindola → ted.mielczarek
I screwed up my trychooser syntax, so I cancelled those builds and re-pushed: https://tbpl.mozilla.org/?tree=Try&rev=10c17afb8e2b
Try run showed a few problems: 1) A new source file was added to mac upstream, needs to be added to our makefiles 2) The linux64 opt build spewed a ton of extra warnings during symbol dumping. I can reproduce this locally. This seems to have choked the build somehow.
This just got fixed upstream. Ted, can you give r1062 a try?
Pushed the newer revision to try: https://tbpl.mozilla.org/?tree=Try&rev=7ed44fe38c7e I also updated the makefile to fix the mac build.
Try run looked good, pushed to inbound: http://hg.mozilla.org/integration/mozilla-inbound/rev/8732e019a26d Only slightly related, I also landed an update-breakpad.sh to make my life easier: http://hg.mozilla.org/integration/mozilla-inbound/rev/c9329b363280
Awesome. Thanks! I checked the XUL.sym produced for linux (32 and 64 bits) and OS X and they have argument information in them.
https://hg.mozilla.org/mozilla-central/rev/8732e019a26d
Status: ASSIGNED → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
(In reply to Ed Morley [:edmorley UTC+1] from comment #28) > https://hg.mozilla.org/mozilla-central/rev/8732e019a26d fwiw, this landed for Firefox 19, marking as affected for 18 and 17 as per tracking flags.
status-firefox17: --- → affected
status-firefox18: --- → affected
status-firefox19: --- → fixed
Ted, do you think it would be a good idea to backport this?
We could land it on 18 without much trouble since the larger Breakpad update already landed for that. Landing it for 17 seems like a lot more work (unless your patches apply cleanly there). We determined that this bug has basically always existed though (since jimb rewrote the DWARF parsing), right? In that case I don't think there's a huge need to push it across releases.
(In reply to Ted Mielczarek [:ted] from comment #31) > We could land it on 18 without much trouble since the larger Breakpad update > already landed for that. Landing it for 17 seems like a lot more work > (unless your patches apply cleanly there). We determined that this bug has > basically always existed though (since jimb rewrote the DWARF parsing), > right? In that case I don't think there's a huge need to push it across > releases. I haven't checked the history of the dwarf code in breakpad, but yes, the code that was there had no support for finding the argument types, so this was at least not a recent regression.
Untracking based on comment 31, if this has always been present we don't need to try and push for alternate patches & uplift across branches.
tracking-firefox17: + → ---
(In reply to Paul Silaghi [QA] from comment #34) > There are 2 crashes on FF 19b5: > https://crash-stats.mozilla.com/report/index/37b687af-5edc-411f-933b- > 691702130217 > https://crash-stats.mozilla.com/report/index/cc9d9ee3-0b7d-4d43-b8e0- > 1dc1a2130213 > Any thoughts? Is the raw build saved somewhere? If not I can try to reproduce the build locally, but it would take some time.
(In reply to Paul Silaghi [QA] from comment #34) > There are 2 crashes on FF 19b5: > https://crash-stats.mozilla.com/report/index/37b687af-5edc-411f-933b- > 691702130217 > https://crash-stats.mozilla.com/report/index/cc9d9ee3-0b7d-4d43-b8e0- > 1dc1a2130213 > Any thoughts? I don't understand exactly what you're trying to say here. This bug isn't about a specific crash, it's about the lack of some information in the crash signature.
> I don't understand exactly what you're trying to say here. This bug isn't > about a specific crash, it's about the lack of some information in the crash > signature. Yes. I assume the info is missing from the .sym files, so if we have the objdir somewhere I can just run dump_syms and see why it is failing to demagle (or maybe which files are missing the mangled function name in the dwarf).
Looking in: http://symbols.mozilla.org/firefox/firefox-19.0-Darwin-20130206083616-x86-macosx64-symbols.txt You can find: XUL/37CEA6100E1137FCB534539BCC410E820/XUL.dSYM.tar.bz2 So the XUL .dSYM for this build is available as: http://symbols.mozilla.org/firefox/XUL/37CEA6100E1137FCB534539BCC410E820/XUL.dSYM.tar.bz2
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: