Open Bug 1732571 Opened 3 years ago Updated 3 years ago

Lines missing in some Socorro stack traces

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Platform:
Unspecified
macOS
Type:
defect

Tracking

()

People

(Reporter: smichaud, Unassigned)

References

(Blocks 1 open bug)

Details

"Mac crash info" (supported as of bug 1577886) sometimes contains a "backtrace" field. When you compare its contents with the Socorro crash stack it's associated with, you often find that many lines are missing from the Socorro crash stack.

https://crash-stats.mozilla.org/search/?mac_crash_info=~backtrace&platform=Mac%20OS%20X&date=%3E%3D2021-03-26T17%3A39%3A00.000Z&date=%3C2021-09-26T17%3A39%3A00.000Z&_facets=signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

In the following comments I'll post some examples -- one per comment.

Blocks: 1711944

bp-57348e80-71b2-442c-9aef-e063e0210925 has the following Socorro crash stack:

    0  AppKit  -[NSApplication _crashOnException:]   context
    1  AppKit  __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke   cfi
    2  AppKit  ___NSRunLoopObserverCreateWithHandler_block_invoke   cfi
    3  CoreFoundation  __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__   cfi
    4  CoreFoundation  __CFRunLoopDoObservers   cfi
    5  CoreFoundation  __CFRunLoopRun   cfi
    6  CoreFoundation  CFRunLoopRunSpecific   cfi
    7  HIToolbox  RunCurrentEventLoopInMode   cfi
    8  HIToolbox  ReceiveNextEventCommon   cfi
    9  HIToolbox  _BlockUntilNextEventMatchingListInModeWithFilter   cfi
    10  AppKit  _DPSNextEvent   cfi
    11  AppKit  -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]   cfi
    12  XUL  -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  widget/cocoa/nsAppShell.mm:173  cfi
    13  AppKit  -[NSApplication run]   cfi
    14  XUL  nsAppShell::Run()  widget/cocoa/nsAppShell.mm:775  cfi
    15  XUL  nsAppStartup::Run()  toolkit/components/startup/nsAppStartup.cpp:289  cfi
    16  XUL  XREMain::XRE_mainRun()  toolkit/xre/nsAppRunner.cpp:5291  cfi
    17  XUL  XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5476  cfi
    18  XUL  XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5535  cfi
    19  firefox  main  browser/app/nsBrowserApp.cpp:386  cfi
    20  libdyld.dylib  start   cfi

But you can tell it's missing many lines at the top by comparing it to the "backtrace" from the same crash report:

    "backtrace": "0   CoreFoundation                      0x00007fff204e283b __exceptionPreprocess + 242
    1   libobjc.A.dylib                     0x00007fff2021ad92 objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff2056534d -[NSObject(NSObject) __retain_OA] + 0
    3   CoreFoundation                      0x00007fff2044a8cb ___forwarding___ + 1448
    4   CoreFoundation                      0x00007fff2044a298 _CF_forwarding_prep_0 + 120
    5   XUL                                 0x0000000116d33403 _ZN7mozilla6layers13NativeLayerCA14Representation14EnqueueSurfaceEP11__IOSurface + 259
    6   XUL                                 0x0000000114c04a85 _ZN7mozilla6layers17NativeLayerRootCA14Representation6CommitENS1_19WhichRepresentationERK8nsTArrayI6RefPtrINS0_13NativeLayerCAEEEb + 3973
    7   XUL                                 0x0000000114c039ca _ZN7mozilla6layers17NativeLayerRootCA14CommitToScreenEv + 74
    8   XUL                                 0x000000011830616e _ZN11nsChildView29HandleMainThreadCATransactionEv + 238
    9   XUL                                 0x000000011522bf42 -[ChildView updateRootCALayer] + 66
    10  AppKit                              0x00007fff22d4803c _NSViewUpdateLayer + 98
    11  AppKit                              0x00007fff22c5a275 +[NSAppearance _performWithCurrentAppearance:usingBlock:] + 66
    12  AppKit                              0x00007fff22eba423 __30-[_NSViewBackingLayer display]_block_invoke + 113
    13  AppKit                              0x00007fff22c9948d -[NSFocusStack performWithFocusView:inWindow:usingBlock:] + 92
    14  AppKit                              0x00007fff22d479c5 -[_NSViewBackingLayer display] + 382
    15  QuartzCore                          0x00007fff26bf5853 _ZN2CA5Layer17display_if_neededEPNS_11TransactionE + 863
    16  QuartzCore                          0x00007fff26d40d3e _ZN2CA7Context18commit_transactionEPNS_11TransactionEdPd + 666
    17  QuartzCore                          0x00007fff26bd6cdf _ZN2CA11Transaction6commitEv + 713
    18  AppKit                              0x00007fff22df1bfc __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke + 285
    19  AppKit                              0x00007fff235474d3 ___NSRunLoopObserverCreateWithHandler_block_invoke + 41
    20  CoreFoundation                      0x00007fff20467d01 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    21  CoreFoundation                      0x00007fff20467b95 __CFRunLoopDoObservers + 543
    22  CoreFoundation                      0x00007fff20467028 __CFRunLoopRun + 841
    23  CoreFoundation                      0x00007fff2046661c CFRunLoopRunSpecific + 563
    24  HIToolbox                           0x00007fff286aba83 RunCurrentEventLoopInMode + 292
    25  HIToolbox                           0x00007fff286ab7e5 ReceiveNextEventCommon + 587
    26  HIToolbox                           0x00007fff286ab583 _BlockUntilNextEventMatchingListInModeWithFilter + 70
    27  AppKit                              0x00007fff22c6d502 _DPSNextEvent + 864
    28  AppKit                              0x00007fff22c6bcd5 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1364
    29  XUL                                 0x0000000115232701 -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 177
    30  AppKit                              0x00007fff22c5e049 -[NSApplication run] + 586
    31  XUL                                 0x0000000115233a2b _ZN10nsAppShell3RunEv + 171
    32  XUL                                 0x000000011909539c _ZN12nsAppStartup3RunEv + 60
    33  XUL                                 0x0000000115684fd1 _ZN7XREMain11XRE_mainRunEv + 2369
    34  XUL                                 0x000000011911754f _ZN7XREMain8XRE_mainEiPPcRKN7mozilla15BootstrapConfigE + 991
    35  XUL                                 0x0000000119117a35 _Z8XRE_mainiPPcRKN7mozilla15BootstrapConfigE + 133
    36  firefox                             0x000000010aedf3c2 main + 658
    37  libdyld.dylib                       0x00007fff2038af3d start + 1
    38  ???                                 0x0000000000000001 0x0 + 1",

Another example is bp-b8c9c32f-08bd-40b1-9aa4-d5a0f0210925.

Here's the Socorro stack trace:

    0  AppKit  -[NSApplication _crashOnException:]   context
    1  AppKit  __65+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayRefresh]_block_invoke   frame_pointer
    2  CoreFoundation  __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__   frame_pointer
    3  CoreFoundation  __CFRunLoopDoObservers   frame_pointer
    4  CoreFoundation  __CFRunLoopRun   frame_pointer
    5  CoreFoundation  CFRunLoopRunSpecific   frame_pointer
    6  HIToolbox  RunCurrentEventLoopInMode   frame_pointer
    7  HIToolbox  ReceiveNextEventCommon   frame_pointer
    8  HIToolbox  _BlockUntilNextEventMatchingListInModeWithFilter   frame_pointer
    9  AppKit  _DPSNextEvent   frame_pointer
    10  AppKit  -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]   frame_pointer
    11  XUL  -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  widget/cocoa/nsAppShell.mm:173  frame_pointer
    12  AppKit  -[NSApplication run]   cfi
    13  XUL  nsAppShell::Run()  widget/cocoa/nsAppShell.mm:775  frame_pointer
    14  XUL  nsAppStartup::Run()  toolkit/components/startup/nsAppStartup.cpp:289  cfi
    15  XUL  XREMain::XRE_mainRun()  toolkit/xre/nsAppRunner.cpp:5291  cfi
    16  XUL  XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5476  cfi
    17  XUL  XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5535  cfi
    18  firefox  main  browser/app/nsBrowserApp.cpp:386  cfi
    19  libdyld.dylib  start   cfi

And here's the "backtrace":

    "backtrace": "0   CoreFoundation                      0x00007fff2aa80f3b __exceptionPreprocess + 171
    1   libobjc.A.dylib                     0x00007fff51ddd942 objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff2ab18414 -[NSObject(NSObject) doesNotRecognizeSelector:] + 132
    3   CoreFoundation                      0x00007fff2a9f7f03 ___forwarding___ + 1443
    4   CoreFoundation                      0x00007fff2a9f78d8 _CF_forwarding_prep_0 + 120
    5   XUL                                 0x0000000113b375f3 _ZN7mozilla6layers13NativeLayerCA14Representation14EnqueueSurfaceEP11__IOSurface + 259
    6   XUL                                 0x0000000111a01b45 _ZN7mozilla6layers17NativeLayerRootCA14Representation6CommitENS1_19WhichRepresentationERK8nsTArrayI6RefPtrINS0_13NativeLayerCAEEEb + 3973
    7   XUL                                 0x0000000111a00a8a _ZN7mozilla6layers17NativeLayerRootCA14CommitToScreenEv + 74
    8   XUL                                 0x0000000115116a6e _ZN11nsChildView29HandleMainThreadCATransactionEv + 238
    9   XUL                                 0x000000011202fb02 -[ChildView updateRootCALayer] + 66
    10  AppKit                              0x00007fff2882e964 _NSViewUpdateLayer + 45
    11  AppKit                              0x00007fff280c849e -[_NSViewBackingLayer display] + 391
    12  QuartzCore                          0x00007fff35da9e29 _ZN2CA5Layer17display_if_neededEPNS_11TransactionE + 633
    13  QuartzCore                          0x00007fff35da98e7 _ZN2CA5Layer28layout_and_display_if_neededEPNS_11TransactionE + 35
    14  QuartzCore                          0x00007fff35da88d8 _ZN2CA7Context18commit_transactionEPNS_11TransactionE + 326
    15  QuartzCore                          0x00007fff35da8475 _ZN2CA11Transaction6commitEv + 487
    16  AppKit                              0x00007fff2887fa72 __65+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayRefresh]_block_invoke + 283
    17  CoreFoundation                      0x00007fff2aa19507 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    18  CoreFoundation                      0x00007fff2aa1942f __CFRunLoopDoObservers + 511
    19  CoreFoundation                      0x00007fff2a9fbef8 __CFRunLoopRun + 1240
    20  CoreFoundation                      0x00007fff2a9fb787 CFRunLoopRunSpecific + 487
    21  HIToolbox                           0x00007fff29cdbd96 RunCurrentEventLoopInMode + 286
    22  HIToolbox                           0x00007fff29cdbb06 ReceiveNextEventCommon + 613
    23  HIToolbox                           0x00007fff29cdb884 _BlockUntilNextEventMatchingListInModeWithFilter + 64
    24  AppKit                              0x00007fff27f88a3b _DPSNextEvent + 2085
    25  AppKit                              0x00007fff2871ee34 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 3044
    26  XUL                                 0x0000000112036211 -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 177
    27  AppKit                              0x00007fff27f7d84d -[NSApplication run] + 764
    28  XUL                                 0x000000011203753b _ZN10nsAppShell3RunEv + 171
    29  XUL                                 0x0000000115ea573c _ZN12nsAppStartup3RunEv + 60
    30  XUL                                 0x0000000112489fa1 _ZN7XREMain11XRE_mainRunEv + 2369
    31  XUL                                 0x0000000115f27c3f _ZN7XREMain8XRE_mainEiPPcRKN7mozilla15BootstrapConfigE + 991
    32  XUL                                 0x0000000115f28125 _Z8XRE_mainiPPcRKN7mozilla15BootstrapConfigE + 133
    33  firefox                             0x0000000109f7a3c2 main + 658
    34  libdyld.dylib                       0x00007fff529f8015 start + 1
    35  ???                                 0x0000000000000001 0x0 + 1",

Yet another example is bp-2f74cc94-f887-41c7-8ff5-3ce6b0210923, this time on ARM64 hardware:

Here's the Socorro stack trace:

    0  AppKit  -[NSApplication _crashOnException:]   context
    1  AppKit  -[NSApplication _crashOnException:]   frame_pointer
    2  AppKit  __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke   frame_pointer
    3  AppKit  ___NSRunLoopObserverCreateWithHandler_block_invoke   frame_pointer
    4  CoreFoundation  __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__   frame_pointer
    5  CoreFoundation  __CFRunLoopDoObservers   frame_pointer
    6  CoreFoundation  __CFRunLoopRun   frame_pointer
    7  CoreFoundation  CFRunLoopRunSpecific   frame_pointer
    8  HIToolbox  RunCurrentEventLoopInMode   frame_pointer
    9  HIToolbox  ReceiveNextEventCommon   frame_pointer
    10  HIToolbox  _BlockUntilNextEventMatchingListInModeWithFilter   frame_pointer
    11  AppKit  _DPSNextEvent   frame_pointer
    12  AppKit  -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]   frame_pointer
    13  XUL  -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]  widget/cocoa/nsAppShell.mm:173  frame_pointer
    14  AppKit  -[NSApplication run]   frame_pointer
    15  XUL  nsAppShell::Run()  widget/cocoa/nsAppShell.mm:775  frame_pointer
    16  XUL  nsAppStartup::Run()  toolkit/components/startup/nsAppStartup.cpp:289  frame_pointer
    17  XUL  XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5476  frame_pointer
    18  XUL  XRE_main(int, char**, mozilla::BootstrapConfig const&)  toolkit/xre/nsAppRunner.cpp:5535  frame_pointer
    19  firefox  main  browser/app/nsBrowserApp.cpp:386  frame_pointer
    20  libdyld.dylib  start   frame_pointer

And here's the "backtrace":

    backtrace": "0   CoreFoundation                      0x000000018ddcb838 __exceptionPreprocess + 240
    1   libobjc.A.dylib                     0x000000018daf50a8 objc_exception_throw + 60
    2   CoreFoundation                      0x000000018de5c694 -[NSObject(NSObject) __retain_OA] + 0
    3   CoreFoundation                      0x000000018dd2ccd4 ___forwarding___ + 1444
    4   CoreFoundation                      0x000000018dd2c670 _CF_forwarding_prep_0 + 96
    5   XUL                                 0x0000000107ed19f8 _ZN7mozilla6layers13NativeLayerCA14Representation14EnqueueSurfaceEP11__IOSurface + 256
    6   XUL                                 0x00000001055766bc _ZN7mozilla6layers17NativeLayerRootCA14Representation6CommitENS1_19WhichRepresentationERK8nsTArrayI6RefPtrINS0_13NativeLayerCAEEEb + 2708
    7   XUL                                 0x0000000105575af0 _ZN7mozilla6layers17NativeLayerRootCA14CommitToScreenEv + 92
    8   XUL                                 0x0000000109395e68 _ZN11nsChildView29HandleMainThreadCATransactionEv + 248
    9   XUL                                 0x0000000105c8e080 -[ChildView updateRootCALayer] + 92
    10  AppKit                              0x0000000190617784 _NSViewUpdateLayer + 108
    11  AppKit                              0x0000000190529dac +[NSAppearance _performWithCurrentAppearance:usingBlock:] + 92
    12  AppKit                              0x000000019077fb88 __30-[_NSViewBackingLayer display]_block_invoke + 148
    13  AppKit                              0x0000000190569d8c -[NSFocusStack performWithFocusView:inWindow:usingBlock:] + 112
    14  AppKit                              0x00000001906170d8 -[_NSViewBackingLayer display] + 436
    15  QuartzCore                          0x000000019400b384 _ZN2CA5Layer17display_if_neededEPNS_11TransactionE + 780
    16  QuartzCore                          0x00000001941373bc _ZN2CA7Context18commit_transactionEPNS_11TransactionEdPd + 512
    17  QuartzCore                          0x0000000193fed310 _ZN2CA11Transaction6commitEv + 684
    18  AppKit                              0x00000001906ba750 __62+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayLink]_block_invoke + 304
    19  AppKit                              0x0000000190e0fcd4 ___NSRunLoopObserverCreateWithHandler_block_invoke + 64
    20  CoreFoundation                      0x000000018dd4c888 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 36
    21  CoreFoundation                      0x000000018dd4c6d4 __CFRunLoopDoObservers + 572
    22  CoreFoundation                      0x000000018dd4bc1c __CFRunLoopRun + 764
    23  CoreFoundation                      0x000000018dd4b1a8 CFRunLoopRunSpecific + 600
    24  HIToolbox                           0x0000000195c70b40 RunCurrentEventLoopInMode + 292
    25  HIToolbox                           0x0000000195c708b4 ReceiveNextEventCommon + 552
    26  HIToolbox                           0x0000000195c70674 _BlockUntilNextEventMatchingListInModeWithFilter + 72
    27  AppKit                              0x000000019053d660 _DPSNextEvent + 836
    28  AppKit                              0x000000019053c000 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1292
    29  XUL                                 0x0000000105ca26fc -[GeckoNSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 204
    30  AppKit                              0x000000019052de4c -[NSApplication run] + 596
    31  XUL                                 0x0000000105ca3968 _ZN10nsAppShell3RunEv + 192
    32  XUL                                 0x0000000109ce4ce0 _ZN12nsAppStartup3RunEv + 84
    33  XUL                                 0x0000000109d5abf4 _ZN7XREMain8XRE_mainEiPPcRKN7mozilla15BootstrapConfigE + 4104
    34  XUL                                 0x0000000109d5b1d4 _Z8XRE_mainiPPcRKN7mozilla15BootstrapConfigE + 128
    35  firefox                             0x00000001042c3368 main + 612
    36  libdyld.dylib                       0x000000018dc6d430 start + 4",

Gabriele, do you have any ideas here? Note that the "app notes" stacks have more lines than the Socorro crash stacks, so the lines missing from the Socorro crash stacks may not be missing from the "app notes" stacks -- though it's hard to tell, since the "app notes" stacks aren't symbolicated.

Later I'll dig into this by using a HookCase hook library to reproduce these crashes.

Flags: needinfo?(gsvelto)
See Also: → 1602352

It looks to me like the backtrace in the "app notes" is taken at a different point compared to the one in the Socorro stack. That would be logical because the macOS code will likely:

  1. Detect the issue
  2. Generate the backtrace
  3. Throw an exception <- we capture the minidump only here

Additionally we don't support inlined frames yet. If they do that could account for things like the extra frame in XUL.

Flags: needinfo?(gsvelto)

This makes sense. But all the backtraces that I gave examples of have objc_exception_throw and __exceptionPreprocess at the top. So you'd think that the exception would be thrown immediately, and that Breakpad would capture the entire stack in its minidump.

I'll dig into this later. It may be a while.

The severity field is not set for this bug.
:gsvelto, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gsvelto)
Severity: -- → S4
Flags: needinfo?(gsvelto)
You need to log in before you can comment on or make changes to this bug.