Closed Bug 1679896 Opened 4 years ago Closed 4 years ago

Suspiciously few signatures for macOS content process crashes on Apple Silicon machines

Categories

(Toolkit :: Crash Reporting, defect)

Product:
Toolkit
Component:
Crash Reporting
Platform:
ARM64
macOS
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: smichaud, Unassigned)

Details

There are many fewer different signatures for macOS content process crashes on ARM64 than there are on AMD64. I suspect this means that Breakpad is either not reporting all the crashes it should, or is reporting them incorrectly (with incorrect signatures).

Here are the macOS content process crashes reported over the last 14 days on Apple Silicon:

    1  _pthread_join   Add term  1070  79.26 %  1679513 1678226
    2  libsystem_pthread.dylib@0x8ad8   Add term  276  20.44 %  1679373
    3  IPCError-browser | ShutDownKill | js::NativeGetOwnPropertyDescriptor   Add term  1  0.07 %  
    4  XUL@0x6692280 | XUL@0x248dc48 | XUL@0x248dc48 | XUL@0x248e6cc | XUL@0x258cd1c | XUL@0x73bc20 | XUL@0x685304 | XUL@0x6842f4 | XUL@0x684d3c | XUL@0xee770 | XUL@0xed490 | XUL@0xec730 | XUL@0xf05a4 | XUL@0xfb274 | XUL@0xfecf4 | XUL@0x687368 | XUL@0x648118 ...   Add term  1  0.07 %  
    5  core::ops::function::Fn::call   Add term  1  0.07 %  
    6  mozilla::TrackBuffersManager::UpdateEvictionIndex   Add term  1  0.07 %

https://crash-stats.mozilla.org/search/?cpu_arch=arm64&platform=Mac%20OS%20X&process_type=content&date=%3E%3D2020-11-16T22%3A49%3A00.000Z&date=%3C2020-11-30T22%3A49%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Here are those reported over the last 14 days on Intel machines:

    1  js::GCMarker::processMarkStackTop   Add term  4912  8.96 %  1476239 1415708 1400478 1395740 1386875 1257309 952381 941491 792226 772441 745334 730283 719114
    2  <name omitted> | std::__1::__sift_up<T>   Add term  3686  6.73 %  1654335
    3  nsHtml5TreeOperation::Perform   Add term  3351  6.12 %  780611 616421
    4  js::ObjectGroup::sweep   Add term  1911  3.49 %  1582214 1454398 1112741
    5  std::__1::__sift_up<T>   Add term  1818  3.32 %  1660297
    6  _os_lock_corruption_abort | _OSSpinLockLockSlow   Add term  1265  2.31 %  1153809
    7  js::XDRScript<T>   Add term  1128  2.06 %  1444442 790047
    8  mozilla::H264::DecodeNALUnit   Add term  957  1.75 %  1660368
    9  mp4parse_get_indice_table   Add term  944  1.72 %  1584956
    10  js::gc::TraceEdgeInternal<T>   Add term  876  1.60 %  1474623
    11  std::__1::__pop_heap<T>   Add term  783  1.43 %  
    12  js::jit::MaybeEnterJit   Add term  679  1.24 %  1665579 1665432 1461724 1461480 1451720 1425132 858032
    13  core::option::expect_none_failed | lucet_wasi::ctx::WasiCtxBuilder::fd_dup_for_io_desc   Add term  676  1.23 %  1670878
    14  Interpret   Add term  615  1.12 %  1260683 1185961 962141 937914 935348 917792 884194 682573
    15  gfxSparseBitSet::SetRange   Add term  555  1.01 %  
    16  js::TenuringTracer::traverse<T>   Add term  515  0.94 %  1582214 1358073
    17  nsTArray_Impl<T>::AppendElementsInternal<T> | nsHtml5TreeBuilder::startTag   Add term  457  0.83 %  
    18  js::TraceManuallyBarrieredGenericPointerEdge   Add term  449  0.82 %  1203273
    19  (anonymous namespace)::BytecodeParser::parse   Add term  447  0.82 %  1670481
    20  GraphWalker<T>::DoWalk   Add term  384  0.70 %  1490016 500105
    21  mach_port_insert_right   Add term  346  0.63 %  
    22  nsTArray_Impl<T>::ReplaceElementsAtInternal<T> | mozilla::TrackBuffersManager::InsertFrames   Add term  307  0.56 %  
    23  nsCycleCollector::ScanWeakMaps   Add term  297  0.54 %  
    24  nsTArray_Impl<T>::AppendElementsInternal<T> | nsHtml5TreeBuilder::createElement   Add term  291  0.53 %  
    25  mozilla::TrackBuffersManager::GetSample   Add term  287  0.52 %  1247189
    26  AVF_CreateMediaAcceleratorInterface   Add term  282  0.51 %  
    27  js::jit::ICStub::trace   Add term  280  0.51 %  1651704 1247257 1236759 1192988
    28  mach_msg_trap   Add term  266  0.49 %  1409699 1234370
    29  js::GCMarker::eagerlyMarkChildren   Add term  261  0.48 %  1441002 1359657 1337578 668583
    30  nsGlobalWindowInner::ClearDocumentDependentSlots   Add term  259  0.47 %  1576452 1573464 1422313 1405521
    31  OOM | unknown | js::AutoEnterOOMUnsafeRegion::crash | pref_Lookup   Add term  243  0.44 %  
    32  js::frontend::GeneralParser<T>::functionFormalParametersAndBody   Add term  242  0.44 %  
    33  nsTArray_Impl<T>::AppendElementsInternal<T> | nsHtml5TreeBuilder::endTag   Add term  241  0.44 %  
    34  vDSP_vclipD   Add term  234  0.43 %  
    35  MD_PrepareForGTPin   Add term  231  0.42 %  
    36  HALObject::Teardown   Add term  226  0.41 %  
    37  CCGraphBuilder::NoteWeakMapping   Add term  225  0.41 %  1538552 1125653
    38  core::option::expect_none_failed | lucet_wasi::ctx::WasiCtxBuilder::new   Add term  221  0.40 %  
    39  mozilla::dom::JSActorManager::ReceiveRawMessage   Add term  209  0.38 %  1563825
    40  mozilla::dom::FragmentOrElement::CanSkip   Add term  201  0.37 %  791437
    41  VideoToolbox@0xff416   Add term  199  0.36 %  
    42  nsCycleCollector::Collect   Add term  197  0.36 %  1528699
    43  OSSpinLockLock   Add term  196  0.36 %  
    44  CTFontCopyFamilyName   Add term  194  0.35 %  
    45  js::array_push   Add term  192  0.35 %  
    46  ClearCycleCollectorCleanupData   Add term  182  0.33 %  979059
    47  js::TenuringTracer::traceObject   Add term  179  0.33 %  1494271 1243151
    48  IPCError-browser | ShutDownKill | __psynch_cvwait | mozilla::TaskController::GetRunnableForMTTask | mozilla::ipc::MessagePump::Run   Add term  176  0.32 %  1674388
    49  IOAccelResourceTestEvent   Add term  171  0.31 %  
    50  js::gc::Arena::finalize<T>   Add term  167  0.30 %  1358871 702531

https://crash-stats.mozilla.org/search/?cpu_arch=amd64&platform=Mac%20OS%20X&process_type=content&date=%3E%3D2020-11-16T22%3A49%3A00.000Z&date=%3C2020-11-30T22%3A49%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Note also the disproportionately large number of the following types of content process crashes on ARM64:

    1  _pthread_join   Add term  1070  79.26 %  1679513 1678226
    2  libsystem_pthread.dylib@0x8ad8   Add term  276  20.44 %  1679373

I suspect that at least some of these crashes actually happened elsewhere (with different signatures).

There's a race in the crash reporting code that's causing content process crashes to crash the main process. See bug 1644249 and bug 1624467 for the fix.

As best I can tell, this disparity isn't present for browser/main process crashes, or at least not to the same degree:

macOS browser process crashes reported over the last 14 days on Apple Silicon:

    1  MacFileUtilities::MachoWalker::FindHeader   Add term  1948  74.55 %  1676102
    2  CrashReporter::OnChildProcessDumpWritten   Add term  502  19.21 %  1644249
    3  _pthread_join   Add term  74  2.83 %  1679513 1678226
    4  libsystem_pthread.dylib@0x8ad8   Add term  33  1.26 %  1679373
    5  OUTLINED_FUNCTION_0   Add term  9  0.34 %  
    6  mozilla::a11y::GeckoTextMarker::ContainerAsHyperTextWrap   Add term  8  0.31 %  1679075
    7  CrashChannel::OpenContentStream   Add term  6  0.23 %  1515987
    8  XUL@0x541af3c | XUL@0x541af38 | XUL@0x541d334 | _pthread_start   Add term  4  0.15 %  
    9  shutdownhang | __psynch_cvwait | _pthread_cond_wait   Add term  4  0.15 %  
    10  XUL@0x38a7f38 | XUL@0x38a7f34 | XUL@0x38ab3f8 | XUL@0x38ab1e0 | libsystem_pthread.dylib@0x7068   Add term  3  0.11 %  
    11  _platform_memcmp | nsAuthGSSAPI::nsAuthGSSAPI   Add term  3  0.11 %  
    12  dyld_stub_binder   Add term  3  0.11 %  730633
    13  libsystem_pthread.dylib@0x8ad0   Add term  3  0.11 %  
    14  XUL@0x38b26c8 | XUL@0x38b1e58 | XUL@0x38b1e58 | XUL@0x38b1c9c | XUL@0x38af808 | XUL@0x38af644 | XUL@0x38ae724 | XUL@0x38ad2d0 | XUL@0x38ab360 | XUL@0x38ab1c0 | libsystem_pthread.dylib@0x7068   Add term  2  0.08 %  
    15  XUL@0x3584cf4 | XUL@0x66f885c | XUL@0x3577804 | XUL@0x1b8468 | XUL@0x1b143c | XUL@0x5ac508 | XUL@0x5a9d54 | XUL@0x5a97d4 | XUL@0x5b52b8 | XUL@0x84561c | XUL@0x757bf4 | XUL@0x6852f8 | XUL@0x6842e8 | XUL@0x684d30 | XUL@0xee770 | XUL@0xed490 | XUL@0xec730...   Add term  1  0.04 %  
    16  XUL@0x388e880 | XUL@0x388e87c | XUL@0x389233c | XUL@0x3892124 | libsystem_pthread.dylib@0x7068   Add term  1  0.04 %  
    17  XUL@0xa4c4cc | XUL@0x658d570 | XUL@0xa32260 | XUL@0x11176c | XUL@0xa5a72c | XUL@0xa5bd20 | XUL@0x398cea4 | XUL@0x398dfc8 | XUL@0x3b14414 | XUL@0x398fe58 | XUL@0x3986298 | XUL@0x3981634 | XUL@0x398e37c | XUL@0x39b1bf0 | XUL@0x39b1ebc | XUL@0x39b1d50 | X...   Add term  1  0.04 %  
    18  __pthread_kill | pthread_kill   Add term  1  0.04 %  
    19  _dispatch_lane_class_dispose.cold.3   Add term  1  0.04 %  
    20  _dispatch_main_queue_callback_4CF   Add term  1  0.04 %  
    21  dlopen_internal   Add term  1  0.04 %  
    22  js::ctypes::ConvertToJS   Add term  1  0.04 %  1448144 1371207 1363630 718227 630033
    23  mozilla::dom::WindowGlobalParent::GetBrowserParent   Add term  1  0.04 %  1668682
    24  mozilla::dom::ipc::SharedStringMap::SharedStringMap   Add term  1  0.04 %  1514734
    25  shutdownhang | libsystem_kernel.dylib@0x4548   Add term  1  0.04 %

https://crash-stats.mozilla.org/search/?cpu_arch=arm64&platform=Mac%20OS%20X&process_type=browser&date=%3E%3D2020-11-16T23%3A16%3A00.000Z&date=%3C2020-11-30T23%3A16%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

macOS browser process crashes reported over the last 14 days on Intel:

    1  objc_exception_rethrow | -[NSOpenGLLayer drawInOpenGLContext:pixelFormat:forLayerTime:displayTime:]   Add term  4414  7.35 %  1601985
    2  mozilla::dom::ipc::WritableSharedMap::WritableSharedMap   Add term  2431  4.05 %  1490846
    3  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | mozilla::detail::MutexImpl::unlock | XPCArrayHomogenizer::GetTypeForArray   Add term  2043  3.40 %  1655236
    4  __pthread_kill | abort | gpusGenerateCrashLog.cold.1   Add term  1757  2.93 %  1665887 1576767 1535120
    5  objc_msgSend | _NSAccessibilityUIElementForSpecifier   Add term  1708  2.84 %  
    6  shutdownhang | __CFRunLoopServiceMachPort   Add term  1353  2.25 %  
    7  NS_CycleCollectorSuspect3   Add term  1085  1.81 %  1586494 1579858 1562232 1544357 1517464 1442854 1402181 1395104 1379913 1377387 1367715 1220667 1198500 1183651 1151643 1133892 1132339 1132199 1116067 1031852 1031697 991626 970767 967985 899761 892588
    8  shutdownhang | __psynch_cvwait | _pthread_cond_wait | _pthread_ulock_cond_cleanup   Add term  1051  1.75 %  1664472
    9  objc_msgSend | CA::Layer::remove_sublayer   Add term  790  1.32 %  1658432
    10  pthread_mutex_lock | CFRunLoopSourceSignal   Add term  728  1.21 %  1664355
    11  CrashReporter::OnChildProcessDumpWritten   Add term  692  1.15 %  1644249
    12  __pthread_kill | abort | gpusGenerateCrashLog   Add term  620  1.03 %  1535120
    13  AsyncShutdownTimeout | Places Connection shutdown | PlacesUtils read-only connection closing as part of Places shutdown   Add term  612  1.02 %  1402103 1326309
    14  __pthread_kill | abort | _RegisterApplication   Add term  583  0.97 %  1655801
    15  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | NS_InvokeByIndex   Add term  575  0.96 %  
    16  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | mozilla::net::nsSocketTransportService::OnDispatchedEvent   Add term  556  0.93 %  
    17  AsyncShutdownTimeout | Places Clients shutdown | sanitize.js: Sanitize on shutdown   Add term  508  0.85 %  1578273 1524200 1507171 1426941 1404105
    18  js::GCMarker::processMarkStackTop   Add term  475  0.79 %  1476239 1415708 1400478 1395740 1386875 1257309 952381 941491 792226 772441 745334 730283 719114
    19  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | mozilla::detail::MutexImpl::unlock | nsThread::Shutdown | nsThreadManager::Shutdown   Add term  446  0.74 %  
    20  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | xpc_DumpJSStack   Add term  444  0.74 %  
    21  nsXPCWrappedJS::Release   Add term  436  0.73 %  1517464 1427103 1353660 1133892 912110
    22  shutdownhang | __psynch_cvwait | _pthread_cond_wait | _pthread_cond_check_init   Add term  410  0.68 %  
    23  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | xpc::AutoScriptActivity::AutoScriptActivity   Add term  407  0.68 %  
    24  MacFileUtilities::MachoWalker::FindHeader   Add term  380  0.63 %  1676102
    25  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | firefox@0x2ce2   Add term  353  0.59 %  
    26  _dispatch_client_callout   Add term  352  0.59 %  1631803
    27  mozilla::dom::ServiceWorkerRegistrar::GetShutdownPhase   Add term  313  0.52 %  1603684 1471720 1403348
    28  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | mozilla::TaskController::GetRunnableForMTTask | arena_t::MallocSmall | mozilla::detail::MutexImpl::unlock | mozilla::layers::CompositorThreadHolder::Shutdown   Add term  302  0.50 %  
    29  <name omitted> | nsBaseWidget::GetLayerManager   Add term  298  0.50 %  
    30  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | _pthread_cond_updateval   Add term  280  0.47 %  
    31  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | mozilla::ipc::MessageChannel::SynchronouslyClose   Add term  273  0.45 %  
    32  mozilla::XPCOMThreadWrapper::Dispatch   Add term  271  0.45 %  
    33  wr_moz2d_render_cb   Add term  261  0.43 %  1664653 1630774 1524280 1516011 1511131 1509870 1486198 1466613 1449545 1449056 1449054 1447997 1447863
    34  shutdownhang | libsystem_kernel.dylib@0x1ca16   Add term  256  0.43 %  1676851 1518645
    35  __pthread_kill | abort | gpusKillClient   Add term  250  0.42 %  1660003 1379408
    36  webrender::prepare::prepare_interned_prim_for_render   Add term  241  0.40 %  1660675 1511493
    37  IOAccelContextAddResource   Add term  240  0.40 %  1533143
    38  objc_exception_throw | -[NSObject doesNotRecognizeSelector:] | ___forwarding___ | __forwarding_prep_0___ | _OPTCEventHandler   Add term  234  0.39 %  
    39  lookUpImpOrForward   Add term  230  0.38 %  1569462
    40  alloc::vec::Vec<T>::reserve   Add term  229  0.38 %  
    41  libpkcs11-dnietif.so@0xb2c4   Add term  225  0.37 %  1533422
    42  __pthread_kill | abort | cvms_element_get_content_internal   Add term  222  0.37 %  
    43  @0x0 | CamTwist@0x299f   Add term  220  0.37 %  
    44  CFRelease | -[__NSDictionaryM dealloc]   Add term  217  0.36 %  
    45  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | __mac_syscall   Add term  216  0.36 %  
    46  shutdownhang | __psynch_cvwait | _pthread_cond_wait | pthread_cond_signal_thread_np | <name omitted> | WatchdogManager::RecordContextActivity   Add term  203  0.34 %  
    47  _os_unfair_lock_corruption_abort | _os_unfair_lock_lock_slow   Add term  201  0.33 %  
    48  -[ChildView pasteboard:item:provideDataForType:]   Add term  196  0.33 %  
    49  shutdownhang | libsystem_kernel.dylib@0x16db6   Add term  190  0.32 %  1379619 1279691 1268528 1149287
    50  objc_msgSend | __CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__   Add term  186  0.31 %

https://crash-stats.mozilla.org/search/?cpu_arch=amd64&platform=Mac%20OS%20X&process_type=browser&date=%3E%3D2020-11-16T23%3A16%3A00.000Z&date=%3C2020-11-30T23%3A16%3A00.000Z&_facets=signature&_facets=version&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

(In reply to Gabriele Svelto [:gsvelto] from comment #1)

There's a race in the crash reporting code that's causing content process crashes to crash the main process. See bug 1644249 and bug 1624467 for the fix.

Ah yes! OK.

Crash reporting for AArch64 macs seem to be working now, closing this.

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.