1631803 - Crash in [@ _dispatch_client_callout]

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[[:philipp]]

This bug is for crash report bp-a574933e-b108-40ea-991d-8b3960200421.

Top 8 frames of crashing thread:

0 XUL CrashReporter::TerminateHandler toolkit/crashreporter/nsExceptionHandler.cpp:1807
1 libc++abi.dylib std::terminate 
2 libdispatch.dylib _dispatch_client_callout 
3 libdispatch.dylib _dispatch_root_queue_drain 
4 libdispatch.dylib _dispatch_worker_thread2 
5 libsystem_pthread.dylib _pthread_wqthread 
6 libsystem_pthread.dylib start_wqthread 
7 libdispatch.dylib _dispatch_root_queues_init_once 

these crashes from all macos versions with MOZ_CRASH(Unhandled exception) have been around for a while in low-to-mid volume. numerous comments in crash reports are mentioning using "camtwist" as software and urls are all sorts of sites that may request webcam access.

Comment 1

[Stephen A Pohl [:spohl]]

Many crash reports show that there are additional modules such as "CamTwist" or "EpocCamPlugIn" installed. It is conceivable that these modules fail to properly handle exceptions from the OS, which triggers our crash reporter. Sending over to WebRTC for a look.

Comment 2

[Karl Tomlinson (:karlt)]

This sounds potentially related to video capture. Any reason to suspect MediaRecorder?

Comment 3

[Nico Grunbaum [:ng, @chew:mozilla.org]]

Karl, no, I was mistaken about the component. I was trying to find a more specific home than WebRTC: A/V. I guess there may not be one.

Comment 4

[Alex Chronopoulos [:achronop]]

Dan, can you please help me triage this one? I think it is camera related

Comment 5

[Dan Minor [:dminor]]

I wasn't familiar with them, but it appears that both CamTwist and EpocCamPlugIn pretend to be webcams, so it's plausible that they're causing problems with our OS X camera capture code. It looks like EpocCamPlugin has a free version, so I guess we could try installing that and seeing if we can reproduce this. I'm not super excited about installing it on my laptop.

Another option would be to audit the current capture code and see if there's any places where we're failing to check for failures.

We're also running deprecated camera code on OS X, we have Bug 1451394 on file to consider switching to something newer.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jib, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 7

[Dan Minor [:dminor]]

Unfortunately, I haven't found time to investigate this further.

Comment 8

[ben]

This is anecdotal, but it may be useful for identification and remediation.

Earlier today I'd submitted crash report b8f7918d-45ff-42c6-ac83-1813a0210210 which identified as potentially being related to this bug. It was a hard crash I could replicate consistently when visiting a Google Meet link. In this ticket I noticed Steven's original mention that CamTwist may have been related to the issue. I remembered I had it installed on my machine for a purpose I no longer remember, so I removed it. Subsequent attempts to crash Firefox visiting the same Google Meet link were not successful, so whatever CamTwist was doing to cause the error was corrected when it was removed.

I hope a field report helps to narrow the surface you need to wade through. Please let me know if I can be of any additional assistance.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 5 desktop browser crashes on Mac on beta

:jib, could you consider increasing the severity of this top-crash bug?

For more information, please visit auto_nag documentation.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit auto_nag documentation.

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

Based on the topcrash criteria, the crash signature linked to this bug is not a topcrash signature anymore.

For more information, please visit auto_nag documentation.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The bug is linked to a topcrash signature, which matches the following criterion:

• Top 5 desktop browser crashes on Mac on beta

:jib, could you consider increasing the severity of this top-crash bug?

For more information, please visit auto_nag documentation.

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

bp-4e65a755-f848-44ee-bf70-c9c3d0221005 seems to have a useful stack:

Crashing Thread (95), Name: VideoCapture
Frame 	Module 	Signature 	Source 	Trust
0 	XUL 	CrashReporter::TerminateHandler() 	toolkit/crashreporter/nsExceptionHandler.cpp:1838 	context
1 	libc++abi.dylib 	std::__terminate(void (*)()) 		cfi
2 	libc++abi.dylib 	std::terminate() 		cfi
3 	libobjc.A.dylib 	objc_terminate 		cfi
4 	libdispatch.dylib 	_dispatch_client_callout 		cfi
5 	libdispatch.dylib 	_dispatch_once_callout 		cfi
Ø 6 	exceedshare-mac-virtualcam 	exceedshare-mac-virtualcam@0x000000000000cb20 		cfi
Ø 7 	exceedshare-mac-virtualcam 	exceedshare-mac-virtualcam@0x000000000000f4c0 		frame_pointer
8 	CoreMediaIO 	CMIO::DAL::PlugInManagement::CreatePlugIn(CMIO::DAL::CFPlugIn const*, std::__1::vector<CMIO::DAL::PlugIn*, std::__1::allocator<CMIO::DAL::PlugIn*> >*) 		frame_pointer
9 	CoreMediaIO 	CMIO::DAL::PlugInManagement::CreateOrLazyLoadPlugIns(std::__1::vector<CMIO::DAL::CFPlugIn*, std::__1::allocator<CMIO::DAL::CFPlugIn*> >*, std::__1::vector<CMIO::DAL::PlugIn*, std::__1::allocator<CMIO::DAL::PlugIn*> >*, std::__1::vector<CMIO::DAL::PlugInManagement::MatchingInfo*, std::__1::allocator<CMIO::DAL::PlugInManagement::MatchingInfo*> >*, void (*)(void*, unsigned int)) 		cfi
10 	CoreMediaIO 	CMIO::DAL::PlugInManagement::Initialize() 		cfi
11 	CoreMediaIO 	CMIO::DAL::System::InitializeDevices() 		cfi
12 	CoreMediaIO 	CMIO::DAL::System::CheckOutInstance() 		cfi
13 	CoreMediaIO 	CMIOObjectGetPropertyDataSize 		cfi
14 	AVFCapture 	+[AVCaptureDALDevice _refreshDevices] 		cfi
15 	AVFCapture 	__39+[AVCaptureDALDevice _ensureDeviceList]_block_invoke 		cfi
16 	libdispatch.dylib 	_dispatch_client_callout 		cfi
17 	libdispatch.dylib 	_dispatch_once_callout 		cfi
18 	AVFCapture 	+[AVCaptureDALDevice devices] 		cfi
19 	AVFCapture 	+[AVCaptureDevice_Tundra _devicesWithAllowIOSMacEnvironment:] 		cfi
20 	AVFCapture 	+[AVCaptureDevice_Tundra devicesWithMediaType:] 		cfi
21 	XUL 	+[DeviceInfoIosObjC captureDeviceCount] 	dom/media/systemservices/objc_video_capture/device_info_objc.mm:45 	cfi
22 	XUL 	webrtc::videocapturemodule::DeviceInfoIos::Init() 	dom/media/systemservices/objc_video_capture/device_info.mm:46 	cfi
23 	XUL 	webrtc::videocapturemodule::DeviceInfoIos::DeviceInfoIos() 	dom/media/systemservices/objc_video_capture/device_info.mm:37 	cfi
24 	XUL 	webrtc::videocapturemodule::DeviceInfoIos::DeviceInfoIos() 	dom/media/systemservices/objc_video_capture/device_info.mm:37 	inlined
24 	XUL 	webrtc::videocapturemodule::VideoCaptureImpl::CreateDeviceInfo() 	dom/media/systemservices/objc_video_capture/device_info.mm:35 	cfi
25 	XUL 	mozilla::camera::VideoEngine::GetOrCreateVideoCaptureDeviceInfo() 	dom/media/systemservices/VideoEngine.cpp:184 	cfi
26 	XUL 	mozilla::camera::CamerasParent::SetupEngine(mozilla::camera::CaptureEngine) 	dom/media/systemservices/CamerasParent.cpp:358 	cfi
27 	XUL 	mozilla::camera::CamerasParent::EnsureInitialized(int) 	dom/media/systemservices/CamerasParent.cpp:416 	cfi
28 	XUL 	mozilla::camera::CamerasParent::RecvEnsureInitialized(mozilla::camera::CaptureEngine const&)::$_4::operator()() const 	dom/media/systemservices/CamerasParent.cpp:470

Line 21 in that stack seems to implicate this code added in bug 1629984, which enumerates devices early, though these crashes seem to predate this code, so it may just be the unlucky code that happens to touch devices first now.

I'm marking it as a regression anyway since its stack is involved now, and bug 1629984, even if it's not to blame, gives me an idea for a potential fix: what if we add a try/catch around each device in that for-loop and treated any exception-throwing device as we treat isSuspended?

Paul, WDYT? Or is CamTwist a lost cause? We could also dup this to bug 1451394 I suppose though we don't know if that would help any.

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1629984

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hmm, making it a see-also instead.

Comment 16

[Paul Adenot (:padenot)]

I don't have a better idea, let's do that.

Comment 17

[Paul Adenot (:padenot)]

Attachment: Bug 1631803 - Catch exception when computing the number of cameras on macOS to prevent a crash. r?jib

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4b6425be319
Catch exception when computing the number of cameras on macOS to prevent a crash. r=jib

Comment 19

[Cristian Tuns]

Backed out for causing build bustages on device_info_objc.mm

• Backout link
• Push with failures
• Failure Log
• Failure line: /builds/worker/checkouts/gecko/dom/media/systemservices/objc_video_capture/device_info_objc.mm:53:5: error: use of undeclared identifier 'ctn'

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6436237b8b5d
Catch exception when computing the number of cameras on macOS to prevent a crash. r=jib

Comment 21

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/6436237b8b5d

Comment 22

[Steven Michaud [:smichaud] (Retired)]

For the record, crashes with this signature are still happening in build(s) with the patch from comment #20. But it's not the same bug -- the crash stacks are very different, and not on the VideoCapture thread.

https://crash-stats.mozilla.org/search/?signature=~dispatch_client_callout&build_id=%3E%3D20221010214639&version=108.0a1&platform=Mac%20OS%20X&date=%3E%3D2022-10-06T22%3A32%3A00.000Z&date=%3C2022-10-20T22%3A32%3A00.000Z&_facets=signature&_facets=proto_signature&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-proto_signature