Open Bug 1798503 Opened 2 years ago Updated 5 months ago

[macOS 13] Crash in [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]], [@ _dispatch_client_callout ]

Categories

(Core :: WebRTC: Audio/Video, defect)

Product:
Core
Component:
WebRTC: Audio/Video
Type:
defect

Tracking

()

Tracking Flags:
Tracking Status
firefox-esr102 --- affected
firefox106 --- wontfix
firefox107 --- affected
firefox108 --- affected

People

(Reporter: aryx, Unassigned, NeedInfo)

References

(Blocks 1 open bug,
URL
)

Details

(Keywords: crash, Whiteboard: STR in comment #23)

Crash Data

Crashes across the board of recent Firefox versions but only with macOS 13 13.0.0 22A380 (52 crashes) and 13.0.0 22A5365d (1 crash). If it's a macOS issue, it will likely get fixed in a future alpha version.

Crash report: https://crash-stats.mozilla.org/report/index/8c4c4a83-7fa3-4650-95e7-46e2e0221101

Reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESSTop 10 frames of crashing thread:

0  libobjc.A.dylib  objc_msgSend  
1  AVFCapture  -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]  
2  AVFCapture  -[AVCaptureDALDevice _refreshStreamsAndKVONotify:]  
3  AVFCapture  -[AVCaptureDALDevice _refreshPropertiesAndKVONotify:]  
4  AVFCapture  -[AVCaptureDALDevice _refreshConnectionID:KVONotify:]  
5  AVFCapture  -[AVCaptureDALDevice initWithUniqueID:connectionID:]  
6  AVFCapture  +[AVCaptureDALDevice _refreshDevices]  
7  AVFCapture  __39+[AVCaptureDALDevice _ensureDeviceList]_block_invoke  
8  libdispatch.dylib  _dispatch_client_callout  
9  libdispatch.dylib  _dispatch_once_callout
Crash Signature: [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] → [@ _dispatch_client_callout] [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]]
Summary: [macOS 13] Crash in [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] → [macOS 13] Crash in [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]], [@ _dispatch_client_callout ]

Almost all the crashes with "refreshFormatsAndKVONotify" in their proto signatures happen on the macOS 13 release (build 22A380). Also, all the crash reports have camera-related non-Apple modules in their lists of modules. Though none of these show up in the crash stacks, I suspect they're involved. Almost all the crashes are startup crashes (happening in the first 30 seconds of uptime).

Here are the non-Apple camera-related modules that show up in all the lists of modules that I've examined:

libVtcComLib.dylib
libavCam.dylib
RelayCam

I suspect these crashes happen when macOS tries to load a third party "DAL plugin". I can't find information about any of these modules. Note that two of them are mentioned in bug 1721954.

https://crash-stats.mozilla.org/search/?proto_signature=~refreshFormatsAndKVONotify&platform=Mac%20OS%20X&date=%3E%3D2022-08-01T17%3A20%3A00.000Z&date=%3C2022-11-01T17%3A20%3A00.000Z&_facets=signature&_facets=platform_version&_facets=proto_signature&_facets=uptime&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-platform_version

See Also: → 1721954

There's a company called Vitek, many of whose cameras have model numbers starting with VTC.

"VTC" also seems to mean "Video Teleconferencing" -- e.g. https://ieeexplore.ieee.org/document/473827.

Aryx, why did you add the _dispatch_client_callout signature to this bug? Those crashes seem unrelated.

Flags: needinfo?(aryx.bugmail)

Both signatures have webrtc::videocapturemodule::DeviceInfoIos::Init() as common ancestor in the stack of the crashing thread.

Flags: needinfo?(aryx.bugmail)
Flags: needinfo?(apehrson)

If we look only at macOS 13, then it's 46 reports for webrtc::videocapturemodule::DeviceInfoIos::Init() and 14 without for the last week.

The bug is linked to a topcrash signature, which matches the following criterion:

  • Top 5 desktop browser crashes on Mac on beta

:jib, could you consider increasing the severity of this top-crash bug?

For more information, please visit auto_nag documentation.

Flags: needinfo?(jib)
Keywords: topcrash
Severity: -- → S2

Jeff, maybe you can help diagnose this?

Flags: needinfo?(jib) → needinfo?(jmuizelaar)
Flags: needinfo?(apehrson)
See Also: → 1631803

Another bug involving a DAL plugin.

See Also: → 1793538

The only lead I have on the RelayCam plugin is this thread suggesting it could be part of Mersive's "Solstice Conference Camera".

In the Solstice app there's an option to install drivers. The installer for "SolsticeConference" says:

This software package will enable the support of a Mac virtual monitor, allowing your conferencing software to share a Solstice display externally, as well as an plugin enabling Pod camera during conference.
 
Please note this version of updates is compatible and tested against macOS 11.0.1 (Big Sur) and 12.0 (Monterey).

I installed this, gave Firefox a restart, and camera enumeration now showed "Solstice Conference Camera". A SIGABRT later and I have this crash report that shows all the modules from comment 1; "RelayCam", "libVtcComLib.dylib" and "libavCam.dylib".

Note that I'm on macOS 12.

I filed a support ticket with Mersive for this.

I've now installed Mersive Solstice on macOS 13 and 12.6.1. On macOS 13 I crash (with this bug's crash signature) just by visiting a link that uses any installed video cameras, including virtual ones implemented in DAL plugins (like RelayCam):

https://webrtc.github.io/samples/src/content/devices/input-output/

bp-91280722-9d08-45e6-90f1-e90ef0221109

I don't get any crashes on macOS 12.6.1, though. I tried using the STR at bug 1793538 comment #21.

A SIGABRT later

How did you trigger this, Andreas? Was it just kill -6 [firefox pid]?

(In reply to Steven Michaud [:smichaud] (Retired) from comment #13)

A SIGABRT later

How did you trigger this, Andreas? Was it just kill -6 [firefox pid]?

Yes, basically. I think I did killall -SIGABRT firefox. Just to get a crash report.

For what it's worth, Mersive Solstice also installs two "HAL plugins" (to /Library/Audio/Plug-Ins/HAL):

SolsticeAudio.driver
SolsticeDesktopSpeakers.driver

And it creates a login item called RelayXpc, which is part of SolsticeDesktopSpeakers.driver. This via a com.mersive.ipc.relay.RelayXpc.plist file in /Library/LaunchDaemons. Its operation can be controlled via "Login items" in the System Preferences panel. It seems to be used to start the Mersive Solstice app whenever you log in.

I tried to reproduce this on my M1 Mac with MacOS 13 but it does not seem to reproduce on this hardware for me following the steps provided.

Oops, I just installed Mersive Solstice on my Apple Silicon Mac Mini (running macOS 13.0.1), and (like you) discovered that my STR don't work there. I don't know why. I'll keep testing.

I had no problems reproducing the crashes on an Intel MacBook Pro running macOS 13 and 13.0.1.

Edit: According to crash-stats, all these crashes happen on Intel hardware:

https://crash-stats.mozilla.org/search/?proto_signature=~refreshFormatsAndKVONotify&platform=Mac%20OS%20X&date=%3E%3D2022-08-10T01%3A36%3A00.000Z&date=%3C2022-11-10T01%3A36%3A00.000Z&_facets=signature&_facets=platform_version&_facets=proto_signature&_facets=uptime&_facets=cpu_arch&_sort=-date&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-cpu_arch

FWIW I noticed the plugin logs a bunch of trace info when run under lldb, e.g.,

2022-11-14 14:00:57.390713+0100 firefox[45410:10427594] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x15920dd60> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2022-11-14 14:00:57.597381+0100 firefox[45410:10427594] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x1592dc400> 30010C1C-93BF-11D8-8B5B-000A95AF9C6A
[/Users/build/sandbox/BuildRepos/Sol-Windows-3/External/VirtualMonitor-macOS/MersiveRelayCam/RelayPlugin/VtcComLib/VtcComLib.cpp] initializer()
[/Users/build/sandbox/BuildRepos/Sol-Windows-3/External/VirtualMonitor-macOS/MersiveRelayCam/RelayPlugin/avCam/avCam/avCamLib.mm] initializer()
2022-11-14 14:00:57.639335+0100 firefox[45410:10427594] 😂 FrameInput: DLL version -5.5.0
2022-11-14 14:00:57.639455+0100 firefox[45410:10427594] 😂 FrameInput: Load DLL successfully!
2022-11-14 14:00:57.639523+0100 firefox[45410:10427594] 😂 FrameInput: getInstance -Ref= 1
2022-11-14 14:00:57.640074+0100 firefox[45410:10427594] 😂 FrameInput::detectPodCamera failed to read configuration data
2022-11-14 14:00:57.640122+0100 firefox[45410:10427594] 😂 Device: Initialize
2022-11-14 14:00:57.640241+0100 firefox[45410:10427594] 😂 Stream::Initialize
2022-11-14 14:00:57.640293+0100 firefox[45410:10427594] 😂 setTarget4CC
2022-11-14 14:00:57.640342+0100 firefox[45410:10427594] 😂 Stream::Initialize - is Input
2022-11-14 14:00:57.640413+0100 firefox[45410:10427594] 😂 Stream::Initialize - is Input - set Input clock
2022-11-14 14:00:57.640499+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyDirection
2022-11-14 14:00:57.640611+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyDeviceUID
2022-11-14 14:00:57.640656+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyTransportType
2022-11-14 14:00:57.640732+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=666669642A2A2A2AFFFFFFFF
2022-11-14 14:00:57.640766+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.640837+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.640908+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=6B6E696F2A2A2A2AFFFFFFFF
2022-11-14 14:00:57.640960+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.640995+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.641029+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=207466702A2A2A2AFFFFFFFF
2022-11-14 14:00:57.641066+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.641094+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.676066+0100 firefox[45410:10427594] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x1592dc400> 30010C1C-93BF-11D8-8B5B-000A95AF9C6A
2022-11-14 14:00:57.775397+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyDeviceIsRunningSomewhere
2022-11-14 14:00:57.775475+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyDeviceUID
2022-11-14 14:00:57.775590+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=6E76696C626F6C6700000000
2022-11-14 14:00:57.775655+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.775735+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.775784+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=6B6E696F626F6C6700000000
2022-11-14 14:00:57.775830+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.775888+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.775957+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=236D747374706E6900000000
2022-11-14 14:00:57.776024+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.776096+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.776171+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=64756C7074706E6900000000
2022-11-14 14:00:57.776238+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.776301+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.776379+0100 firefox[45410:10427594] 😂 Device::SetPropertyData -kCMIOObjectPropertyListenerAdded data=75796273626F6C6700000000
2022-11-14 14:00:57.776456+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.776507+0100 firefox[45410:10427594] 😂 Device::PropertyListenerAdded -Done
2022-11-14 14:00:57.776640+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyTransportType
2022-11-14 14:00:57.776692+0100 firefox[45410:10427594] 😂 ==Stream: setPropertyData -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.776746+0100 firefox[45410:10427594] 😂 Stream: IsPropertySettable -kCMIOObjectPropertyListenerAdded
2022-11-14 14:00:57.776819+0100 firefox[45410:10427594] 😂 DP::Relay::Stream::PropertyListenerAdded
2022-11-14 14:00:57.776915+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFormatDescriptions
2022-11-14 14:00:57.776977+0100 firefox[45410:10427594] 😂 Pod resolutions -
2022-11-14 14:00:57.777101+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRateRanges
2022-11-14 14:00:57.777159+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRateRanges
2022-11-14 14:00:57.777276+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRateRanges
2022-11-14 14:00:57.777334+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRateRanges
2022-11-14 14:00:57.777426+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFormatDescription
2022-11-14 14:00:57.777487+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRateRanges
2022-11-14 14:00:57.777581+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRate
2022-11-14 14:00:57.777646+0100 firefox[45410:10427594] 😂 Stream: getPropertyData -kCMIOStreamPropertyFrameRate
2022-11-14 14:00:57.778014+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIOObjectPropertyName
2022-11-14 14:00:57.778074+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyModelUID
2022-11-14 14:00:57.778140+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIOObjectPropertyManufacturer
2022-11-14 14:00:57.778235+0100 firefox[45410:10427594] 😂 Device getPropertyData -kCMIODevicePropertyHogMode
2022-11-14 14:00:57.778303+0100 firefox[45410:10427594] 😂 Device GET kCMIODevicePropertyHogMode -0xffffffff

Summary - this is an issue with a cameras related plugin that crashes when it initializes (Solstice Conference Camera). The crash is restricted to the latest MacOS 13 release. Since our cameras code is currently running in the parent process, this results in a browser crash. The issue also impacts Chrome.

We've reached out to the company and filed a issue ticket, and they indicated they are looking at it. We'll keep pushing there in the hope that they push an update with a fix.

Longer term we need to look at moving our camera access code out of parent into a utility process but that is a larger project we can't knock out in short time frame. This solution would also involve blocking all media related plugins of this type from that process, which might impact other 3rd party products so it's not the preferred fix.

I have heard back from Mersive:

I would like to let you know that the issue is being investigated on our side and our development team has planned a fix for the 5.5.3 release. At the moment macOS Ventura is not yet supported for Solstice.

Based on the topcrash criteria, the crash signatures linked to this bug are not in the topcrash signatures anymore.

For more information, please visit auto_nag documentation.

Keywords: topcrash
Severity: S2 → S3

For what it's worth, visiting the following URL always triggers this bug's crash when Solstice 5.5.1 is installed. To trigger the crash you may need to scroll down a bit. This only works on macOS 13.

https://stackoverflow.com/questions/7697508/how-do-you-codesign-framework-bundles-for-the-mac-app-store

But this STR doesn't work when Solstice 6.0 is installed (the current version). So Mersive may have fixed these crashes.

Has STR: --- → yes
Whiteboard: STR in comment #23

But this STR doesn't work when Solstice 6.0 is installed (the current version). So Mersive may have fixed these crashes.

Oops, not so fast. After restarting my computer, these crashes happen even with Solstice 6.0 -- with the URLs from comment #23 and comment #13.

bp-a00422aa-f47c-40e7-bf1f-943810230611
bp-442feeaf-8f6c-4530-810e-b723d0230611

Mersive has not fixed these crashes.

FWIW, in the release notes for 6.0 under Known Issues they say:

  • A known incompatibility between virtual cameras on macOS 13 (Ventura) running on Intel processors and Room Camera + Mic functions in the Mersive Solstice app can cause other applications to close when they request access to any installed video cameras. Mersive will address this issue in an upcoming release.

For 5.5.2 they say:

  • A known incompatibility exists between the Mersive Solstice Conference drivers used for Solstice-enabled video conferencing and macOS 13 (Ventura) running on Intel processors that crashes other applications when they request access to any installed video cameras, including virtual cameras. Mersive is working to implement a solution as soon as possible. Sharing content with the Mersive Solstice app is supported on macOS Ventura in Solstice 5.5.2, and Ventura is recommended for sharing PowerPoint presentations.

It looks like the issue still persist with Mersive 6.1.

To fix it temporarily, I have deleted everything (deleting drivers specific to Mersive only did not worked) from ‘/Library/Audio/Plug-Ins/HAL/‘ and then rebooted. Now pages are loading fine and I can avoid using Mersive until it gets fixed.

Even though I have deleted Teams/Zoom drivers, both are working fine for now.

(In reply to Rez Haque [:ntrezowan] from comment #26)

It looks like the issue still persist with Mersive 6.1.

To fix it temporarily, I have deleted everything (deleting drivers specific to Mersive only did not worked) from ‘/Library/Audio/Plug-Ins/HAL/‘ and then rebooted. Now pages are loading fine and I can avoid using Mersive until it gets fixed.

Even though I have deleted Teams/Zoom drivers, both are working fine for now.

This fixed my issue for crash bp-e0cefb57-6754-4077-a2c5-342800240604.

Crash Signature: [@ _dispatch_client_callout] [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] → [@ _dispatch_client_callout] [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] [@ object_getClass ]
Crash Signature: [@ _dispatch_client_callout] [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] [@ object_getClass ] → [@ __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ ] [@ _dispatch_client_callout] [@ objc_msgSend | -[AVCaptureDALDevice _refreshFormatsAndKVONotify:]] [@ object_getClass ]
You need to log in before you can comment on or make changes to this bug.