1707060 - Crash in [@ obs-virtualcam-module64.dll | webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS]

Status: REOPENED

(Core :: WebRTC: Audio/Video, defect, P3)

Description

[Sean Feng [:sefeng211]]

Crash report: https://crash-stats.mozilla.org/report/index/5c472183-1fc4-430b-b16e-2dc940210420

Reason: EXCEPTION_ACCESS_VIOLATION_EXEC

Top 10 frames of crashing thread:

0 obs-virtualcam-module64.dll obs-virtualcam-module64.dll@0xcdf7 
1 xul.dll webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS third_party/libwebrtc/webrtc/modules/video_capture/windows/video_capture_ds.cc:47
2 xul.dll rtc::RefCountedObject<webrtc::videocapturemodule::VideoCaptureDS>::~RefCountedObject third_party/libwebrtc/webrtc/rtc_base/refcountedobject.h:54
3 xul.dll rtc::RefCountedObject<webrtc::videocapturemodule::VideoCaptureDS>::Release const third_party/libwebrtc/webrtc/rtc_base/refcountedobject.h:40
4 xul.dll std::_Func_impl_no_alloc<`lambda at /builds/worker/checkouts/gecko/dom/media/systemservices/VideoEngine.cpp:123:19', void, mozilla::camera::VideoEngine::CaptureEntry&>::_Do_call 
5 xul.dll mozilla::camera::VideoEngine::WithEntry dom/media/systemservices/VideoEngine.cpp:244
6 xul.dll mozilla::camera::VideoEngine::ReleaseVideoCapture dom/media/systemservices/VideoEngine.cpp:123
7 xul.dll mozilla::media::LambdaRunnable<`lambda at /builds/worker/checkouts/gecko/dom/media/systemservices/CamerasParent.cpp:790:54'>::Run dom/media/systemservices/MediaUtils.h:76
8 xul.dll MessageLoop::DoWork ipc/chromium/src/base/message_loop.cc:552
9 xul.dll mozilla::ipc::MessagePumpForNonMainUIThreads::DoRunLoop ipc/glue/MessagePump.cpp:375

Comment 1

[Toshihito Kikuchi [:toshi]]

Unlike bug 1691902, this may be caused by a defect in our code, or webrtc. The crash happened because obs-virtualcam-module64.dll was already unloaded when we tried to release webrtc::videocapturemodule here. This may happen if we store the same object under multiple capture IDs (not sure how or even it's possible)? Moving this bug to the WebRTC component to triage.

Comment 2

[Byron Campen [:bwc]]

Any ideas here?

Comment 3

[Kent]

Hello,
I have added information about this on Github.
https://github.com/obsproject/obs-studio/issues/4243

All 3 times, I was ending a call on Doxy.me or changing from single call to group call.
Indeed it is when the camera is loaded and unloaded.

https://user-images.githubusercontent.com/51689757/116640805-5d640c80-a939-11eb-84ff-ae2fc731c75d.png
https://user-images.githubusercontent.com/51689757/116640833-6ce35580-a939-11eb-81ba-037b4df2a690.png
https://user-images.githubusercontent.com/51689757/116640860-7e2c6200-a939-11eb-8bed-474508d91c20.png

Comment 4

[Paul Adenot (:padenot)]

This is being worked on upstream, see the link in comment 3.

Comment 5

[Toshihito Kikuchi [:toshi]]

(In reply to Paul Adenot (:padenot) from comment #4)

This is being worked on upstream, see the link in comment 3.

https://github.com/obsproject/obs-studio/issues/4243 is bug 1691902. As I wrote in comment 1, I think this crash is a different issue.

Comment 6

[Byron Campen [:bwc]]

Still happening somewhat frequently in the field, according to crash-stats.

Comment 7

[Gabriele Svelto [:gsvelto]]

Added a new signature

Comment 8

[Gabriele Svelto [:gsvelto]]

... and another one

Comment 9

[Gabriele Svelto [:gsvelto]]

Affects all supported Windows versions.

Comment 10

[Christian Holler (:decoder)]

Adding back the needinfo. Jan, can you find someone to work on this? Apparently this is a different issue and we have a preliminary analysis in comment 1.

Comment 11

[crossan007]

I'm seeing this issue as well. It seems to happen fairly consistently every time I "hang up" (or turn off video) from a Google Meet which has been running for an hour.

My source "camera" device is the OBS virtual camera, with a resolution of 1920x1080.

Frame 0 and 1 in the stack trace seem to always be consistent (pasting here for search visibility)

Frame 0: obs-virtualcam-module64.dll@0x000000000000e247
Frame 1: void CFGControl::SetFirstVW(struct IVideoWindow*)

Frame 2 seems to be variable (probably a memory address?:)

and then the rest of the stack seems consistent as well:

Frame 3: webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS()
Frame 4: rtc::RefCountedObject<webrtc::videocapturemodule::VideoCaptureDS>::~RefCountedObject()

Link to crash reports:
https://crash-stats.mozilla.org/report/index/a70c755a-0c67-4ec8-922c-c50c80220113
https://crash-stats.mozilla.org/report/index/df95af1b-af6d-4c15-a7ce-9812e0220111
https://crash-stats.mozilla.org/report/index/88da9854-bc4e-4e99-a14f-78b930220111

Comment 12

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

I'm unable to reproduce. crossan007, how frequently does it happen for you? Does length of call or resolution matter? Does it reproduce even on a simple camera page?

Comment 13

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Andreas, any ideas stick out at you from the crashstats stacks in comment 11?

Firefox uses this same code path for all cameras, which I think points to the problem possibly being in the OBS virtual camera. Do you agree?

Comment 14

[crossan007]

That stack trace / crash happens every time I end one of my weekly Google Meet meetings where the duration is always something over an hour. It also occurs if I turn off my camera in the meeting (likely after some unknown duration, as I could not reproduce this in a short testing session).

I tried the simple camera page you linked and left it on overnight; the crash did not occur when I stopped the video the next morning.

Sorry I can't provide more than anecdotes at the moment. I'll try to capture more details and report back.

Comment 15

[crossan007]

Okay; I was just able to reproduce this.

I created a new Google Meet "Instant meeting" with only myself.

I started the meeting, and was in it for about 45 minutes. I clicked the camera icon to turn off my video, and I got this crash report: https://crash-stats.mozilla.org/report/index/6dcd2155-a688-4d60-9e15-a12ca0220121

I'll try again with shorter timeframes in an attempt to find the lower-bound of the duration which causes this issue.

Comment 16

[crossan007]

Happened again; Google Meet meeting started at 10:30 AM ET and ended at 10:50AM ET; as soon as I exited the call, I got this crash: https://crash-stats.mozilla.org/report/index/bb8b13ea-6719-461c-a48e-47ea50220125

Comment 17

[Andreas Pehrson [:pehrsons]]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #13)

Andreas, any ideas stick out at you from the crashstats stacks in comment 11?

Firefox uses this same code path for all cameras, which I think points to the problem possibly being in the OBS virtual camera. Do you agree?

Intuitively I agree, but without more details as to the root cause of this I don't dare say.

Comment 1 provides some clues. But we seem to balance AddRefs and Releases so that I think points to a ref count related obs-virtualcam bug?

I tried to validate this assumption but wasn't able. It's most certainly more complex than that.

Just trying to open Google Meet with a regular camera in a debug build I keep failing this thread check. This did not happen in a simple capture case like webrtc-landing. That suggests to me that we have some races that need to get resolved on our end before we can make assumptions about anything else.

Comment 18

[Gabriele Svelto [:gsvelto]]

I found two more signatures for this problem. The reason why the DLL doesn't show up in these crashes is that it's been unloaded (it appears in the unloaded module list if you have raw dump access).

Comment 19

[Gabriele Svelto [:gsvelto]]

Added new signatures with unloaded modules. Since we include the address of the crash within the unoaded module these are version specific.

Comment 20

[Gabriele Svelto [:gsvelto]]

Adjusted the remaining signatures and removed the empty ones.

Comment 21

[Gabriele Svelto [:gsvelto]]

And one more signature which appears related. The ones starting with BaseThreadInitThunk were added to bug 1691902 instead.

Comment 22

[Andreas Pehrson [:pehrsons]]

OBS shipped a fix for this issue in v28. We cannot see the module version in crash-stats after it has been unloaded, but this signature is of a variant of this bug before the OBS virtualcam module has fully unloaded.

28 does show up: 1, 2.

The unloaded obs-virtualcam modules under other crash signatures do have a code_id field available, so we can figure out their version from that, given the version and code_id known from the crashes when the module was still loaded. Here's a quick code_id to version mapping:
61534e09ce000 -> 27.1.1.0
6212fa56ce000 -> 27.2.1.0
62201cb3ce000 -> 27.2.3.0
630fb1d13b000 -> 28.0.0.0
633893dc3b000 -> 28.0.3.0
63603ed53b000 -> 28.1.0.0

We do have code_id for unloaded modules on the "Raw Data and Minidumps" tab, so here's a crash signature to code_ids found (from skimming through reports from the last 7 days) mapping:
CMsgMutex::Lock -> [5f720994e3000]
CFGControl::SetFirstVW -> [62201cb3ce000, 61534e09ce000, 6212fa56ce000]
(unloaded obs-virtualcam-module64.dll@0xe247) | CFGControl::SetFirstVW -> [61534e09ce000]
(unloaded obs-virtualcam-module64.dll@0xe4df) | CFGControl::SetFirstVW -> [62201cb3ce000]
webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS -> [5fd7c3c7e0000, 60b54185e1000]
(unloaded obs-virtualcam-module64.dll@0xcdf7) | webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS -> [5fd7c3c7e0000]
(unloaded obs-virtualcam-module64.dll@0xdbff) | webrtc::videocapturemodule::VideoCaptureDS::~VideoCaptureDS -> [60b54185e1000]

There are some unknown code_ids here but at least the v28 ones do not show up. I'll file an issue with OBS on the RtlFreeHeap crash that still seems to be ongoing.

Comment 23

[Andreas Pehrson [:pehrsons]]

Having dug around a bit more I think the RtlFreeHeap crash is closer related to bug 1691902.
Mainly because:

• The obs module is not unloaded, and the thread where the unload race crash happened is idle
• It happens on the IMemInputPin::Receive thread (probably while the obs module is doing some processing)
• The similarities with this related report from bug 1691902

Comment 24

[Gabriele Svelto [:gsvelto]]

(In reply to Andreas Pehrson [:pehrsons] from comment #22)

OBS shipped a fix for this issue in v28. We cannot see the module (and its version) in crash-stats after it has been unloaded, but this signature is of a variant of this bug before the OBS virtualcam module has fully unloaded.

The minidumps contain the code ID of the unloaded modules, would it be useful to surface them somehow?

Comment 25

[Andreas Pehrson [:pehrsons]]

I should have edited that sentence, I actually got these on the "Raw Data and Minidumps" tab. If there was a way to show them in an aggregate form (for filtering, grouping, or as a column in the reports list) however, that would certainly have helped.

Comment 26

[Gabriele Svelto [:gsvelto]]

OK, I filed bug 1797742 and bug 1797743 to make identifying those modules easier on Socorro w/o requiring protected data access.

Comment 27

[Andreas Pehrson [:pehrsons]]

FWIW https://github.com/obsproject/obs-studio/issues/7672 has been closed somewhat speculatively as fixed in OBS Studio 30.1 Beta 2.

Comment 28

[Andreas Pehrson [:pehrsons]]

I see a pattern in recent bug reports having ~VideoCaptureDS high in the crashing stack, that they all seem to crash in an unloaded virtual camera DLL. obs-virtualcam-module64.dll makes up most of them and it's open source so I had a look at how it prevents getting unloaded. They don't seem to be adhering to the docs so I filed obs-studio/10566 upstream.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.

Comment 30

[Gabriele Svelto [:gsvelto]]

This is a signature change.

Comment 31

[Gabriele Svelto [:gsvelto]]

More signatures.