1453899 - Crash in vcruntime140.dll@0xc3c7 | mozilla::gmp::GMPPlaneImpl::Copy

Status: RESOLVED DUPLICATE of bug 1436117

(Core :: Audio/Video: GMP, defect, P1)

Description

[zling.ustc]

Hi:
This crash happens easily. i am developing webrtc application on firefox. One peerconnection with multiple streams. if i subscribe/unsubscribe peer video frequently it happens. The subscription is done through inserting stream info and ssrc line to remote offer and un-subscription is done through removing the ssrc line and msid line and set direction to be 'recvonly'.
Any info needed please let me know.
Thx in advance.
Have a nice day.

BR
Rory



This bug was filed from the Socorro interface and is
report bp-62c177d8-efeb-4bf5-bfa5-4d6c00180413.
=============================================================

Top 10 frames of crashing thread:

0 vcruntime140.dll vcruntime140.dll@0xc3c7 
1 xul.dll mozilla::gmp::GMPPlaneImpl::Copy dom/media/gmp/GMPVideoPlaneImpl.cpp:160
2 xul.dll mozilla::gmp::GMPVideoi420FrameImpl::CreateFrame dom/media/gmp/GMPVideoi420FrameImpl.cpp:190
3 xul.dll mozilla::WebrtcGmpVideoEncoder::Encode_g media/webrtc/signaling/src/media-conduit/WebrtcGmpVideoCodec.cpp:400
4 xul.dll mozilla::runnable_args_memfn<mozilla::WebrtcGmpVideoEncoder*, int  media/mtransport/runnable_utils.h:174
5 xul.dll mozilla::SyncRunnable::Run xpcom/threads/SyncRunnable.h:112
6 xul.dll nsThread::ProcessNextEvent xpcom/threads/nsThread.cpp:1040
7 xul.dll NS_ProcessNextEvent xpcom/threads/nsThreadUtils.cpp:517
8 xul.dll mozilla::ipc::MessagePumpForNonMainThreads::Run ipc/glue/MessagePump.cpp:334
9 xul.dll MessageLoop::RunHandler ipc/chromium/src/base/message_loop.cc:319

=============================================================

Comment 1

[Paul Adenot (:padenot)]

memcpy with (a) bad pointer(s) in the content process, this crash is a bad read. Setting to private.

Nils, this is in GMP land, but can you double check the scenario the reporter is describing, to try to get any clue of what could be happening?

jesup, this has been happening for some time, not a lot of crashes, but we might have a way to repro, do you have a minute to have a look?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

few crashes; oldest build is a 58beta build.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

dump files appear not to be useful due to vcruntime :-(

Rory: is there a testcase you can give us access to?  Or detail what exactly you're doing in JS to do the "subscribe/unsubscribe" that triggers this?  Code snippets?  You can provide this privately if you wish to myself or Nils (drno).

Even better if you have access to Linux: could you try a Linux ASAN build, which adds memory checks that would trap exactly what happened, and why it's going past the end of of a bufffer (or using a freed buffer)?  If you can repro on Linux I can get you a link to an ASAN build.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Also, a log with MOZ_LOG=signaling:5,GMP:5,timestamp MOZ_LOG_FILE=some file would help a lot.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Most likely cause is a bug in the code landed from bug 1429390.  Dan?

Comment 6

[Dan Minor [:dminor]]

(In reply to Randell Jesup [:jesup] from comment #5)
> Most likely cause is a bug in the code landed from bug 1429390.  Dan?

That was my thought as well, but unless I'm missing something, when clicking through bp-62c177d8-efeb-4bf5-bfa5-4d6c00180413 it appears this crash is in a version which does not have the changes from bug 1429390:

https://hg.mozilla.org/releases/mozilla-release/annotate/239e434d6d2b/media/webrtc/signaling/src/media-conduit/WebrtcGmpVideoCodec.cpp#l319

Comment 7

[Nils Ohlmeier [:drno]]

(In reply to Randell Jesup [:jesup] from comment #5)
> Most likely cause is a bug in the code landed from bug 1429390.  Dan?

The earliest crash I see happened in 58.0b15 on 2018-01-15 here https://crash-stats.mozilla.com/report/index/aec548e9-e990-4b63-9266-ecf6c0180118

Bug 1429390 never got uplifted into 58.

Bugs which have touched WebrtcGmpVideoCodec.cpp and made into 58 are:
- bug 1417797
- bug 1408523
- bug 1411605
- bug 1415582

From the description this sounds to me more like bug 1394602. Not sure though why this would only start to show in 58. Maybe one of the above patches caused a new side effect when the problem in bug 1394602 kicks in for H264?!

Zling: I'm assuming that "One peerconnection with multiple streams." means you are bundling multiple video m-sections into the same PeerConnection, right?

Comment 8

[zling.ustc]

Yes. one peerconnection with multiple video-m sections. i will upload one firefox log later. one assumption is that it may related to my specific media constraints.
     var _videoConstraintsTwo = {
        width: {max:1280, ideal:1280},
        height: {max:720, ideal:720},
        frameRate: 30,
        aspectRatio: 16/9.0
    };

    var _videoConstraintsOne = {
        width: 160,
        height: 90,
        frameRate: 6,
        aspectRatio: 16/9.0
    };
i capture two streams to simulate simulcast(from firefox code seems h264 simulcast is not well supported).


///////
From the description this sounds to me more like bug 1394602. Not sure though why this would only start to show in 58. Maybe one of the above patches caused a new side effect when the problem in bug 1394602 kicks in for H264?!

Zling: I'm assuming that "One peerconnection with multiple streams." means you are bundling multiple video m-sections into the same PeerConnection, right?

Comment 9

[zling.ustc]

(In reply to Randell Jesup [:jesup] from comment #4)
> Also, a log with MOZ_LOG=signaling:5,GMP:5,timestamp MOZ_LOG_FILE=some file
> would help a lot.

https://www.moxtra.com/service4/#view/CAEqBVpuanZMehhDQjYxSmdDdW9Yb0gzemp3bURHTkhIWkaAAdvbA5ADFA
here is  the firefox log file you needed. Jesup.

For the code i am thinking about how to share with you.
BR
Rory

Comment 10

[zling.ustc]

(In reply to Randell Jesup [:jesup] from comment #3)
> dump files appear not to be useful due to vcruntime :-(
> 
> Rory: is there a testcase you can give us access to?  Or detail what exactly
> you're doing in JS to do the "subscribe/unsubscribe" that triggers this? 
> Code snippets?  You can provide this privately if you wish to myself or Nils
> (drno).
> 
> Even better if you have access to Linux: could you try a Linux ASAN build,
> which adds memory checks that would trap exactly what happened, and why it's
> going past the end of of a bufffer (or using a freed buffer)?  If you can
> repro on Linux I can get you a link to an ASAN build.

Test case is :
1)a(firefox client) turns on camera with two streams(two video m sections with one application section).
2)b turns on camera.
3)a subscribes b's video by setting remote description. use video m section one and insert ssrc and msid info.
The negotiate happens locally. Then i send command to notify server to send data for me(after negotiate successfully).
4)a unsubscribes b'video by setting remote des. remove video m section one's ssrc msid info and set direction to be recvonly.
setRemoteDescription => createAnswer => setLocalDescription.
5)a re-subscribes b' video basically the same time(not really same time but shortly, specific user case). the ssrc and msid value is same as step 3).
i will send you one mail([e-mail]) about our online product to reproduce it.
Thx for your quick response.
BR
Rory

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Hmmm, we reconfigured an encoder <1/2 second before the crash

Comment 12

[zling.ustc]

(In reply to Randell Jesup [:jesup] from comment #11)
> Hmmm, we reconfigured an encoder <1/2 second before the crash

under our case. First we will subscribe 720P video then we will subscribe 90P video. we will use REMB to set uplink bitrate.
SetTargetRates: 100000 bps 
ConfigureEncoder requested.
This can explain  the log.

Comment 13

[zling.ustc]

(In reply to Randell Jesup [:jesup] from comment #11)
> Hmmm, we reconfigured an encoder <1/2 second before the crash

after i made one change the issue is gone.

previously i send two streams(one 720p and one 90P). while no one subscribes my 720p i will disable that track(track.enabled = false). if i remove such logic which means i always send two streams. the issue is gone. from firefox log seems if i disable one track the encoder needs reconfiguration. Then it crashed.

wish this can help.

Comment 14

[zling.ustc]

(In reply to zling.ustc from comment #13)
> (In reply to Randell Jesup [:jesup] from comment #11)
> > Hmmm, we reconfigured an encoder <1/2 second before the crash
> 
> after i made one change the issue is gone.
> 
> previously i send two streams(one 720p and one 90P). while no one subscribes
> my 720p i will disable that track(track.enabled = false). if i remove such
> logic which means i always send two streams. the issue is gone. from firefox
> log seems if i disable one track the encoder needs reconfiguration. Then it
> crashed.
> 
> wish this can help.

Bad news the frequency is low but still crashed. From log the suspend video logic is removed(as i said i do not disable stream one now). but seems the rate control will trigger encoder reconfiguration too.
https://www.moxtra.com/service4/#view/CAEqBWZXNkJsehhDQkY5RnR0S0IzWFo0UW56OTJGRmdZWjCAATSQAxQ
here is the log.

Comment 15

[zling.ustc]

(In reply to Randell Jesup [:jesup] from comment #11)
> Hmmm, we reconfigured an encoder <1/2 second before the crash

Seems renegotiate will cause.
WebrtcVideoConduit::ConfigureSendMediaCodec

There are comments of such function like this.
/**
 * Note: Setting the send-codec on the Video Engine will restart the encoder,
 * sets up new SSRC and reset RTP_RTCP module with the new codec setting.
 *
 * Note: this is called from MainThread, and the codec settings are read on
 * videoframe delivery threads (i.e in SendVideoFrame().  With
 * renegotiation/reconfiguration, this now needs a lock!  Alternatively
 * changes could be queued until the next frame is delivered using an
 * Atomic pointer and swaps.
 */

This crash happens when i renegotiate to not unsubscribe one stream of peer participant. 
under our circumstances there are lots of negotiations. subscribe/unsubscribe peer videos.

Wish it can help.
BR
Rory

Comment 16

[Byron Campen [:bwc]]

I see very similar crashes as far back as 56, like this one:

https://crash-stats.mozilla.com/report/index/be739f03-89eb-4e52-93b6-58b5c0180419

Comment 17

[Byron Campen [:bwc]]

I wonder if this is the h264 manifestation of the crashes we see in copy_and_extend_plane... those were also being caused by invalid reads when trying to copy a not-yet-encoded frame.

Comment 18

[Byron Campen [:bwc]]

Could I get access to the code that reproduces this issue?

Comment 19

[zling.ustc]

(In reply to Byron Campen [:bwc] from comment #18)
> Could I get access to the code that reproduces this issue?

we bypass this crash with latest transceiver api. This crash should relates to we capture two video streams with different resolution and send them to peer side to simulate simulcasting h264 video.

After we use transceiver api and only capture one video stream, then call 'addTransceiver' twice with different 'scaleResolutionDownBy' the crash is gone.

In my opinion seems the capturing module before encoding misuses some parameters(resulting from two capturing streams with different resolution).

Wish it can help.

Comment 20

[zling.ustc]

(In reply to Byron Campen [:bwc] from comment #18)
> Could I get access to the code that reproduces this issue?

we bypass this crash with latest transceiver api. This crash should relates to we capture two video streams with different resolution and send them to peer side to simulate simulcasting h264 video.

After we use transceiver api and only capture one video stream, then call 'addTransceiver' twice with different 'scaleResolutionDownBy' the crash is gone.

In my opinion seems the capturing module before encoding misuses some parameters(resulting from two capturing streams with different resolution).

Wish it can help.


i will try to send you our demo code with previous version sdk code btw.

BR
Rory

Comment 21

[Andreas Pehrson [:pehrsons]]

Could this be another manifestation of the issue bug 1436117 fixed?

Comment 22

[Byron Campen [:bwc]]

I could definitely see that. I do not see any crashes on 60+ after bug 1436117 landed.

Rory: Does this happen on nightly or beta?

Comment 23

[Byron Campen [:bwc]]

I should also note that the copy_and_extend_plane crashes also stopped right after bug 1436117 landed.

Comment 24

[zling.ustc]

(In reply to Byron Campen [:bwc] from comment #22)
> I could definitely see that. I do not see any crashes on 60+ after bug
> 1436117 landed.
> 
> Rory: Does this happen on nightly or beta?

The crash should have been fixed in 60.0b16, i just have confirmed that. 
Thx all for the feedbacks.

BR
Rory

Comment 25

[Julien Cristau [:jcristau] (back April 22)]

Updating status to match bug 1436117

Comment 26

[Andrei Vaida [:avaida]]

Byron, is this something that would benefit from manual testing? If that's a yes, is there a test case that you could share or straightforward steps that we could take to reproduce the bug?

Comment 27

[Byron Campen [:bwc]]

We don't have a test-case that reproduces this.