801392 - crash in webrtc::ProcessThreadImpl::Process on WebRTC demo

Status: RESOLVED DUPLICATE of bug 801221

(Core :: WebRTC, defect, P1)

Description

[Pavel Franc - Mozilla.cz]

If I click the Start button in all but getUserMedia demo on http://people.mozilla.com/~anarayanan/webrtc/, Nighly crashes.

https://crash-stats.mozilla.com/report/index/bp-918f470b-af64-4f15-8928-d4b2b2121014

Comment 1

[Pavel Franc - Mozilla.cz]

And another crash with different signature 
https://crash-stats.mozilla.com/report/index/bp-b5b33b7d-ba54-4c40-8624-47b282121014

Both of these are comming from the Simple PeerConnection Video Test with the Debuger on. The crashes appear after registering all callbacks. Switching the Debugger off results in empty signature report

bp-253433df-8af1-4a25-8720-e319a2121014
bp-d4befe81-4034-4120-badd-51e9a2121014
bp-843807d8-0e88-44b8-901d-a82612121014

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Pavel: can you tell me more about the test environment?  Which Debugger are you referring to?

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Just tried Windows (Win7) nightly and Mac Nightly, both work for me.  Also tried a linux build off a fresh pull of inbound (virtually the same as nightly), no crashes.  So something else (config, machine environment, etc) must be involved.

Do you have a camera attached?  If so, how (usb) and what model/type and driver version?

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

The second signature:
https://crash-stats.mozilla.com/report/index/bp-b5b33b7d-ba54-4c40-8624-47b282121014
may point to a security issue given crashing on memmove in dtls/ssl with an address of 	0x19c00000.  Marking as a security issue and cc-ing relevant people.  Still unclear why it's consistently crashing for the reporter and not for me.

Comment 5

[Christoph Diehl [:posidron]]

This is a dup of https://bugzilla.mozilla.org/show_bug.cgi?id=800969. Run the testcase inside GDB and you get the same callstack.

The testcase will produce a SIGILL exception and you get each time a different callstack for Thread 0.

Comment 6

[Pavel Franc - Mozilla.cz]

Debugger - the JS debuger included in Firefox.
Camera - internal Chicony USB 2.0 (default MS driver)

The video in getUserMedia demo works, and Nighly is crashing on PeerConnection demo even the camera is disabled in the Windows Device Manager 

I tryed new windows profile (and hence new nighly profile), disabled all non default windows services (avast antivirus, apache, ...) and still reproducing the crash.

And I generated another pile of empty signature report plus one non empty

https://crash-stats.mozilla.com/report/index/bp-8fc0166b-d68d-4c24-ab77-83c5f2121014

Comment 7

[Eric Rescorla (:ekr)]

(In reply to Pavel Franc - Mozilla.cz from comment #1)
> And another crash with different signature 
> https://crash-stats.mozilla.com/report/index/bp-b5b33b7d-ba54-4c40-8624-
> 47b282121014
> 
> Both of these are comming from the Simple PeerConnection Video Test with the
> Debuger on. The crashes appear after registering all callbacks. Switching
> the Debugger off results in empty signature report
> 
> bp-253433df-8af1-4a25-8720-e319a2121014
> bp-d4befe81-4034-4120-badd-51e9a2121014
> bp-843807d8-0e88-44b8-901d-a82612121014

This particular trace looks the same as bug 801221 and now we have an indication of where we are doing things in two threads, since this code was not called in the STS thread. I'll supply a patch to start DTLS on the STS thread, which may fix this stack trace but I doubt all of them.

Comment 8

[Eric Rescorla (:ekr)]

I have provided a patch for the threading issue on bug 801221.

Jesup: we should probably land that patch first and see if it eliminates this call stack.

Comment 9

[Eric Rescorla (:ekr)]

Correction: the call stack in comment 4

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Please dup this and open a new bug with the call stack from comment 4, and mark it as depending on bug 801221, then restest once that bug lands.

Comment 11

[Daniel Veditz [:dveditz]]

Rather than do all that, since bug 801221 is now fixed we can have Pavel retest in this bug first. If there's still a problem, though, we may need to clone comment 4 into a new bug just for the sake of clarity.

Comment 12

[Al Billings [:abillings - ex-MoCo]]

The test case URLs here no longer work. Do we have a working repro case that I can test against?

Comment 13

[Daniel Veditz [:dveditz]]

Anant left Mozilla and they zapped his people.m.o storage. since we think this was a dupe of a fixed bug, and the stuff from comment 4 on was a different now-fixed bug it's probably fixed, but otherwise we're going nowhere here.

Comment 14

[Randell Jesup [:jesup] (needinfo me)]

pc_test.html is now hosted on github:
http://mozilla.github.com/webrtc-landing/pc_test.html

That should be basically the same as the original except updated for API changes

Comment 15

[Randell Jesup [:jesup] (needinfo me)]

Since the test still exists (in a different place), we may want to reopen this and retest as indicated in comment 11

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Pavel, can you please re-test? See comment 14.

Comment 17

[Pavel Franc - Mozilla.cz]

So I tried the demo again and I still see Firefox crashing :(
However this time it is niether the webrtc::ProcessThreadImpl::Process() nor ssl_SendSavedWriteData signature.

https://crash-stats.mozilla.com/report/index/bp-29bfd0c8-f902-4d94-9699-abec02121130
https://crash-stats.mozilla.com/report/index/bp-eb909803-bbe9-41ef-af45-52fc22121130
https://crash-stats.mozilla.com/report/index/bp-27b4c050-e115-4549-94a3-e6a452121130

Comment 18

[Randell Jesup [:jesup] (needinfo me)]

I'm having trouble reproducing this (linux or windows).

Any trick other than navigate to the test and hit start?  I tried lots of other bits (reload, navigate away, back, etc.

We have other reports with similar signatures.

Comment 19

[Randell Jesup [:jesup] (needinfo me)]

I do have a possible cause: MediaOperationRunnables have a reference to a DOMMediaStream, which is a MainThread-only object  for ref changes.

In order to handle this, after the runnable is done, it reposts itself to the main thread to die (and drop the ref to the stream).

The problem with this is illustrated by this modification to the source I made:

    if (mType != MEDIA_RELEASE) {
      // nsDOMMediaStreams aren't thread-safe... sigh.
      mType = MEDIA_RELEASE;
      NS_DispatchToMainThread(this);
      sleep(5);
    }
    return NS_OK;
  }

That's a guaranteed crash.

So, back to the drawing board. This may well be the issue with the MediaOperationRunnable bugs; and it's timing-related.

Note this is really a separate bug from the original report here; before landing I'll separate or re-purpose one of the bugs florian or others filed on this issue.  I believe the original cause of this bug is resolved.

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

New issue is bug 817841, so I think we can close this one as a dup