864371 - WebRTC hang during shutdown [@QTCaptureSession::addInput:error]

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[Christoph Diehl [:posidron]]

Attachment: callstack

+++ This bug was initially created as a clone of Bug #864255 +++

This hang is happening during shutdown by randomly closing the Firefox instance while fuzzing JSEP.

Tested with m-i changeset: 129433:a1a2af3cebff

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Thread 38 indicates we're stuck waiting for QTKit ([QTCaptureSession addInput:error:]).  Internally, I believe that sends native events to the main thread (we've run into this before) and can be blocked because the main thread stops handling native events.  (bsmedberg and I dealt with one of these when closing a session; this may be the inverse of that when opening one.)  For added fun, GDB got confused trying to dump all the thread stacks.

Blocking- for now, since we haven't seen it outside of fuzzing, but we might bump that depending on what we find.

Steve - Is this another function that needs to be proxied to the Main thread?  BSmedberg and I had previously proxied a release because in shutdown, the Main Thread stops processing native events before everything is shut down.  See bug 829907 and the comments in MediaEngineWebRTCVideo.cpp

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

-> blocking-webrtc? to keep it on our radar until we figure out more
-> needinfo to smichaud

Comment 3

[Steven Michaud [:smichaud] (Retired)]

> Steve - Is this another function that needs to be proxied to the Main thread?

I don't know.  And I won't really be able to find out until someone finds good STR for this bug, and I have a day or two to spend on it.

It might help if there are any interesting messages in the system console.  That turned out to be crucial for figuring out bug 837539.

Comment 4

[Maire Reavy [:mreavy]]

(In reply to Steven Michaud from comment #3)
> > Steve - Is this another function that needs to be proxied to the Main thread?
> 
> I don't know.  And I won't really be able to find out until someone finds
> good STR for this bug, and I have a day or two to spend on it.
> 
> It might help if there are any interesting messages in the system console. 
> That turned out to be crucial for figuring out bug 837539.

cdiehl, can you help with providing STR and any interesting messages in the system console?

Comment 5

[Christoph Diehl [:posidron]]

I have added him to the initial bug which contains log of the whole fuzzing session.

Comment 6

[Steven Michaud [:smichaud] (Retired)]

I take it that means you don't have STR for the bug, or a testcase (reduced or not) that can reliably reproduce it.

If so, that doesn't help very much.  If at all possible, please provide one or both of these.  If you'd prefer, you can do that in bug 864255.

I didn't see any console messages at all in the fuzzing session log ... but I'm not sure I looked in the right place(s).

Comment 7

[Christoph Diehl [:posidron]]

Sorry for the late response but I had trouble with my ISP.

> I didn't see any console messages at all in the fuzzing session log ... but
> I'm not sure I looked in the right place(s).

It's all the in the file: fuzzing-session

Comment 8

[Christoph Diehl [:posidron]]

Attachment: callstack

I have attached a non-minimized testcase which reproduces the hang.

Load the testcase in a debug build of Firefox and randomly close Firefox after a few seconds.

If you are successful the last line you should see on the console is:

WARNING: nsAppShell::Exit() called redundantly: file /Users/cdiehl/dev/repos/mozilla/mozilla-inbound/widget/cocoa/nsAppShell.mm, line 757

Comment 9

[Christoph Diehl [:posidron]]

Attachment: testcase

Comment 10

[Randell Jesup [:jesup] (needinfo me)]

Steven - does this testcase work for you?  It may take a number of attempts from what I read

Comment 11

[Steven Michaud [:smichaud] (Retired)]

> If you are successful the last line you should see on the console is:
>
> WARNING: nsAppShell::Exit() called redundantly: file
> /Users/cdiehl/dev/repos/mozilla/mozilla-inbound/widget/cocoa/nsAppShell.mm,
> line 757

I see this every time I quit today's mozilla-central-debug nightly, so
it doesn't have any relation to this bug.

I also can't reproduce anything resembling a hang, testing with either
today's mozilla-central-debug nightly or today's mozilla-central
nightly.  I tried five or six times with each, doing Cmd-q either
just after the testcase loaded or waiting 5, 10 or 15 seconds.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

> If you are successful the last line you should see on the console is:
>
> WARNING: nsAppShell::Exit() called redundantly: file
> /Users/cdiehl/dev/repos/mozilla/mozilla-inbound/widget/cocoa/nsAppShell.mm,
> line 757

And no, this was never the last line of output.

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

Cristoph: what OS and webcam (and driver rev) are in use for this test?  Also, just quickly verify this still happens for you.  Can you try it on any other HW/OS combos?

Comment 14

[Christoph Diehl [:posidron]]

ProductName:	Mac OS X
ProductVersion:	10.8.3
BuildVersion:	12D78

It happens with the default MacBook Pro 2012 edition and with the Apple Thunderbold display.

Comment 15

[Christoph Diehl [:posidron]]

Still reproducible with m-i changeset: 131012:e6e9cc3a5b10

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

I suspect this is long fixed, but would like some verification - can you try?

Comment 17

[Christoph Diehl [:posidron]]

Okay, I will look at this ASAP but I am also suspecting this is fixed after that time. :-)

Comment 18

[juan becerra [:juanb]]

I haven't been able to load the testcase in comment #9. I have been trying on Nightly and Dev Edition. I also searched crash-stats for this signature [@QTCaptureSession::addInput:error], and I found no results.

Christoph, can you take a look when you get a chance and see if this still happens for you?

Comment 19

[Christoph Diehl [:posidron]]

Yes, same here. This issue seems gone.