Closed Bug 1608297 Opened 5 years ago Closed 5 years ago

No audio when using video element despite setting the sdp attribute for the video stream to a=recvonly in renegotiation offer.

Categories

(Core :: WebRTC: Signaling, defect)

Product:
Core
Component:
WebRTC: Signaling
Version:
74 Branch
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED INVALID

People

(Reporter: kharwell, Unassigned)

Details

Attachments

(1 file)

bug_data.tar.gz
9.16 MB, application/gzip
Details
Attached file bug_data.tar.gz —

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36

Steps to reproduce:

  1. Firefox sends an offer to a server containing one audio and one video stream
  2. Server replies with accepted audio and video streams (both have a=sendrecv) however server only sends audio packets, and no video packets
  3. No audio from Firefox despite it receiving the audio packets (Note, am using the "video" element for audio/video). From what I understand this is to be expected.
  4. Server then sends a renegotiation offer to Firefox with a similar SDP as in the original reply except now audio has a=sendrecv, and video specifies a=recvonly
  5. Firefox accepts the offer and successfully replies

Actual results:

Firefox plays no audio.

Expected results:

Expected audio to start after renegotiation.

From what I understand it might be correct behavior that audio is not played until video packets are received when using the video element. However, if one repeats steps 1 & 2 above except now the reply from the server to the initial offer (that came from Firefox) sets a=recvonly for the video stream audio can be heard in step 3 (no video packets are being sent from the server as stated by SDP). I would have expected similar behavior upon renegotiation?

I've attached a tar.gz file that contains Firefox debug logs from an occurrence (not sure they'll help). Also for examples the renegotiation_no_audio.txt contains the offer/answer trace from the server side for when no audio is heard. And offer_answer_with_audio.txt contains a trace from the server side when audio is heard. Note, the SIP protocol is used for negotiation, but the SDP are contained within. rtp.txt just shows that audio is being send/received from server while video is being received, but not sent.

Other versions tried: 71.0, 72.0.1, and 74.0a1

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → WebRTC: Signaling
Product: Firefox → Core

Maybe the AV sync code doesn't cope well with this case? Any ideas?

Flags: needinfo?(jib)
Flags: needinfo?(dminor)

Thank you for your report.

I had a look at the logs; I'm not seeing any errors and audio packets are being transmitted and received. In case it is the AV sync code, I've placed a build with sync disabled here [1], kharwell, would it be possible to test this build and let us know if it changes anything? I'm also curious if this is working as expected in Chrome? Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=e0f862a04f3f71fddf524ab8fb75b864ba95eadb

Flags: needinfo?(dminor) → needinfo?(kharwell)

Let's see if we can isolate this some. Are the results the same if you play the audio in a separate audio element?

Flags: needinfo?(jib)

@dminor

Chrome does not seem to have a problem with this scenario. I am able to hear audio with Chrome.

Also I believe I tested your patch, and unfortunately it didn't seem to fix the problem. I've never built Firefox from source though, so to make sure here are the basic steps I followed:

  1. Downloaded, and setup a build environment via $ python bootstrap.py
  2. Within the mozilla-unified directory I built Firefox using $ ./mach build
  3. Applied your patch
  4. Rebuilt Firefox using $ ./mach build
  5. Packaged things up using $ ./mach package
  6. Copied the tarball to the test machine, unpackaged and ran it
Flags: needinfo?(kharwell)

@jib No the results are not the same. If I use an audio element I am able to hear audio in this scenario. Using the audio element is a valid, current workaround that I am now using.

(In reply to kharwell from comment #5)

I've never built Firefox from source though, so to make sure here are the basic steps I followed:

Thanks for doing that! I could see how it wasn't obvious, but links from our try server also contain binaries you can conveniently download, for next time. FYI: Click on B, then look under "Job Details", locate and dowload target.tar.bz2.

If I use an audio element I am able to hear audio in this scenario.

This to me suggests this is not a problem with peer connection, but rather with HTMLVideoElement and its MediaStream handling. I have a vague recollection of discussing this with pehrsons that we may have a difference in behavior here compared to other browsers in regards to whether we wait for audio packets or not, when srcObject is a MediaStream. But I wasn't immediately able to find a bug on it.

Bryce or Paul, does this ring any bells?

Flags: needinfo?(padenot)
Flags: needinfo?(bvandyk)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #7)

..., but links from our try server also contain binaries you can conveniently download, for next time. FYI: Click on B, then look under "Job Details", ...

oh haha I wondered as much. Thanks for the info though as that's much easier. Also I downloaded the tarball from the link, and confirmed again that the patch does not fix the issue to be sure.

This to me suggests this is not a problem with peer connection, but rather with HTMLVideoElement and its MediaStream handling. I have a vague recollection of discussing this with pehrsons that we may have a difference in behavior here compared to other browsers in regards to whether we wait for audio packets or not, when srcObject is a MediaStream. But I wasn't immediately able to find a bug on it.

Yes to be clear audio packets are being sent to/from the browser, and seemingly received by the browser but no audio is heard (even the little audio icon is not displayed in the browser tab). The video element's srcObject is being set to a stream with two tracks: one audio, one video. However, initially no video packets are being sent from the server to the browser despite the server specifying the a=sendrecv attribute for video. At this point no audio is heard. In Chrome however I can hear audio. Also, as mentioned using the audio element instead outputs audio at this point from Firefox.

Maybe this is a block autoplay problem?

Flags: needinfo?(padenot)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #7)

...

Bryce or Paul, does this ring any bells?

I'm afraid not. Clearing NI as I understand others are looking at this. Ping me again if I can help.

Flags: needinfo?(bvandyk)

Byron, does Comment 7 above ring any bells for you?

Flags: needinfo?(docfaraday)

Sorry, no.

Flags: needinfo?(docfaraday)

I've tried to reproduce the situation using https://jsfiddle.net/jib1/L2egv9u1/ but everything works for me (I hear Edit: see audio bars even initially). if you are able to modify the fiddle to produce the symptom that would be great!

The sdp dumps from the "server" in the fiddle differ quite a bit from yours (since they're from Firefox not an asterisk server), but mids and directions match, and I'm not able to get into the state you explain in step 3 in comment 0. What's different?

Based on this I have to challenge the initial assertion that the difference in behavior in a video element vs audio element "is to be expected." - If you have any simpler reproduce steps to show this behavior that would be much appreciated, so we can figure out if that's a bug or not.

Since I cannot repro that state, I cannot speak to whether a renegotiation should get us out of that state. Happy for tips on how to repro this.

Flags: needinfo?(kharwell)

Uh, I fooled myself by only looking at the audio bars (from the spectrum analyzer in the fiddle which uses web audio).

The audio bars show audio is coming in, but I am in fact able to reproduce that the video element isn't audible, because it is not playing, in spite of <autoplay>. I have to click the ▶ play button to hear it.

But this matches Chrome, so I'm not sure if this is what you're reporting (although hitting ▶ in Chrome doesn't work, so that's worse than Firefox).

Workaround: Once I add await video.play() to the fiddle then it works in Firefox (but not in Chrome): https://jsfiddle.net/jib1/L2egv9u1/77/

This could be the problem. I was doing an autoplay, but not directly calling play. If I add a video.play() like you mentioned to my code then I can now hear audio.

Somewhat of a minor side issue, but one small thing I noticed when using the play command. Although I can now hear audio the little "audio" icon on the tab is missing. If I right-click and choose "mute tab" the icon appears with a slash in it, but audio can still be heard. So something might be out of sync there as well. Just thought I'd mention it.

Flags: needinfo?(kharwell)

I'm glad it now works.

Thank you very much for mentioning the icon problem, it's indeed of importance. Would there be a way for us to try to reproduce this (i.e. is it on a public facing website, or can you send a reasonably small test case, or any way we could figure it out)? Thanks again!

Flags: needinfo?(kharwell)

(In reply to Paul Adenot (:padenot) from comment #16)

Thank you very much for mentioning the icon problem, it's indeed of importance. Would there be a way for us to try to reproduce this

Yeah, you can actually use the fiddle code @jib created to replicate the original issue: https://jsfiddle.net/jib1/L2egv9u1/77/

Using that you should be able to hear audio, and then note there is no icon. Right click, and choose "mute tab". The icon appears (audio symbol with slash through it) overlaying the text, but audio can still be heard. Clicking the icon removes it completely (i.e. there is no enabled audio icon with no slash).

Flags: needinfo?(kharwell)

Super, I can repro.

I'm going to close this as INVALID because it was expected behaviour from Firefox, but open another one for what you just reported. Thanks for following up on this!

Status: UNCONFIRMED → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID

Created a new issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1610919

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: