1651745 - portaluat.mvine.com - no audio

Status: RESOLVED FIXED

(Core :: WebRTC: Audio/Video, defect, P1)

Description

[Mike Taylor [:miketaylr]]

(this may be the wrong component, please move if so)

Originally reported @ https://github.com/webcompat/web-bugs/issues/55227

It sounds like a possible regression, somewhere between 68 and 78.

Firefox 68esr works fine when making audio calls via Websockets to our Janus/Asterisk server. Switching to the latest ESR (78) and only on Windows 10 (Linux and Mac are fine) there is no audio if Windows 10 initiates the call. Audio is OK on Windows 10 if any other browser (including Firefox 68 ESR) initiates the call.
Our server is Debian Buster, with the Janus and Asterisk packages installed. We have created a few entries in Asterisk's sip.conf file, and are using a slightly modified version of Janus' own siptest page.

There are logs linked from https://github.com/webcompat/web-bugs/issues/55227#issuecomment-656104159

Comment 1

[Mike Taylor [:miketaylr]]

It would be really nice if we could get a regression bisected using https://mozilla.github.io/mozregression/, but without credentials that's hard.

Comment 2

[raymond.field]

I'm happy to supply credentials, please let me know the best way to get them to you.

Comment 3

[Mike Taylor [:miketaylr]]

That would be great, can you email them to [e-mail] please?

Comment 4

[raymond.field]

Attachment: screenshots of 2 regression runs

We have run a couple of regression tests, and found the following - I hope this info is useful. Attached are a couple of screenshots from the runs.

test 1:

• tested mozilla-central build: 5f77291b - verdict BAD - the other PC wasn't able to hear me which deemed the test a fail

app_name: firefox
build_date: 2019-12-03 11:58:18.126000
build_file: C:\Users\uzov.mozilla\mozregression\persist\8e08311f2d87-shippable--mozilla-central--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cAo7l6BTT7mQgdVtnwmvmQ/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: 8e08311f2d873bd86fe624b60d01df1ba510e2aa
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8e08311f2d873bd86fe624b60d01df1ba510e2aa&tochange=5f77291b48ff276ee0d328834530f54a52a6ba03
repo_name: mozilla-central
repo_url: https://hg.mozilla.org/mozilla-central
task_id: cAo7l6BTT7mQgdVtnwmvmQ

• tested mozilla-central build: 8e08311f - verdict GOOD - audio was OK for both sides

app_name: firefox
build_date: 2019-12-03 11:58:18.126000
build_file: C:\Users\uzov.mozilla\mozregression\persist\8e08311f2d87-shippable--mozilla-central--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cAo7l6BTT7mQgdVtnwmvmQ/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: 8e08311f2d873bd86fe624b60d01df1ba510e2aa
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8e08311f2d873bd86fe624b60d01df1ba510e2aa&tochange=5f77291b48ff276ee0d328834530f54a52a6ba03
repo_name: mozilla-central
repo_url: https://hg.mozilla.org/mozilla-central
task_id: cAo7l6BTT7mQgdVtnwmvmQ

test 2:

Tested autoland build: 66954a0b (verdict:g):

app_name: firefox
build_date: 2019-12-19 00:11:22.565000
build_file: C:\Users\Admin.mozilla\mozregression\persist\66954a0b1144-shippable--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ig9fFnmBSU6cG7ioA95FgQ/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: 66954a0b1144c9b1aa56ea132df6ac4e346be08b
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=66954a0b1144c9b1aa56ea132df6ac4e346be08b&tochange=5e8b48c8cd93ae318b2963de1b3c1db0710c0242
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland
task_id: Ig9fFnmBSU6cG7ioA95FgQ

Tested autoland build: 8b96d124 (verdict:b): - [e-mail] called [e-mail] and no audio

app_name: firefox
build_date: 2019-12-19 01:37:50.615000
build_file: C:\Users\Admin.mozilla\mozregression\persist\8b96d1241d11-shippable--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/XaJksXm9RauUBHIlVN0h3A/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: 8b96d1241d111eb9d94446f89df812f81b6bd08d
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=66954a0b1144c9b1aa56ea132df6ac4e346be08b&tochange=8b96d1241d111eb9d94446f89df812f81b6bd08d
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland
task_id: XaJksXm9RauUBHIlVN0h3A

Tested autoland build: f748822a (verdict:g):

app_name: firefox
build_date: 2019-12-19 00:46:33.793000
build_file: C:\Users\Admin.mozilla\mozregression\persist\f748822a58d0-shippable--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Xr_aBakyTx-tHbdS14dErA/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: f748822a58d0583a13d63020203380dc3eb1fb82
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f748822a58d0583a13d63020203380dc3eb1fb82&tochange=8b96d1241d111eb9d94446f89df812f81b6bd08d
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland
task_id: Xr_aBakyTx-tHbdS14dErA

Tested autoland build: bbef5fbd (verdict:b): - [e-mail] called [e-mail] and no audio

app_name: firefox
build_date: 2019-12-19 01:39:28.687000
build_file: C:\Users\Admin.mozilla\mozregression\persist\bbef5fbdb566-shippable--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UC08NfeZTgW5bRLUtkFrBw/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: bbef5fbdb566bf0bae7ba9394a0aa154d13f8c2c
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f748822a58d0583a13d63020203380dc3eb1fb82&tochange=bbef5fbdb566bf0bae7ba9394a0aa154d13f8c2c
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland
task_id: UC08NfeZTgW5bRLUtkFrBw

Tested autoland build: 6ad8baec (verdict:g):
app_name: firefox
build_date: 2019-12-19 00:59:08.314000
build_file: C:\Users\Admin.mozilla\mozregression\persist\6ad8baec4ca9-shippable--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RvRYZShtTUCAx0s_nalhPg/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: 6ad8baec4ca949e4ede0e79a0a2219cdc10f8bfd
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=6ad8baec4ca949e4ede0e79a0a2219cdc10f8bfd&tochange=bbef5fbdb566bf0bae7ba9394a0aa154d13f8c2c
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland

task_id: RvRYZShtTUCAx0s_nalhPg

Comment 5

[raymond.field]

Attachment: cflmemhjakfbmlib.png

Comment 6

[raymond.field]

Attachment: hjamcfohdldjgmco.jpg

Comment 7

[Mike Taylor [:miketaylr]]

Oh, that's super helpful, thanks Raymond.

Looking at https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=6ad8baec4ca949e4ede0e79a0a2219cdc10f8bfd&tochange=bbef5fbdb566bf0bae7ba9394a0aa154d13f8c2c, there's 2 possible regressing bugs:

Bug 1542321
Bug 1586370

Nils, I see that Andreas is on leave until August 1 -- is there someone who can take a look at this possible regression? I have some test credentials that Raymond emailed to me if it's useful.

Comment 8

[Nils Ohlmeier [:drno]]

Paul since you are probably the person with the second best knowledge of Andreas patches could you please have a look what is going wrong here?

Comment 9

[Paul Adenot (:padenot)]

Mike, would you mind sending those credentials to padenot@m.c ?

Comment 10

[Mike Taylor [:miketaylr]]

Done.

Comment 11

[raymond.field]

Hi,

Are there any updates on this issue that you can tell me about please?

Kind regards,

Raymond Field

Comment 12

[Paul Adenot (:padenot)]

Andreas is back, now.

Comment 13

[Andreas Pehrson [:pehrsons]]

Raymond, on a local build (84) on Windows 10 I called a regular Nightly (also 84) on Ubuntu. That worked fine, and I hear audio in both directions.

I tried calling from ESR78 on Win10 to Nightly on Ubuntu. Still works.

I tried calling from Nightly on Ubuntu to ESR78 on Win10. That hosed audio in ESR78. Audio device enumeration seemed busted too, because I didn't even get a permission prompt in another tab.

Calling from Nightly on Ubuntu to a local 84 build on Win10 worked though, so unless this is intermittent we did something to fix it.

Raymond, does this sound like what you would expect?

Comment 14

[Andreas Pehrson [:pehrsons]]

Hmm, the local build is consistently working but launching today's Nightly through mozregression isn't. Timing and debug vs non-debug I suppose. I'll dig in.

Comment 15

[Andreas Pehrson [:pehrsons]]

An opt build from mozregression + logging works. It looks like we for some reason switch to a new audio driver with very short intervals. Unclear who is doing this and why. There are a couple of callsites that cannot be ruled out.

I still haven't gotten a local build to reproduce, even in opt non-debug. Makes it a bit harder to incrementally add logging and retrying...

Comment 16

[Andreas Pehrson [:pehrsons]]

When it ends up silent the graph is flip-flopping between wanting 1 and 2 output channels, so it keeps recreating the audio driver accordingly here.

When everything works there is no flip-flopping in the output channels.

Whether bug 1586370 is in play here I'm not sure. It seems likely that the root cause lies elsewhere and that bug exacerbates the problem by running on a system clock driver all the time (it made us run on a system clock driver while waiting for the audio driver to have started, and we're starting new audio drivers constantly -- I counted for a random 1-second sample in the log 12 new audio drivers).

Comment 17

[Andreas Pehrson [:pehrsons]]

I understand this now. Here's what happens:

0. Precondition: Stereo mic, and another audio output that is mono (from a peerconnection). The important thing is that the mic must have a higher channel count than the other audio track.
1. The other audio output starts first. We open an output-only audio driver in mono.
2. The mic opens through getUserMedia. We open a full-duplex audio driver in mono.
3. An HTMLMediaElement plays a MediaStream with the microphone audio track, probably muted (or we'd hear feedback).
4. The graph detects the presence of stereo audio on an output (though muted). We open a full-duplex audio driver in stereo.
5. While the new driver is starting (takes a little while) we run the fallback driver (new behavior in bug 1586370). It appends silence to the mic track.
6. The graph detects no presence of stereo audio on an output (there is just silence, with no channel count), resorts to mono from the other output (it takes the max). We open a full-duplex audio driver in mono.
7. Go to step 4.

So yes, bug 1586370 is the regressor, though its not the root source of the flip-flopping channel count.

Workaround: For the self-view, let the HTMLVideoElement play a MediaStream containing the camera track only. Raymond, you might want to try this out until I have this fixed.

Comment 18

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Memoize the max channel count in AudioSegment::MaxChannelCount(). r?padenot!

When an AudioCallbackDriver starts a fallback SystemClockDriver is running in
its stead. The AudioTrack getting fed data from the input stream of the driver
will get real data while the AudioCallbackDriver is running, and silence while
the fallback driver is running.

If the MediaStreamTrack representing the microphone source is part of a
MediaStream being played by an HTMLMediaElement; and another MediaStreamTrack
representing another source with a lower channel count than the microphone is
part of another MediaStream being played by another HTMLMediaElement; then:

1. We start the graph with the other source's output channel count, and a
   fallback driver.
2. Once the audio driver has started, it adds data at a higher channel count
   than the graph's to its MediaTrack. The driver switches audio driver to
   match the new channel count.
3. The new driver starts with a fallback driver, which adds silence to the
   track. Silence has no channel count, so the graph sees only the channel
   count of the other source and switches audio driver to match this.
4. Go to 1.

This patch fixes makes us memoize a previously returned max channel count for an
AudioSegment for use when there is only null data (e.g., silence) present in the
segment. This applies to step 3 above, where no audio driver would be switched
because the graph still sees the mic's channel count.

Comment 19

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Make channel count explicit for every audio MediaTrack. r?padenot!

Comment 20

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Change MediaEngineSource::SetTrack to take a MediaTrack base class. r?padenot!

This allows us to pass in different sub classes, which the next patch will do.

Comment 21

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Use a dedicate ProcessedMediaTrack subclass for feeding microphone capture. r?padenot!

This was mainly driven by the need of querying this track for its channel count,
but it also moves us one usage away from SourceMediaTrack, which is a
longer-term goal (because of SourceMediaTrack::mMutex).

Comment 22

[Andreas Pehrson [:pehrsons]]

https://treeherder.mozilla.org/jobs?repo=try&revision=5bc41de515d56ec8d89f46f67416d3285cdf675b

Comment 23

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Add methods to inspect buffered amount and to clear AudioPacketizer. r?padenot!

This patch also constifies members where applicable, and adds a unittest for the
new methods.

Comment 24

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Simplify the same-iteration assertions in AudioInputProcessing. r?padenot

With a dedicated MediaTrack subclass for microphone input we can now coordinate
appending real data and pulling silence in a single place. This makes it easier
to control the amount of buffering needed, and the timing expectations around
pulling silence.

All in all, we can remove most of the state for the assertions, and the complex
logic surrounding them.

Comment 25

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Use the right thread for AudioInputProcessing::SetPassThrough and constify some ControlMessages. r?padenot!

Comment 26

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Signal AudioInputProcessing on input stream stop instead of driver start. r?padenot!

Without this patch, AudioInputProcessing wouldn't be aware of an audio driver
changing to another driver, be it a system driver or an audio driver that starts
on its fallback.

It could fail an assert since neither of those new drivers would append any
input data, so AudioInputProcessing would run out of buffered data but not know
to reset the state doing the bookkeeping for this.

Comment 27

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Don't buffer more than necessary in AudioInputProcessing. r?padenot!

When opening a second input track, there will already be some data from its
first instantiation in the driver's scratch buffer. If we ignore this data, we
end up buffering too much in AudioInputProcessing::Pull and tripping an assert.

Comment 28

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1651745 - Update logging for MediaEngineWebRTCMicrophoneSource et al. r?padenot!

This patch adds identifiers to existing log messages in the mic source and
AudioInputProcessing, and adds new log messages for complete tracing of frames.

Comment 29

[raymond.field]

Thank you for all of the updates, we haven't had a chance to test your suggestion about the HTMLMediaElement yet, but will update as soon as we have.

Will this patch be made available in the current ESR version?

Comment 30

[Andreas Pehrson [:pehrsons]]

(In reply to raymond.field from comment #29)

Thank you for all of the updates, we haven't had a chance to test your suggestion about the HTMLMediaElement yet, but will update as soon as we have.

Will this patch be made available in the current ESR version?

I think it's reasonable to uplift the first patch in the queue. It does enough to avoid the problem. The rest are fixes of the root cause, optimizations and cleanup.

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8e2da562e9fe
Memoize the max channel count in AudioSegment::MaxChannelCount(). r=padenot
https://hg.mozilla.org/integration/autoland/rev/2799805eef3b
Make channel count explicit for every audio MediaTrack. r=padenot
https://hg.mozilla.org/integration/autoland/rev/eb376fd954e1
Change MediaEngineSource::SetTrack to take a MediaTrack base class. r=padenot
https://hg.mozilla.org/integration/autoland/rev/605d00735b78
Use a dedicate ProcessedMediaTrack subclass for feeding microphone capture. r=padenot
https://hg.mozilla.org/integration/autoland/rev/c504995adbb4
Add methods to inspect buffered amount and to clear AudioPacketizer. r=padenot
https://hg.mozilla.org/integration/autoland/rev/bd01b1a3f7d7
Simplify the same-iteration assertions in AudioInputProcessing. r=padenot
https://hg.mozilla.org/integration/autoland/rev/a76d9158cdfd
Use the right thread for AudioInputProcessing::SetPassThrough and constify some ControlMessages. r=padenot
https://hg.mozilla.org/integration/autoland/rev/a00ad7266f54
Signal AudioInputProcessing on input stream stop instead of driver start. r=padenot
https://hg.mozilla.org/integration/autoland/rev/0ae30eae800d
Don't buffer more than necessary in AudioInputProcessing. r=padenot
https://hg.mozilla.org/integration/autoland/rev/abe51ad662bc
Update logging for MediaEngineWebRTCMicrophoneSource et al. r=padenot

Comment 32

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/8e2da562e9fe
https://hg.mozilla.org/mozilla-central/rev/2799805eef3b
https://hg.mozilla.org/mozilla-central/rev/eb376fd954e1
https://hg.mozilla.org/mozilla-central/rev/605d00735b78
https://hg.mozilla.org/mozilla-central/rev/c504995adbb4
https://hg.mozilla.org/mozilla-central/rev/bd01b1a3f7d7
https://hg.mozilla.org/mozilla-central/rev/a76d9158cdfd
https://hg.mozilla.org/mozilla-central/rev/a00ad7266f54
https://hg.mozilla.org/mozilla-central/rev/0ae30eae800d
https://hg.mozilla.org/mozilla-central/rev/abe51ad662bc

Comment 33

[Andreas Pehrson [:pehrsons]]

Comment on attachment 9185833 [details]
Bug 1651745 - Memoize the max channel count in AudioSegment::MaxChannelCount(). r?padenot!

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: See user impact. Sites can work around this, but users cannot.
• User impact if declined: Users might get silent audio input and output when doing microphone capture. The output picks up again if the microphone capture stops.
• Fix Landed on Version: 84
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The patch's simplicity. No changes to threading or timing.
• String or UUID changes made by this patch:

Comment 34

[Julien Cristau [:jcristau]]

Comment on attachment 9185833 [details]
Bug 1651745 - Memoize the max channel count in AudioSegment::MaxChannelCount(). r?padenot!

approved for 78.6esr

Comment 35

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr78/rev/0c00345f3345