970729 - Implement an audio stream verification based on the signal in the stream

Status: RESOLVED FIXED

(Core :: WebRTC, defect, P4)

Description

[Nils Ohlmeier [:drno]]

As the canPlayThrough event does not seem to be a reliable criteria to judge if an audio stream is actually delivering a working audio stream we need additional means to check this.
One way which has been suggested could be to utilize the WebAudio API (https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#AnalyserNode) to verify that the received audio stream actually has the 8kHz signal in it.

Comment 1

[Nils Ohlmeier [:drno]]

Attachment: add_audio_frequency_check_points.patch

Comment 2

[Nils Ohlmeier [:drno]]

Even with the audio players being visible I still get one of the remote stream analyzers reporting zero quite often (not always).
Taking out the analyzers for the gUM fake audio at the beginning also does not really help.
I'm wondering if the setup in terms of how many audioContext's is troublesome. Or if I need to properly terminate/delete (how?) the webAudio chains (especially the analyzers) after I'm done using them?

Comment 3

[Nils Ohlmeier [:drno]]

Attachment: plot.sh

A little script to extract audio analyzer results from the test log and feed them into gnuplot.

Comment 4

[Nils Ohlmeier [:drno]]

Attachment: plot.gp

GnuPlot input file for plotting the audio graphs.

Comment 5

[Nils Ohlmeier [:drno]]

I updated the frequency graph for easier understanding: http://people.mozilla.org/~nohlmeier/freq/plot.png

Comment 6

[Paul Adenot (:padenot)]

Attachment: DO NOT LAND diagnostic patch.

So, I'm not really sure where the bug is. I amended your test to plot the data
using a <canvas>, and while the test still fails, I can clearly see non-zero
bins. This smells like a race to me.

I'm thinking that maybe a more robust way of writing this (instead of the
setTimeout, that should ideally be banned from mochitests), is to wait until we
actually see data (using a ScriptProcessorNode, or an AnalyserNode, getting time
domain data), and then get frequency data and run the analysis.

Waiting 500ms would go orange sometime anyways, because the time to open an
AudioStream can be slow on some platform/some cases.

Comment 7

[Paul Adenot (:padenot)]

(did not mean to take this bug, the tool I used to upload the patch did it for me).

Comment 8

[Nils Ohlmeier [:drno]]

Thanks Paul for the real time visualization!
That helped to identify the problem: a few seconds into the call the AEC kills the outgoing 8kHz signal as it receives exactly an 8 kHz signal from the far end.

Proposed solution: make the frequency of the fake audio configurable so that both parties can use different frequencies.

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Cute, AEC working causng a test to break...

Comment 10

[Maire Reavy [:mreavy]]

I think we've handled this already in another bug  - Andreas?

Comment 11

[Andreas Pehrson [:pehrsons]]

I landed webaudio verification of audio flow a while back, but it's only used for one or two webaudio-over-peerconnection tests.

Not sure how we should proceed with this, but making audio frequency configurable and enabling this for a couple more tests sounds workable. drno?

I'll also add that audio input on Linux comes from a virtual hardware device (unless explicitly fake). Not sure how that could be dealt with.

Comment 12

[Nils Ohlmeier [:drno]]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #11)
> I landed webaudio verification of audio flow a while back, but it's only
> used for one or two webaudio-over-peerconnection tests.

Could you provide a pointer to that code, please? I'm curious what you are doing there already.

> Not sure how we should proceed with this, but making audio frequency
> configurable and enabling this for a couple more tests sounds workable. drno?

I think the configurable audio frequency would be really helpful for verifying that we actually receive two distinct audio streams in the multi stream test cases (right now we have nothing which verifies that we have two distinct contents being streamed).

> I'll also add that audio input on Linux comes from a virtual hardware device
> (unless explicitly fake). Not sure how that could be dealt with.

Yeah I'm aware of that. That is the result of another plan which only got half implemented. Currently it proves that our code can work also with other devices then the Firefox build in fake devices.
I assume that we would only enable such an audio content stream verification only for a small amount of tests to not overload our test beds. So we could simply skip if the OS fake devices are in use.

Comment 13

[Andreas Pehrson [:pehrsons]]

(In reply to Nils Ohlmeier [:drno] from comment #12)
> (In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #11)
> > I landed webaudio verification of audio flow a while back, but it's only
> > used for one or two webaudio-over-peerconnection tests.
> 
> Could you provide a pointer to that code, please? I'm curious what you are
> doing there already.

checkReceivingToneFrom is the most high level piece: http://hg.mozilla.org/mozilla-central/file/6cd7a42a18db/dom/media/tests/mochitest/pc.js#l1543

There is also an effort to test multiple tones existing in a stream by padenot for his audio capture work in bug 1156472.

Comment 14

[Bulk Bug Changes for mreavy's org]

Mass change P3->P4 to align with new Mozilla triage process.

Comment 15

[Nils Ohlmeier [:drno]]

I think we have now in mochitest ways to verify audio going through.