1543496 - Intermittent dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html | maxDifference: 0.9999571442604065, first bad index: 1792 with test-data offset 0 and expected-data offset 0; corresponding values 0 and -0.6874497532844543 --- differences

Status: RESOLVED WORKSFORME

(Core :: Web Audio, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: ccoroiu [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=239425792&repo=autoland

https://queue.taskcluster.net/v1/task/ccKe0dE5Q7KmQSr77TattA/runs/1/artifacts/public/logs/live_backing.log

19:23:46 INFO - TEST-START | dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html
19:23:46 INFO - GECKO(3472) | NOW: enabled
19:23:46 INFO - TEST-INFO | started process screenshot
19:23:46 INFO - TEST-INFO | screenshot: exit 0
19:23:46 INFO - Buffered messages logged at 19:23:46
19:23:46 INFO - TEST-PASS | dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html | Correct number of channels for expected buffer 0
19:23:46 INFO - TEST-PASS | dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html | Correct number of expected frames
19:23:46 INFO - Buffered messages finished
19:23:46 INFO - TEST-UNEXPECTED-FAIL | dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html | maxDifference: 0.9999571442604065, first bad index: 1792 with test-data offset 0 and expected-data offset 0; corresponding values 0 and -0.6874497532844543 --- differences - got 255, expected +0
19:23:46 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:320:16
19:23:46 INFO - compareChannels@dom/media/webaudio/test/webaudio.js:78:3
19:23:46 INFO - compareBuffers@dom/media/webaudio/test/webaudio.js:103:5
19:23:46 INFO - testOutput/sp.onaudioprocess@dom/media/webaudio/test/webaudio.js:214:11
19:23:46 INFO - EventHandlerNonNull*testOutput@dom/media/webaudio/test/webaudio.js:211:9
19:23:46 INFO - runTestOnContext@dom/media/webaudio/test/webaudio.js:202:9
19:23:46 INFO - testOnNormalContext@dom/media/webaudio/test/webaudio.js:222:7
19:23:46 INFO - runTestFunction@dom/media/webaudio/test/webaudio.js:253:5
19:23:46 INFO - rval@SimpleTest/SimpleTest.js:146:30
19:23:46 INFO - EventHandlerNonNull*this.addLoadEvent@SimpleTest/SimpleTest.js:171:13
19:23:46 INFO - @SimpleTest/SimpleTest.js:1453:5
19:23:46 INFO - Not taking screenshot here: see the one that was previously logged
19:23:46 INFO - TEST-UNEXPECTED-FAIL | dom/media/webaudio/test/test_mediaStreamTrackAudioSourceNode.html | maxDifference: 0.9999571442604065, first bad index: 1792 with test-data offset 0 and expected-data offset 0; corresponding values 0 and 0.6874497532844543 --- differences - got 255, expected +0
19:23:46 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:320:16
19:23:46 INFO - compareChannels@dom/media/webaudio/test/webaudio.js:78:3
19:23:46 INFO - compareBuffers@dom/media/webaudio/test/webaudio.js:103:5
19:23:46 INFO - testOutput/sp.onaudioprocess@dom/media/webaudio/test/webaudio.js:214:11
19:23:46 INFO - EventHandlerNonNull*testOutput@dom/media/webaudio/test/webaudio.js:211:9
19:23:46 INFO - runTestOnContext@dom/media/webaudio/test/webaudio.js:202:9
19:23:46 INFO - testOnNormalContext@dom/media/webaudio/test/webaudio.js:222:7
19:23:46 INFO - runTestFunction@dom/media/webaudio/test/webaudio.js:253:5
19:23:46 INFO - rval@SimpleTest/SimpleTest.js:146:30
19:23:46 INFO - EventHandlerNonNull*this.addLoadEvent@SimpleTest/SimpleTest.js:171:13
19:23:46 INFO - @SimpleTest/SimpleTest.js:1453:5

Comment 1

[Dorel Luca [:dluca]]

This likely started when bug 1324548 landed here:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=2bb8fcac7edfed5ffad76f0b2df33682b830f48b

https://hg.mozilla.org/integration/autoland/rev/397a9577d91b77a4d2bfb46f09375345fb022df1

Paul: Could you please take a look at this new failure?

Comment 2

[Intermittent Failures Robot]

15 failures in 509 pushes (0.029 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 7
• mozilla-central: 6
• mozilla-inbound: 2

Platform breakdown:

• windows7-32-shippable: 1
• osx-10-10-shippable: 10
• android-hw-p2-8-0-arm7-api-16: 1
• linux64-shippable: 2
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-10&endday=2019-04-10&tree=all

Comment 3

[Intermittent Failures Robot]

47 failures in 607 pushes (0.077 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• mozilla-central: 6
• mozilla-inbound: 9
• try: 1

Platform breakdown:

• osx-10-10-shippable: 35
• windows10-aarch64: 1
• windows7-32-shippable: 7
• android-hw-p2-8-0-arm7-api-16: 3
• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-11&endday=2019-04-11&tree=all

Comment 4

[Intermittent Failures Robot]

32 failures in 529 pushes (0.06 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 7
• autoland: 14
• mozilla-central: 9
• try: 2

Platform breakdown:

• osx-10-10-shippable: 23
• windows7-32-shippable: 6
• android-hw-p2-8-0-arm7-api-16: 1
• windows10-aarch64: 1
• macosx64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-12&endday=2019-04-12&tree=all

Comment 5

[Andrei Ciure[:aciure]]

Retriggered:

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=android%2C8.0%2Cpixel2%2Cpgo%2Cmochitests%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fpgo-mochitest-media-2%2Cm%28mda2%29&tochange=2e24668fe392b448a6a9f001bf9195cf29726498&fromchange=59c870edc6771c3de626e692bd6db1501a58e0e9&selectedJob=240195891

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=os%2Cx%2C10.10%2Cshippable%2Copt%2Cmochitests%2Cwith%2Ce10s%2Ctest-macosx64-shippable%2Fopt-mochitest-media-e10s%2Cm-e10s%28mda%29&tochange=2e24668fe392b448a6a9f001bf9195cf29726498&fromchange=59c870edc6771c3de626e692bd6db1501a58e0e9&selectedJob=240181870&group_state=expanded

Based on the above, Bug 1324548 seems to be the root for this.
https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=239455141&searchStr=android%2C8.0%2Cpixel2%2Cpgo%2Cmochitests%2Ctest-android-hw-p2-8-0-arm7-api-16%2Fpgo-mochitest-media-2%2Cm%28mda2%29&revision=2bb8fcac7edfed5ffad76f0b2df33682b830f48b

Comment 6

[Intermittent Failures Robot]

19 failures in 141 pushes (0.135 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 7
• mozilla-inbound: 5
• try: 3
• mozilla-central: 4

Platform breakdown:

• osx-10-10-shippable: 9
• windows7-32-shippable: 6
• macosx64-devedition: 3
• android-hw-p2-8-0-arm7-api-16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-13&endday=2019-04-13&tree=all

Comment 7

[Intermittent Failures Robot]

126 failures in 3025 pushes (0.042 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 66
• mozilla-central: 27
• mozilla-inbound: 25
• try: 8

Platform breakdown:

• windows7-32-shippable: 22
• osx-10-10-shippable: 87
• android-hw-p2-8-0-arm7-api-16: 6
• linux64-shippable: 2
• windows10-64-shippable: 2
• windows10-aarch64: 2
• macosx64-devedition: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-08&endday=2019-04-14&tree=all

Comment 8

[Intermittent Failures Robot]

137 failures in 417 pushes (0.329 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 116
• mozilla-inbound: 6
• mozilla-central: 10
• try: 5

Platform breakdown:

• osx-10-10-shippable: 47
• macosx64-devedition: 2
• windows7-32-shippable: 37
• android-hw-p2-8-0-arm7-api-16: 4
• linux32-shippable: 14
• linux64-shippable: 14
• windows10-64-shippable-qr: 10
• windows10-64-shippable: 6
• linux64: 1
• windows7-32: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-15&endday=2019-04-15&tree=all

Comment 9

[Paul Adenot (:padenot)]

Is there still a way to buffer some data internally in a track or somewhere else? I think this is a phase shift caused by buffering something somewhere.

Comment 10

[Andreas Pehrson [:pehrsons]]

There are some buffers.

In the MSG we have the track buffers in StreamTracks, i.e., MediaStream::mTracks, [1]. In a SourceMediaStream this buffer could outgrow or underrun an iteration. A TrackUnionStream that copies from an input track will insert silence if the input track underran (i.e., is blocked). For a microphone source there are some conditions early on after starting where underrunning (i.e., pushing silence instead of real samples) is OK, [2]. If we're using a fake microphone (MediaEngineDefaultAudio) we should just be pulled instead, and no underruns can occur here. Any other streams than SourceMediaStream can't underrun either, unless they're not doing ProcessInput() properly.

This test doesn't seem to be using getUserMedia, so the StreamTracks buffers should just follow the iterations. No guarantees for this are in place really (we must still support SourceMediaStream's pushing), so possibly there's a bug somewhere.

Other buffers I can think of would be in AudioInputProcessing::PacketizeAndProcess so that would again not be used in this test.

I can't speak to any other buffers in WebAudio, where I suppose something could also be happening.

[1] https://searchfox.org/mozilla-central/rev/1b2636e8517aa48422ed516affe4d28cb7fa220a/dom/media/MediaStreamGraph.h#545
[2] https://searchfox.org/mozilla-central/rev/1b2636e8517aa48422ed516affe4d28cb7fa220a/dom/media/webrtc/MediaEngineWebRTCAudio.cpp#821-822

Comment 11

[Andreea Pavel [:apavel]]

The frequency here is very high and this is occuring on the beta sims:

https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunnable&revision=459f1df70cd2f9894cf477f95da6c43c53450a7a&searchStr=mda3&selectedJob=240650525

There are 398 total failures in the last 7 days on

pgo: android-hw-p2-8-0-arm7-api-16
opt: linux32-shippable, linux64-shippable, osx-10-10-shippable, windows10-64-shippable, windows7-32, windows7-32-shippable

Paul, can we disable this?

Comment 12

[Paul Adenot (:padenot)]

Yes I don't understand what's going on here.

Comment 13

[Andreea Pavel [:apavel]]

Joel is there a special condition to disable just shippables?

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

:apavel, can you disable on !debug?

Comment 15

[Andreea Pavel [:apavel]]

Attachment: Bug 1543496 - disabled test_mediaStreamTrackAudioSourceNode.html on everything except debug

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/637a1287318c
disabled test_mediaStreamTrackAudioSourceNode.html on everything except debug r=jmaher

Comment 17

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/637a1287318c

Comment 18

[Intermittent Failures Robot]

263 failures in 607 pushes (0.433 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 189
• mozilla-central: 33
• mozilla-inbound: 20
• try: 21

Platform breakdown:

• linux64-shippable: 26
• osx-10-10-shippable: 62
• windows7-32-shippable: 88
• windows10-64-shippable-qr: 26
• windows10-64-shippable: 11
• linux64: 1
• linux32-shippable: 19
• windows7-32: 9
• windows10-64-qr: 1
• windows10-aarch64: 5
• windows10-64: 2
• macosx64-devedition: 2
• linux64-devedition: 1
• windows10-64-devedition: 1
• linux32: 1
• windows7-32-devedition: 3
• android-hw-p2-8-0-arm7-api-16: 4
• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-16&endday=2019-04-16&tree=all

Comment 19

[Intermittent Failures Robot]

402 failures in 2673 pushes (0.15 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 305
• mozilla-inbound: 27
• mozilla-central: 43
• try: 27

Platform breakdown:

• osx-10-10-shippable: 109
• macosx64-devedition: 4
• windows7-32-shippable: 125
• android-hw-p2-8-0-arm7-api-16: 8
• linux32-shippable: 33
• linux64-shippable: 40
• windows10-64-shippable-qr: 36
• windows10-64-shippable: 17
• linux64: 2
• windows7-32: 13
• windows10-64-qr: 1
• windows10-aarch64: 5
• windows10-64: 2
• linux64-devedition: 1
• windows10-64-devedition: 1
• linux32: 1
• windows7-32-devedition: 3
• android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-04-15&endday=2019-04-21&tree=all

Comment 20

[Intermittent Failures Robot]

1 failures in 3801 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-10-07&endday=2019-10-13&tree=all

Comment 21

[Intermittent Failures Robot]

1 failures in 5144 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2019-12-02&endday=2019-12-08&tree=all

Comment 22

[Intermittent Failures Robot]

1 failures in 3364 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2020-02-03&endday=2020-02-09&tree=all

Comment 23

[Intermittent Failures Robot]

1 failures in 5483 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2020-03-02&endday=2020-03-08&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures in 5336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1543496&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Green on Try now and being re-enabled by https://phabricator.services.mozilla.com/D133090.