Intermittent test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"

REOPENED
Assigned to

Status

()

P3
normal
REOPENED
3 years ago
6 days ago

People

(Reporter: cbook, Assigned: kdavis)

Tracking

({intermittent-failure})

unspecified
intermittent-failure
Points:
---

Firefox Tracking Flags

(firefox41 unaffected, firefox42 unaffected, firefox43 disabled, firefox-esr38 unaffected, b2g-master disabled)

Details

(Whiteboard: [systemsfe][test disabled on B2G][leave open], URL)

(Reporter)

Description

3 years ago
https://treeherder.mozilla.org/logviewer.html#?job_id=12506892&repo=mozilla-inbound

 01:49:37 INFO - 5967 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Updated

3 years ago
Assignee: nobody → kdavis
(Assignee)

Comment 3

3 years ago
Looking at the history of this problem[1] it seems problems with this test started appearing
about commit c8c81a51b3fc which changed the way media-playback notifications were given out
and disappeared when this commit was rolled back 3b39c0e12d03. 

After c8c81a51b3fc was backed out it looks like the only similar failure was in a7860794b00e
out of what was, at the time of this writing, about 50 different runs of the test.

In addition, the test was re-ran 10 times on my initial commit for the test[2] and it worked
all 10 times.

My working assumption at this point is that the test is fine and the problems in the test
were introduced through the commit c8c81a51b3fc which was rolled back in commit 3b39c0e12d03

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=65cdf156bf22&filter-searchStr=B2G%20ICS%20Emulator%20opt%20Mochitest%20Mochitest%20M%286%29
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ee1a6097272
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INVALID
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Updated

3 years ago
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Whiteboard: [systemsfe]
Keywords: intermittent-failure
Comment hidden (Treeherder Robot)
Duplicate of this bug: 1191977
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Assignee)

Comment 43

3 years ago
This is a note to my future self.	

There are many threading issues with this, and other, speech recognition tests.
Below I'll look at one of them.

This test makes use of the file

                 dom/media/webspeech/recognition/test/head.js

which currently creates an EventManager and then starts it. The start method on
the EventManager is as follows:

  self.start = function EventManager_start() {
    isSendingAudioData = true;
    var audioTag = document.createElement("audio");
    audioTag.src = self.audioSampleFile;

    var stream = audioTag.mozCaptureStreamUntilEnded();
    audioTag.addEventListener("ended", function() {
      info("Sample stream ended, requesting queued events");
      isSendingAudioData = false;
      while (queuedEventRequests.length) {
        self.requestFSMEvent(queuedEventRequests.shift());
      }
    });

    audioTag.play();
    sr.start(stream);
  }

Now as audioTag.play() does not play the audio in the main thread,
by the time sr.start(stream) is called the state of the stream is
random:

 * It could have not started playing
 * It could have started playing but not gotten to the speech
 * It could have started playing, but be somewhere in the middle of the speech.
 * It could have started playing and be finished with the speech
 * It could have finished playing

So, the events that the SpeechRecognition engine "sr" fires are also random.
It may fire:

 * audiostart, soundstart, speechstart, speechend, soundend, audioend
 * soundstart, speechstart, speechend, soundend, audioend
 * speechend, soundend, audioend
 * soundend, audioend
 * no events

What it actually does is completely random.

The errors one sees for this test are all of the form:

 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"

which, in light of the threading issues above, are no surprise.

In particular it looks as if the errors associated with this
bug are a result of the final three cases:

 * It could have started playing, but be somewhere in the middle of the speech.
 * It could have started playing and be finished with the speech
 * It could have finished playing

This the try below switches the order of play() and start() above to:

    sr.start(stream);
    audioTag.play();

This should guarantee that the events fired by "sr" are as follows:

 * audiostart, soundstart, speechstart, speechend, soundend, audioend
 
So, in particular, the error

 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
 
should not occur as "sr" will always be presented with the speech.

The try for this patch is running here https://treeherder.mozilla.org/#/jobs?repo=try&revision=a92b336195a8

In this try the test B2G ICS Emulator opt M(6) was run about 72 times
and of these 72 times it failed in the manner described in this bug 3
times. With logs of the form:

11:41:30     INFO -  6304 INFO TEST-START | dom/media/webspeech/recognition/test/test_audio_capture_error.html
11:41:57     INFO -  6305 INFO Queuing event EVENT_AUDIO_ERROR until we're done sending audio data
11:41:57     INFO -  6306 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event audioend
11:41:57     INFO -  6307 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | audioend must come after audiostart
11:41:57     INFO -      @dom/media/webspeech/recognition/test/head.js:85:1
11:41:57     INFO -  6308 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event end
11:41:57     INFO -  6309 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event error
11:41:57     INFO -  6310 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
11:41:57     INFO -      SimpleTest.is@SimpleTest/SimpleTest.js:277:5
11:41:57     INFO -      buildErrorCallback/<@dom/media/webspeech/recognition/test/head.js:142:5
11:41:57     INFO -      @dom/media/webspeech/recognition/test/head.js:89:1

The log presented here seems impossible to achieve given the changes
made to head.js in the test. In particular, the audioend coming be-
fore audiostart seems impossible.  So, further work is required,
and there is likely at least one other threading issue involved
in this bug.
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
ni? myself to disable this tomorrow
Flags: needinfo?(ryanvm)
status-b2g-master: --- → disabled
status-firefox41: --- → unaffected
status-firefox42: --- → unaffected
status-firefox43: --- → disabled
status-firefox-esr38: --- → unaffected
Flags: needinfo?(ryanvm)
Whiteboard: [systemsfe] → [systemsfe][test disabled on B2G][leave open]
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
8 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* android-4-3-armv7-api11: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-01-11&endday=2016-01-17&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 5
* mozilla-central: 1
* fx-team: 1

Platform breakdown:
* android-4-3-armv7-api15: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-05-23&endday=2016-05-29&tree=all
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* fx-team: 1

Platform breakdown:
* android-4-3-armv7-api15: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-05-30&endday=2016-06-05&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 7

Platform breakdown:
* android-4-3-armv7-api15: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-06-06&endday=2016-06-12&tree=all
9 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 6
* mozilla-aurora: 1
* fx-team: 1
* autoland: 1

Platform breakdown:
* android-4-3-armv7-api15: 9

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-06-27&endday=2016-07-03&tree=all
36 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 18
* try: 8
* autoland: 5
* fx-team: 4
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 36

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-07-04&endday=2016-07-10&tree=all
12 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* autoland: 3
* try: 1

Platform breakdown:
* android-4-3-armv7-api15: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-07-11&endday=2016-07-17&tree=all
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* fx-team: 2
* try: 1
* mozilla-aurora: 1

Platform breakdown:
* android-4-3-armv7-api15: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-07-18&endday=2016-07-24&tree=all
21 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 11
* fx-team: 5
* autoland: 5

Platform breakdown:
* android-4-3-armv7-api15: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-08-15&endday=2016-08-21&tree=all
12 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 8
* autoland: 4

Platform breakdown:
* android-4-3-armv7-api15: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-08-22&endday=2016-08-28&tree=all

Comment 98

2 years ago
Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.
Priority: -- → P3
16 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* autoland: 5
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 16

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-08-29&endday=2016-09-04&tree=all
10 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 5
* mozilla-inbound: 3
* try: 2

Platform breakdown:
* android-4-3-armv7-api15: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-09-05&endday=2016-09-11&tree=all
35 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 16
* autoland: 13
* fx-team: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 35

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-09-12&endday=2016-09-18&tree=all
24 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 10
* autoland: 8
* mozilla-beta: 2
* fx-team: 2
* try: 1
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 24

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-09-19&endday=2016-09-25&tree=all
10 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 4
* autoland: 3
* mozilla-beta: 1
* mozilla-aurora: 1
* fx-team: 1

Platform breakdown:
* android-4-3-armv7-api15: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-09-26&endday=2016-10-02&tree=all
7 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* autoland: 2
* mozilla-beta: 1
* mozilla-aurora: 1
* fx-team: 1

Platform breakdown:
* android-4-3-armv7-api15: 7

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-10-03&endday=2016-10-09&tree=all
6 failures in 715 pushes (0.008 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* autoland: 4
* mozilla-inbound: 1
* mozilla-beta: 1

Platform breakdown:
* android-4-3-armv7-api15: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2016-11-14&endday=2016-11-20&tree=all
1 failures in 1032 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* android-4-3-armv7-api16: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1191270&startday=2017-09-11&endday=2017-09-17&tree=all
2 failures in 765 pushes (0.003 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* android-4-3-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-04-02&endday=2018-04-08&tree=trunk
1 failures in 669 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-06-04&endday=2018-06-10&tree=trunk
1 failures in 2579 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* autoland: 1

Platform breakdown:
* windows10-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-08-06&endday=2018-08-12&tree=all
4 failures in 2672 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1
* autoland: 1
* mozilla-inbound: 2

Platform breakdown:
* android-em-4-3-armv7-api16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-08-13&endday=2018-08-19&tree=all
9 failures in 2568 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 1
* mozilla-central: 1
* try: 1
* mozilla-inbound: 2
* autoland: 4

Platform breakdown:
* android-em-4-3-armv7-api16: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-08-20&endday=2018-08-26&tree=all
10 failures in 2545 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 1
* mozilla-central: 3
* mozilla-inbound: 1
* autoland: 5

Platform breakdown:
* android-em-4-3-armv7-api16: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-08-27&endday=2018-09-02&tree=all
10 failures in 2667 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* try: 2
* mozilla-beta: 2
* autoland: 6

Platform breakdown:
* android-em-4-3-armv7-api16: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-09-03&endday=2018-09-09&tree=all
1 failures in 2684 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* android-em-4-3-armv7-api16: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1191270&startday=2018-09-10&endday=2018-09-16&tree=all
You need to log in before you can comment on or make changes to this bug.