Open Bug 1191270 Opened 8 years ago Updated 1 year ago

Intermittent test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"

Categories

(Core :: Web Speech, defect, P3)

Product:
Core
Component:
Web Speech
Type:
defect

Tracking

()

Status:
REOPENED
Tracking Flags:
Tracking Status
firefox41 --- unaffected
firefox42 --- unaffected
firefox43 --- disabled
firefox-esr38 --- unaffected
b2g-master --- disabled

People

(Reporter: cbook, Unassigned)

References

(
URL
)

Details

(Keywords: intermittent-failure, Whiteboard: [systemsfe][test disabled on B2G][leave open]) 

https://treeherder.mozilla.org/logviewer.html#?job_id=12506892&repo=mozilla-inbound

 01:49:37 INFO - 5967 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
Assignee: nobody → kdavis
Looking at the history of this problem[1] it seems problems with this test started appearing
about commit c8c81a51b3fc which changed the way media-playback notifications were given out
and disappeared when this commit was rolled back 3b39c0e12d03. 

After c8c81a51b3fc was backed out it looks like the only similar failure was in a7860794b00e
out of what was, at the time of this writing, about 50 different runs of the test.

In addition, the test was re-ran 10 times on my initial commit for the test[2] and it worked
all 10 times.

My working assumption at this point is that the test is fine and the problems in the test
were introduced through the commit c8c81a51b3fc which was rolled back in commit 3b39c0e12d03

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=65cdf156bf22&filter-searchStr=B2G%20ICS%20Emulator%20opt%20Mochitest%20Mochitest%20M%286%29
[2] https://treeherder.mozilla.org/#/jobs?repo=try&revision=2ee1a6097272
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INVALID
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Whiteboard: [systemsfe]
This is a note to my future self.	

There are many threading issues with this, and other, speech recognition tests.
Below I'll look at one of them.

This test makes use of the file

                 dom/media/webspeech/recognition/test/head.js

which currently creates an EventManager and then starts it. The start method on
the EventManager is as follows:

  self.start = function EventManager_start() {
    isSendingAudioData = true;
    var audioTag = document.createElement("audio");
    audioTag.src = self.audioSampleFile;

    var stream = audioTag.mozCaptureStreamUntilEnded();
    audioTag.addEventListener("ended", function() {
      info("Sample stream ended, requesting queued events");
      isSendingAudioData = false;
      while (queuedEventRequests.length) {
        self.requestFSMEvent(queuedEventRequests.shift());
      }
    });

    audioTag.play();
    sr.start(stream);
  }

Now as audioTag.play() does not play the audio in the main thread,
by the time sr.start(stream) is called the state of the stream is
random:

 * It could have not started playing
 * It could have started playing but not gotten to the speech
 * It could have started playing, but be somewhere in the middle of the speech.
 * It could have started playing and be finished with the speech
 * It could have finished playing

So, the events that the SpeechRecognition engine "sr" fires are also random.
It may fire:

 * audiostart, soundstart, speechstart, speechend, soundend, audioend
 * soundstart, speechstart, speechend, soundend, audioend
 * speechend, soundend, audioend
 * soundend, audioend
 * no events

What it actually does is completely random.

The errors one sees for this test are all of the form:

 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"

which, in light of the threading issues above, are no surprise.

In particular it looks as if the errors associated with this
bug are a result of the final three cases:

 * It could have started playing, but be somewhere in the middle of the speech.
 * It could have started playing and be finished with the speech
 * It could have finished playing

This the try below switches the order of play() and start() above to:

    sr.start(stream);
    audioTag.play();

This should guarantee that the events fired by "sr" are as follows:

 * audiostart, soundstart, speechstart, speechend, soundend, audioend
 
So, in particular, the error

 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
 
should not occur as "sr" will always be presented with the speech.

The try for this patch is running here https://treeherder.mozilla.org/#/jobs?repo=try&revision=a92b336195a8

In this try the test B2G ICS Emulator opt M(6) was run about 72 times
and of these 72 times it failed in the manner described in this bug 3
times. With logs of the form:

11:41:30     INFO -  6304 INFO TEST-START | dom/media/webspeech/recognition/test/test_audio_capture_error.html
11:41:57     INFO -  6305 INFO Queuing event EVENT_AUDIO_ERROR until we're done sending audio data
11:41:57     INFO -  6306 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event audioend
11:41:57     INFO -  6307 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | audioend must come after audiostart
11:41:57     INFO -      @dom/media/webspeech/recognition/test/head.js:85:1
11:41:57     INFO -  6308 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event end
11:41:57     INFO -  6309 INFO TEST-PASS | dom/media/webspeech/recognition/test/test_audio_capture_error.html | received event error
11:41:57     INFO -  6310 INFO TEST-UNEXPECTED-FAIL | dom/media/webspeech/recognition/test/test_audio_capture_error.html | expect correct error code - got "no-speech", expected "audio-capture"
11:41:57     INFO -      SimpleTest.is@SimpleTest/SimpleTest.js:277:5
11:41:57     INFO -      buildErrorCallback/<@dom/media/webspeech/recognition/test/head.js:142:5
11:41:57     INFO -      @dom/media/webspeech/recognition/test/head.js:89:1

The log presented here seems impossible to achieve given the changes
made to head.js in the test. In particular, the audioend coming be-
fore audiostart seems impossible.  So, further work is required,
and there is likely at least one other threading issue involved
in this bug.
ni? myself to disable this tomorrow
Flags: needinfo?(ryanvm)
status-b2g-master: --- → disabled
status-firefox41: --- → unaffected
status-firefox42: --- → unaffected
status-firefox43: --- → disabled
status-firefox-esr38: --- → unaffected
Flags: needinfo?(ryanvm)
Whiteboard: [systemsfe] → [systemsfe][test disabled on B2G][leave open]
Bulk assigning P3 to all open intermittent bugs without a priority set in Firefox components per bug 1298978.
Priority: -- → P3
Assignee: kdavis → nobody
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.