1086545 - [AccessFu] Speech sometimes gets interrupted when swiping and it is supposed to speak the newly focused item

Status: VERIFIED FIXED

(Core :: Disability Access APIs, defect)

Description

[Marco Zehe (:MarcoZ)]

I first saw this in a build from October 20 which I was able to flash onto my Flame after a week of pause or so.

Two scenarios:
Scenario 1:
1. Turn on screen reader.
2. Swipe through items on the home screen. Don't wait for speech to finish speaking, just swipe, let speech start, then swipe again repeatedly.

Observation: Sometimes, speech gets interrupted, but doesn't resume when speaking the newly focused item. The sound for the newly focused item is heard, but speech doesn't pick up. It's as if it is interrupting itself for too long to be able to pick up speaking the new item.

Scenario 2:
1. Double-tap the address bar at the top.
2. Start exploring the keyboard.

Observation: Sometimes when exploring from one letter to the next while the other might still be spoken, the newly focused letter is not spoken.

Both above scenarios used to work reliably, e. g. speech interrupting itself and speakin the new item. This is a recent regression.

Comment 1

[Marco Zehe (:MarcoZ)]

[Tracking Requested - why for this release]: It's a regression on the Flame introduced in the 2.2 cycle that should be fixed for release. It affects the screen reader when navigating the UI on a Flame device.

Comment 2

[Marco Zehe (:MarcoZ)]

Eitan and Yura, have either of you also been able to reproduce this? Or seen it at all when you were testing something else?

Comment 3

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Marco Zehe (:MarcoZ) from comment #2)
> Eitan and Yura, have either of you also been able to reproduce this? Or seen
> it at all when you were testing something else?

I tried this and could not reliably notice significant issues. I did notice some crashes before related to audio channel and power manager issues. After speaking with Marco, we'll try to look into this issue deeper next week in Portland.

Comment 4

[Eitan Isaacson [:eeejay]]

Sounds good!

Comment 5

[Wesley Huang [:wesley_huang] (EPM) (NI me)]

Hi Marco,
Are you still planning to work on this bug?


Triage: broken function, regression

Comment 6

[Marco Zehe (:MarcoZ)]

I reported it, and can still reproduce it in latest nightly builds on the Flame device, but not on  ZTE Open running a community build of Nightly. Someone else will have to work on actually debuggingand fixing  this, though.

Comment 7

[KTucker [:KTucker][Inactive 3/4/2016]]

I don't know what exactly the qawanted is for but let's get a branch check here.

Comment 8

[Brogan Zumwalt [Inactive]]

Issue DOES occur on Flame 2.1 and 3.0

With screen reader enabled, scrolling through homescreen icons occasionally skips speaking app name when scrolling continuously. Also skips speaking some letters when scrolling through keyboard. This issue is easy to reproduce and usually occurs in 10 or less focus changes.

Device: Flame 2.1
Build ID: 20150205001711
Gaia: 17bf14f12e43043654498330d610d469b8b55e64
Gecko: bdebcc47ec7a
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 34.0 (2.1)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:34.0) Gecko/34.0 Firefox/34.0

Device: Flame 3.0 Master
Build ID: 20150205010209
Gaia: 2b83a6d5d1185a438b5bbd287497ac2743b501db
Gecko: 34a66aaaca81
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 38.0a1 (3.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:38.0) Gecko/38.0 Firefox/38.0


Issue does NOT occur on Flame 2.0

Scrolling focus through homescreen items and keyboard letters with screen reader enabled does not skip speaking any focused item when scrolling continuously.

Device: Flame 2.0
Build ID: 20150205000205
Gaia: 2989f2b2bd12fcc0e9c017d2db766e76a55873b8
Gecko: 69057b33ef5b
Gonk: e7c90613521145db090dd24147afd5ceb5703190
Version: 32.0 (2.0)
Firmware Version: v18D-1
User Agent: Mozilla/5.0 (Mobile; rv:32.0) Gecko/32.0 Firefox/32.0

Comment 9

[KTucker [:KTucker][Inactive 3/4/2016]]

Let's get a regression window then.

Comment 10

[Shally Li（leave from Mozilla）]

I'm finding the broken window.

Comment 11

[Shally Li（leave from Mozilla）]

Regression Window:

I can repro this bug on the earliest Flame v2.1 (2014-10-08-16-12-01).

See attachments: verify_v2.1.MP4 and logcat_v2.1_0509.txt
Reproduce rate: 5/5

Repro STR:
1.Enable screen reader in Settings.
2.Go to Homescreen.
3.Swipe through items on the homescreen. Don't wait for speech to finish speaking, just swipe, let speech start, then swipe again repeatedly.
**Speech gets interrupted when user scroll on the next icon and it will continue to speak the newly  icon.(Please see video)
  
4.Tap on Rocket bar and start exploring the keyboard.
**When exploring from one letter to the next letter while the other might still be spoken, the newly focused letter will continue to speak.
**Sometimes,when long tap a letter on the keyboard,some icons will disappear or cut off on the homescreen.(please see video and log) 

Flame earliest 2.1 build:
Build ID               20141008161201
Gaia Revision          7ef2e1e59637a34ca4489c329b3bdee93df3ac6c
Gaia Date              2014-10-08 12:32:21
Gecko Revision         https://hg.mozilla.org/releases/mozilla-b2g34_v2_1/rev/e3d495eb85c6
Gecko Version          34.0a2
Device Name            flame
Firmware(Release)      4.4.2
Firmware(Incremental)  eng.cltbld.20150208.050436
Firmware Date          Sun Feb  8 05:04:47 EST 2015
Bootloader             L1TC000118D0

Comment 12

[Shally Li（leave from Mozilla）]

Hi Reporter,

    Could you help to confirm above steps and results in Comment 11?

Thank you very much?

Comment 13

[Shally Li（leave from Mozilla）]

Attachment: logcat_v2.1_0509.txt

Comment 14

[Shally Li（leave from Mozilla）]

Attachment: verify_v2.1.MP4

Comment 15

[Marco Zehe (:MarcoZ)]

Yes, that is basically what I'm seeing. Note that I cannot confirm visuals, since I am totally blind and can only *hear* that something is wrong. But the steps sound quite right.

Comment 16

[Eitan Isaacson [:eeejay]]

Attachment: Relevent PRLog

I did some more digging, and this seems to be some kind of race condition in our MediaGraph code.

Paul, to reproduce, you need to turn on the screen reader[1] and then in the homescreen swipe and move through the icons quickly. Eventually you will hear a tick that is not followed by any speech.

https://wiki.mozilla.org/Accessibility/Mobile/ScreenReader

Comment 17

[Eitan Isaacson [:eeejay]]

I think the logging I attached might give a good hint. The 'Music2 link' utterance didn't happen, but the ones above worked.

Comment 18

[Paul Adenot (:padenot)]

Yeah I think what happened is that there was a little window of time where there were 0 streams with an audio track, so the graph switched to using a system thread to save battery, and then it found it could shut down, so it did that.

This never happens with Web Audio API because we have always have a stream where all the other streams are connected to (the AudioDestinationNode), I think you should try to replicate this design: keep a stream with an audio track, that would have the AudioOuput, and then, when you need to play a sound, connect your WebSpeech stream to this new stream. This will assure that the graph always uses a audio thread to run, and keep the latency minimal as a nice bonus.

Comment 19

[Yura Zenevich [:yzen] (please use "needinfo")]

I already have a WebAudio patch so I will take this one.

Comment 20

[Eitan Isaacson [:eeejay]]

Attachment: Possible solution.

I am not familiar with the media API, am I in the right direction? I think I notice better latency, and I can't reproduce this bug.

Comment 21

[Paul Adenot (:padenot)]

Comment on attachment 8570195 [details] [diff] [review]
Possible solution.

Review of attachment 8570195 [details] [diff] [review]:
-----------------------------------------------------------------

That ought to work, but this really looks like a _perfect_ use case for the Web Audio API.

Comment 22

[Eitan Isaacson [:eeejay]]

(In reply to Paul Adenot (:padenot) from comment #21)
> Comment on attachment 8570195 [details] [diff] [review]
> Possible solution.
> 
> Review of attachment 8570195 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> That ought to work, but this really looks like a _perfect_ use case for the
> Web Audio API.

I'm not really sure what you mean. How would we use a JS API here? Just instantiate an AudioContext in cpp? That seems like a bit of overkill, I can't find any examples of where that is used, are there any? If we do this, we would be using the global top-level window as a parent because of the architecture of the voice service.

Are there any battery/performance considerations for having a permanent stream like you suggest in comment #18, and like I implemented? Should there be some kind of timeout?

Comment 23

[Paul Adenot (:padenot)]

(In reply to Eitan Isaacson [:eeejay] from comment #22)
> (In reply to Paul Adenot (:padenot) from comment #21)
> > Comment on attachment 8570195 [details] [diff] [review]
> > Possible solution.
> > 
> > Review of attachment 8570195 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > That ought to work, but this really looks like a _perfect_ use case for the
> > Web Audio API.
> 
> I'm not really sure what you mean. How would we use a JS API here? Just
> instantiate an AudioContext in cpp? That seems like a bit of overkill, I
> can't find any examples of where that is used, are there any? If we do this,
> we would be using the global top-level window as a parent because of the
> architecture of the voice service.

It's not used, but it should work, you'd just bypass the bindings, right?
 
> Are there any battery/performance considerations for having a permanent
> stream like you suggest in comment #18, and like I implemented? Should there
> be some kind of timeout?

There are battery and performance implications, basically, there will be a thread running all the time, and the audio hardware will be awake until you tell the MediaStreamGraph to pause. Work in bug 1094764 can be useful, here, but, you can implement a timeout if you want, yes, it's just a matter of removing the "sink" stream on timeout.

Comment 24

[Kai-Chih Hu [:khu]]

Sheila, can you help to find a best fit from platform team to own this bug? Thank you.

Comment 25

[Kai-Chih Hu [:khu]]

So, what's the next action? Ready to review or? 
And, Eitan, can you be the owner for this bug? 

Thanks.

Comment 26

[Shing Lyu [:shinglyu]]

This bug has been idle for a long time, and it's marked blocking-b2g=2.2+, can we make it before 4/6 FC?

Comment 27

[Eitan Isaacson [:eeejay]]

Sorry, I was researching alternatives. And nothing came up. I'm going to double check this patch, and prepare it for review.

Comment 28

[Eitan Isaacson [:eeejay]]

Attachment: Bind speech task streams to a parent stream held by voice registry.

The input port count does not accumulate and goes back to 0 after a nsSpeechTask is destroyed, so I assume it is not keeping the thread alive, and we are not killing battery.

Comment 29

[Paul Adenot (:padenot)]

Comment on attachment 8586934 [details] [diff] [review]
Bind speech task streams to a parent stream held by voice registry.

Review of attachment 8586934 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/media/webspeech/synth/nsSpeechTask.cpp
@@ +162,2 @@
>  
>    // XXX: Is there setup overhead here that hurtls latency?

You can probably remove this.

Comment 30

[Eitan Isaacson [:eeejay]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/717878ab4b45

Comment 31

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/717878ab4b45

Comment 32

[Eitan Isaacson [:eeejay]]

Comment on attachment 8586934 [details] [diff] [review]
Bind speech task streams to a parent stream held by voice registry.

NOTE: Please see https://wiki.mozilla.org/Release_Management/B2G_Landing to better understand the B2G approval process and landings.

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 
User impact if declined: Speech gets clipped when navigating quickly with screen reader.
Testing completed: Yes.
Risk to taking this patch (and alternatives if risky): Low
String or UUID changes made by this patch: No.

Comment 33

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g37_v2_2/rev/22373af050a2

Comment 34

[Pi Wei Cheng [:piwei] (inactive)]

This issue is verified as fixed on central Flame and Aries. Device now properly announce newly focused items if previous item announcing was interrupted. Tested this several minutes on each device/branch and heard no significant issues.

Device: Flame 2.6
BuildID: 20151210030222
Gaia: 961528f4391668bc89ec0be14fa367cea099b588
Gecko: b40ba117fa757861c9caa660ae989122718b494b
Gonk: 205ac4204bbbb2098a8046444acba551ba5dc75a
Version: 45.0a1 (2.6) 
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:45.0) Gecko/45.0 Firefox/45.0

Device: Aries 2.6
BuildID: 20151210122424
Gaia: 7e962276913bd4da7ce5fa7540767107ce322c78
Gecko: 412e4d7ce98ca4dbc37de133d0f26d7e1a59946f
Gonk: a19052e4389c3ae2d8fc3e7a74a475401baacc56
Version: 45.0a1 (2.6) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:45.0) Gecko/45.0 Firefox/45.0

---------

This issue is NOT fixed on Flame 2.5, Aries 2.5. The bug reproduces on these branches, with 2.2 exhibiting the issue much less frequently than 2.5. Newly focused item doesn't always get announced if previous announcing was interrupted.

I'll be filing a separate bug for this.

Device: Flame 2.5
BuildID: 20151209170211
Gaia: 7ca639a7bb0bacf27f548841c52617bfc0e3b21f
Gecko: a35e8eb98969970d1af28b265bf99a9edd11e9c2
Gonk: 205ac4204bbbb2098a8046444acba551ba5dc75a
Version: 44.0a2 (2.5) 
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:44.0) Gecko/44.0 Firefox/44.0

Device: Aries 2.5
BuildID: 20151209171644
Gaia: 7ca639a7bb0bacf27f548841c52617bfc0e3b21f
Gecko: a35e8eb98969970d1af28b265bf99a9edd11e9c2
Gonk: a19052e4389c3ae2d8fc3e7a74a475401baacc56
Version: 44.0a2 (2.5) 
Firmware Version: D5803_23.1.A.1.28_NCB.ftf
User Agent: Mozilla/5.0 (Mobile; rv:44.0) Gecko/44.0 Firefox/44.0

Device: Flame 2.2
BuildID: 20151209032501
Gaia: 885647d92208fb67574ced44004ab2f29d23cb45
Gecko: 4381c4b69b9c
Gonk: bd9cb3af2a0354577a6903917bc826489050b40d
Version: 37.0 (2.2) 
Firmware Version: v18D
User Agent: Mozilla/5.0 (Mobile; rv:37.0) Gecko/37.0 Firefox/37.0