1372325 - Allow the same SpeechSynthesisUtterance to be played multiple times in speechSynthesis.speak()

Status: RESOLVED DUPLICATE of bug 1523920

(Core :: Web Speech, defect, P3)

Description

[Jason Duell]

Go to

  https://www.google.com/search?q=les+m%C3%A2ts&ie=utf-8&oe=utf-8#q=glitch+in+italian

Click the speaker icon (for either original English or Italian translation).  It will play.

Click it again--it won't reply the audio.  Replaying audio works fine on Chrome.

The only way to hear the audio again is to reload the page, and that gets frustrating because if you change the word you're translating, that doesn't change the URI, so reloading brings you back to whatever word you first translated.

Interestingly, I just noticed that when you change the word you want translated, you can hear the audio for the new word--but just once.

Comment 1

[James Cheng[:JamesCheng]]

I can reproduce it but the behavior is not consistent between OSs.

Windows
original English (cannot replay)
Italian translation (can replay)

Linux (OK)
original English (can replay)
Italian translation (can replay)

Mac
original English (cannot replay)
Italian translation (cannot replay)

I can have a first look and try to find what's wrong.

Comment 2

[James Cheng[:JamesCheng]]

Hi bwu and Anthony,

Please see my analysis below,



This webpage [2] uses different approach to play the sound.

Windows and Mac use WebSpeech synthesis API to play audio.

Ubuntu uses HTMLMediaElement to play audio.

The issue occurred by using WebSpeech API(Even I change Mac's firefox to Ubuntu's user-agent, it still uses WebSpeech which I cannot the reason and the JS code is too complex to read.)

If you pref off "media.webspeech.synth.enabled", Mac and Windows can play it well by HTMLMediaElement.

Chrome uses WebSpeech API in Windows and Mac without bug and uses HTMLMediaElemnt in Linux.

In conclusion,

I suggest changing the component to "Web Speech" and find WebCompact team for help to request Google using the HTMLMediaElement for playing audio.

The reason is [1] uses HTMLMediaElement to play audio for all platforms, it does't make sense why they choose WebSpeech in [2].

I think it is a huge UX impact that we cannot play audio over once...

What do you think?


[1]
https://translate.google.com.tw/?hl=zh-TW#zh-CN/it/glitch


[2]
https://www.google.com/search?q=les+m%C3%A2ts&ie=utf-8&oe=utf-8#q=glitch+in+italian

Comment 3

[Blake Wu [:bwu][:blakewu]]

Sweet. Let's change the component to Web Speech to get their help.

Comment 4

[James Cheng[:JamesCheng]]

Are you sure we don't need to do more action as I mention in comment 2?

Comment 5

[Blake Wu [:bwu][:blakewu]]

(In reply to James Cheng[:JamesCheng] from comment #4)
> Are you sure we don't need to do more action as I mention in comment 2?
Sure. That is a good idea. Let's just get some opinions from web speech team first.

Comment 6

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This is an assigned P1 bug without activity in two weeks. 

If you intend to continue working on this bug for the current release/iteration/sprint, remove the 'stale-bug' keyword.

Otherwise we'll reset the priority of the bug back to '--' on Monday, August 28th.

Comment 7

[James Cheng[:JamesCheng]]

it still can be reproduced...

Hi Eric,
According to comment 2, Do you or your teammate can move this bug forward?

I think it is a bad UX when they want to listen to the pronunciation again but it can't.

Thanks.

Comment 8

[Makoto Kato [:m_kato]]

Although I don't investigate Google's javascript, I think that SpeechSynthesisUtterance is reused.  So aUtterance.mState in SpeechSynthesis::Speak is STATE_END, so speak method is nothing action.

Comment 9

[Eric Tsai (no more review request accepted)]

My opinion is: using WebSpeech API might be a business decision or tech decision, we don't need to force them to change the decision.
From the analysis in comment 2, my understanding is WebSpeech API works in Chrome but not in Firefox. WebCompat team could help to reach Google Translate team to see if we can cowork to fix this issue. But finally this bug still expert in WebSpeech to help to solve.

Comment 10

[Makoto Kato [:m_kato]]

Edge, Safari and Chrome will speak forever by the following script.  But Firefox doesn't.

var utter = new SpeechSynthesisUtterance("Hello Word");
utter.onend = () => {
window.speechSynthesis.speak(utter);
};
window.speechSynthesis.speak(utter);


Although fix is on Gecko side, is is webcompat issue?

Comment 11

[Makoto Kato [:m_kato]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cfd1925cd2c3e37d00d6ef40f8a50a4d322e41f6

Comment 12

[James Cheng[:JamesCheng]]

(In reply to Makoto Kato [:m_kato] from comment #10)
> Edge, Safari and Chrome will speak forever by the following script.  But
> Firefox doesn't.
> 
> var utter = new SpeechSynthesisUtterance("Hello Word");
> utter.onend = () => {
> window.speechSynthesis.speak(utter);
> };
> window.speechSynthesis.speak(utter);
> 
> 
> Although fix is on Gecko side, is is webcompat issue?

It would be nice to fix by ourselves.

I just notice that Google "doesn't" use WebSpeech on Linux and in Google Translation web page.

If they can unify the API usage, it can save the debugging time for programmer that I thought they use HTMLMediaelement on all platforms.

Comment 13

[Makoto Kato [:m_kato]]

Attachment: Bug 1372325 - Allow to reuse SpeechSynthesisUtterance.

Google translate in search page reuses SpeechSynthesisUtterance.  So after output utterance, it no longer work until page is reloaded.  Edge, Safari and Blink can reuse SpeechSynthesisUtterance, so we should allow to reuse it.

Review commit: https://reviewboard.mozilla.org/r/180158/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/180158/

Comment 14

[Eitan Isaacson [:eeejay]]

Comment on attachment 8908488 [details]
Bug 1372325 - Allow to reuse SpeechSynthesisUtterance.

https://reviewboard.mozilla.org/r/180158/#review185542

I originally implemented utterances to not be reusable because I thought it was in the spec. I don't see that now, maybe I was wrong. This patch makes me nervous, but I can't find anything wrong with it up front. Having tests is good.

The one thing that does need to be fixed before r+: An utterance cannot be reused by another SpeechSynthesis instance[1].

1. https://w3c.github.io/speech-api/webspeechapi.html#tts-methods

Comment 15

[Blake Wu [:bwu][:blakewu]]

[Tracking Requested - why for this release]:
Google translation should be used a lot. It would be good to fix this.

Comment 16

[Makoto Kato [:m_kato]]

https://github.com/w3c/speech-api/issues/7

Comment 18

[Karl Dubost💡 :karlcow]

Web compatibility rationale for it

1. Edge, Chrome and Safari allows it. Firefox don't
2. Used by at least one major site: Google Search.

This is unlikely that we are able to move the world under our feet. 
So we need to implement this too.

Comment 19

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Makoto, this bug is nominated for tracking in 57. I noticed that the proposed fix was r-. Is there another fix in the works? Given the many moving pieces in 57 release, if the fix is not ready soon (a week or so), the likelihood of fixing this in 57 is low. Does it make sense to wontfix this for 57 and attempt to fix this in 58 instead?

Comment 20

[Makoto Kato [:m_kato]]

(In reply to Ritu Kothari (:ritu) from comment #19)
> Hi Makoto, this bug is nominated for tracking in 57. I noticed that the
> proposed fix was r-. Is there another fix in the works? Given the many
> moving pieces in 57 release, if the fix is not ready soon (a week or so),
> the likelihood of fixing this in 57 is low. Does it make sense to wontfix
> this for 57 and attempt to fix this in 58 instead?

Although Google site is broken for our Firefox, This is Web Speech API spec issue (undocumented behavior).  Until https://github.com/w3c/speech-api/issues/7 is resolved, we cannot fix on Firefox side.

Comment 21

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Makoto, given that I'll just wontfix for 57.

Comment 22

[Philip Jägenstedt [:foolip]]

A test for this now exists in web-platform-tests and passes in Chrome, Edge and Safari:
http://w3c-test.org/speech-api/SpeechSynthesis-speak-twice.html

(From https://github.com/w3c/web-platform-tests/pull/7510)

Per https://github.com/w3c/speech-api/issues/7#issuecomment-332860183 I would argue that this wasn't undefined in the spec, and suggest reopening this issue.

(I'm checking in here as part of https://docs.google.com/a/google.com/document/d/1j6vEd-lsB8SZF55xsa6QTr48JJLRtoYwqy2SLrAUdeM/edit?usp=sharing.)

Comment 23

[Philip Jägenstedt [:foolip]]

Oh, this issue wasn't closed, it was just wontfix for FF57. Makes sense.

Comment 24

[Mike Taylor [:miketaylr]]

(In reply to Philip Jägenstedt from comment #23)
> Oh, this issue wasn't closed, it was just wontfix for FF57. Makes sense.

(yeah we just use that to track what bugs should block a particular release)

Comment 25

[Mike Taylor [:miketaylr]]

ni? m_kato for comment #22.

Comment 26

[Makoto Kato [:m_kato]]

:eeejay, Do you have any comment?

Comment 27

[Eitan Isaacson [:eeejay]]

I think Phillip's concern was marking this as wontfix for 57, but later understood that this is not a general wontfix.

Once you update this patch to throw an error when an utterance is sent to a different window, I think we should land this.

Comment 28

[Philip Jägenstedt [:foolip]]

When making that throw, if you could pick a sensible exception, notify https://github.com/w3c/speech-api/issues/8 and update https://github.com/w3c/web-platform-tests/blob/master/speech-api/SpeechSynthesis-speak-ownership.html, that'd be great. Maybe "NotSupportedError"?

Comment 29

[a8le]

I can still reproduce this error/bug. FF only bug, very frustrating.

Comment 30

[Sylvestre Ledru [:Sylvestre]]

Moving to p3 because no activity for at least 24 weeks.

Comment 31

[Firefox Bug Husbandry Bot]

Migrating Webcompat whiteboard priorities to project flags. See bug 1547409.

Comment 32

[Firefox Bug Husbandry Bot]

See bug 1547409. Migrating whiteboard priority tags to program flags.

Comment 33

[C.M.Chang[:chunmin]]

It seems like this bug is duplicate of bug 1523920, which is a bug I am currently working on. Feel free to reopen it if this is a different case. We might not need to add a new test case. There is an existing wpt test: SpeechSynthesis-speak-twice.html to check this.