Allow the same SpeechSynthesisUtterance to be played multiple times in speechSynthesis.speak()

NEW
Assigned to

Status

()

P3
normal
2 years ago
a month ago

People

(Reporter: jduell.mcbugs, Assigned: m_kato)

Tracking

unspecified
Points:
---
Bug Flags:
webcompat +

Firefox Tracking Flags

(firefox56 wontfix, firefox57- wontfix, firefox58 affected)

Details

(Whiteboard: [webcompat], URL)

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
Go to

  https://www.google.com/search?q=les+m%C3%A2ts&ie=utf-8&oe=utf-8#q=glitch+in+italian

Click the speaker icon (for either original English or Italian translation).  It will play.

Click it again--it won't reply the audio.  Replaying audio works fine on Chrome.

The only way to hear the audio again is to reload the page, and that gets frustrating because if you change the word you're translating, that doesn't change the URI, so reloading brings you back to whatever word you first translated.

Interestingly, I just noticed that when you change the word you want translated, you can hear the audio for the new word--but just once.
Component: Untriaged → Audio/Video: Playback
Product: Firefox → Core
Priority: -- → P1
I can reproduce it but the behavior is not consistent between OSs.

Windows
original English (cannot replay)
Italian translation (can replay)

Linux (OK)
original English (can replay)
Italian translation (can replay)

Mac
original English (cannot replay)
Italian translation (cannot replay)

I can have a first look and try to find what's wrong.
Assignee: nobody → jacheng
Hi bwu and Anthony,

Please see my analysis below,



This webpage [2] uses different approach to play the sound.

Windows and Mac use WebSpeech synthesis API to play audio.

Ubuntu uses HTMLMediaElement to play audio.

The issue occurred by using WebSpeech API(Even I change Mac's firefox to Ubuntu's user-agent, it still uses WebSpeech which I cannot the reason and the JS code is too complex to read.)

If you pref off "media.webspeech.synth.enabled", Mac and Windows can play it well by HTMLMediaElement.

Chrome uses WebSpeech API in Windows and Mac without bug and uses HTMLMediaElemnt in Linux.

In conclusion,

I suggest changing the component to "Web Speech" and find WebCompact team for help to request Google using the HTMLMediaElement for playing audio.

The reason is [1] uses HTMLMediaElement to play audio for all platforms, it does't make sense why they choose WebSpeech in [2].

I think it is a huge UX impact that we cannot play audio over once...

What do you think?


[1]
https://translate.google.com.tw/?hl=zh-TW#zh-CN/it/glitch


[2]
https://www.google.com/search?q=les+m%C3%A2ts&ie=utf-8&oe=utf-8#q=glitch+in+italian
Flags: needinfo?(bwu)
Flags: needinfo?(ajones)
Sweet. Let's change the component to Web Speech to get their help.
Flags: needinfo?(bwu)
Flags: needinfo?(ajones)
Component: Audio/Video: Playback → Web Speech
Are you sure we don't need to do more action as I mention in comment 2?
(In reply to James Cheng[:JamesCheng] from comment #4)
> Are you sure we don't need to do more action as I mention in comment 2?
Sure. That is a good idea. Let's just get some opinions from web speech team first.
This is an assigned P1 bug without activity in two weeks. 

If you intend to continue working on this bug for the current release/iteration/sprint, remove the 'stale-bug' keyword.

Otherwise we'll reset the priority of the bug back to '--' on Monday, August 28th.
Keywords: stale-bug
it still can be reproduced...

Hi Eric,
According to comment 2, Do you or your teammate can move this bug forward?

I think it is a bad UX when they want to listen to the pronunciation again but it can't.

Thanks.
Flags: needinfo?(etsai)
Whiteboard: [webcompat]
(Assignee)

Comment 8

a year ago
Although I don't investigate Google's javascript, I think that SpeechSynthesisUtterance is reused.  So aUtterance.mState in SpeechSynthesis::Speak is STATE_END, so speak method is nothing action.
My opinion is: using WebSpeech API might be a business decision or tech decision, we don't need to force them to change the decision.
From the analysis in comment 2, my understanding is WebSpeech API works in Chrome but not in Firefox. WebCompat team could help to reach Google Translate team to see if we can cowork to fix this issue. But finally this bug still expert in WebSpeech to help to solve.
Flags: needinfo?(etsai)
(Assignee)

Comment 10

a year ago
Edge, Safari and Chrome will speak forever by the following script.  But Firefox doesn't.

var utter = new SpeechSynthesisUtterance("Hello Word");
utter.onend = () => {
window.speechSynthesis.speak(utter);
};
window.speechSynthesis.speak(utter);


Although fix is on Gecko side, is is webcompat issue?
(In reply to Makoto Kato [:m_kato] from comment #10)
> Edge, Safari and Chrome will speak forever by the following script.  But
> Firefox doesn't.
> 
> var utter = new SpeechSynthesisUtterance("Hello Word");
> utter.onend = () => {
> window.speechSynthesis.speak(utter);
> };
> window.speechSynthesis.speak(utter);
> 
> 
> Although fix is on Gecko side, is is webcompat issue?

It would be nice to fix by ourselves.

I just notice that Google "doesn't" use WebSpeech on Linux and in Google Translation web page.

If they can unify the API usage, it can save the debugging time for programmer that I thought they use HTMLMediaelement on all platforms.
Comment hidden (mozreview-request)
(Assignee)

Updated

a year ago
Assignee: jacheng → m_kato

Comment 14

a year ago
mozreview-review
Comment on attachment 8908488 [details]
Bug 1372325 - Allow to reuse SpeechSynthesisUtterance.

https://reviewboard.mozilla.org/r/180158/#review185542

I originally implemented utterances to not be reusable because I thought it was in the spec. I don't see that now, maybe I was wrong. This patch makes me nervous, but I can't find anything wrong with it up front. Having tests is good.

The one thing that does need to be fixed before r+: An utterance cannot be reused by another SpeechSynthesis instance[1].

1. https://w3c.github.io/speech-api/webspeechapi.html#tts-methods
Attachment #8908488 - Flags: review?(eitan) → review-
[Tracking Requested - why for this release]:
Google translation should be used a lot. It would be good to fix this.
status-firefox56: --- → affected
status-firefox57: --- → affected
tracking-firefox57: --- → ?
status-firefox56: affected → wontfix
Keywords: stale-bug
Duplicate of this bug: 1314065
Summary: Firefox won't replay Google translation audio → Allow the same SpeechSynthesisUtterance to be played multiple times in speechSynthesis.speak()
Web compatibility rationale for it

1. Edge, Chrome and Safari allows it. Firefox don't
2. Used by at least one major site: Google Search.

This is unlikely that we are able to move the world under our feet. 
So we need to implement this too.
Flags: webcompat+
Hi Makoto, this bug is nominated for tracking in 57. I noticed that the proposed fix was r-. Is there another fix in the works? Given the many moving pieces in 57 release, if the fix is not ready soon (a week or so), the likelihood of fixing this in 57 is low. Does it make sense to wontfix this for 57 and attempt to fix this in 58 instead?
Flags: needinfo?(m_kato)
(Assignee)

Comment 20

a year ago
(In reply to Ritu Kothari (:ritu) from comment #19)
> Hi Makoto, this bug is nominated for tracking in 57. I noticed that the
> proposed fix was r-. Is there another fix in the works? Given the many
> moving pieces in 57 release, if the fix is not ready soon (a week or so),
> the likelihood of fixing this in 57 is low. Does it make sense to wontfix
> this for 57 and attempt to fix this in 58 instead?

Although Google site is broken for our Firefox, This is Web Speech API spec issue (undocumented behavior).  Until https://github.com/w3c/speech-api/issues/7 is resolved, we cannot fix on Firefox side.
Flags: needinfo?(m_kato)
Thanks Makoto, given that I'll just wontfix for 57.
status-firefox57: affected → wontfix
status-firefox58: --- → affected
tracking-firefox57: ? → -
A test for this now exists in web-platform-tests and passes in Chrome, Edge and Safari:
http://w3c-test.org/speech-api/SpeechSynthesis-speak-twice.html

(From https://github.com/w3c/web-platform-tests/pull/7510)

Per https://github.com/w3c/speech-api/issues/7#issuecomment-332860183 I would argue that this wasn't undefined in the spec, and suggest reopening this issue.

(I'm checking in here as part of https://docs.google.com/a/google.com/document/d/1j6vEd-lsB8SZF55xsa6QTr48JJLRtoYwqy2SLrAUdeM/edit?usp=sharing.)
Oh, this issue wasn't closed, it was just wontfix for FF57. Makes sense.
(In reply to Philip Jägenstedt from comment #23)
> Oh, this issue wasn't closed, it was just wontfix for FF57. Makes sense.

(yeah we just use that to track what bugs should block a particular release)
ni? m_kato for comment #22.
Flags: needinfo?(m_kato)
(Assignee)

Comment 26

a year ago
:eeejay, Do you have any comment?
Flags: needinfo?(m_kato) → needinfo?(eitan)
I think Phillip's concern was marking this as wontfix for 57, but later understood that this is not a general wontfix.

Once you update this patch to throw an error when an utterance is sent to a different window, I think we should land this.
Flags: needinfo?(eitan)
When making that throw, if you could pick a sensible exception, notify https://github.com/w3c/speech-api/issues/8 and update https://github.com/w3c/web-platform-tests/blob/master/speech-api/SpeechSynthesis-speak-ownership.html, that'd be great. Maybe "NotSupportedError"?

Comment 29

8 months ago
I can still reproduce this error/bug. FF only bug, very frustrating.
Moving to p3 because no activity for at least 24 weeks.
Priority: P1 → P3
You need to log in before you can comment on or make changes to this bug.