1294765 - Reader Mode Narrate - Language/Country names next to voices don't reflect the language they speak if the article is in English / some other language

Status: RESOLVED FIXED

(Toolkit :: Reader Mode, defect, P5)

Description

[julesmyers2011]

(1) I went to https://blog.monotonous.org/2016/03/07/narrate-a-new-feature-in-firefox-nightly/
(2) Put the page in reader mode
(3) Clicked on Narrate and hit play
(4) For voice I choose Carmit(Hebrew)

What I was expecting was someone to speak in Hebrew. Instead the voice is speaking English.

Comment 1

[julesmyers2011]

I also ran into this problem with:
Damayanit(Indonesian)
Kanya (Thai)
Lekha (Hindi)
Mei-Jia (Chinese)
Melina (Greek)
Minena (Russia)

Comment 2

[Moin Shaikh]

(In reply to julesmyers2011 from comment #1)
> I also ran into this problem with:
> Damayanit(Indonesian)
> Kanya (Thai)
> Lekha (Hindi)
> Mei-Jia (Chinese)
> Melina (Greek)
> Minena (Russia)

Me too found this troubling. 

[testday-20160812]

Comment 3

[Justin Dolske [:Dolske]]

(Please file new bugs in ::Untriaged unless you're sure they belong in a specific component.)

Comment 4

[:Gijs (he/him)]

Narrate isn't a translation feature. What the different voices let you do is have an intonation and accent appropriate to your language. That some voices may be able to articulate text in different languages might be helpful to make text sound more natural.

As a counterexample, my car has a sattelite navigation (GPS) system that speaks British English. When we were in France/Germany/Italy, it used the 'right' sounds to pronounce placenames (so e.g. "Lille" sounds something like "Leel" would if pronounced by an English speaker, rather than like "Lailuh" or whatever), but it still sounds English and mangles some combinations of letters. If I'd switched the system to use a French or German or Italian voice, placenames would have sounded better (but I wouldn't have known whether to go left/right/whatever if I didn't speak those languages).

So if you go to e.g. http://www.haaretz.co.il/gallery/music/.premium-1.3033420 (an article, if I understand correctly, vaguely related to Taylor Swift), then Carmit will speak Hebrew, and "Bruce (English)" will just read out "Hebrew letter, hebrew letter, hebrew letter" (very helpful...).

It's not clear to me what is the best way forward here. The voices are provided by your OS - we can't teach Bruce Hebrew unless Apple does, and equally we can't stop Carmit from speaking English. "Detecting" the language is difficult and error-prone, but in principle possible. We could try and factor out UI to detect and/or pick a language, and then narrow down the list of voices, or something, so we don't offer a Hebrew voice to read an English article or vice versa. Both of those are hard, though, so I'm picking P5 to reflect that while I would consider patches that improve the UI here, I don't think it's a big priority. If there's something simpler we can do to make the UI clearer here I would consider reprioritizing.

Eitan / reporter, what do you think?

Comment 5

[Eitan Isaacson [:eeejay]]

I'm tempted to close this as wontfix. Seems like a misunderstanding of the feature. The label explicitly says "voice" not "language". So if you select Carmit who is a Hebrew speaker, you will get the English article spoken in a Hebrew accent, not translated to Hebrew.

The only thing that troubles me about this as that two separate people reported this issue, so the feature may not be clear enough, and it may be a problem on our end.

I'll wait for Jules reply to see if this feature makes sense after Gijs's explanation.

Comment 6

[julesmyers2011]

Hi Gijs and Eitan,

So, when I see a name and it has a language next to it that tells me that the voice will speak that language. Some do and some don't which makes understand it's function confusing. As a user I have no idea which languages my OS will speak or not, that is up to Mozilla to deal with.

Since you don't have a solid enough technology to have all the listings speak each language, I say just make a UI change. You might want to write a script that will search the OS (if possible) to see which languages the OS can speak. Then only list those languages/options that will work with the OS. 

Another idea is to just list the languages that all OS will speak. If it's only English and Spanish so be it.

A third idea is to keep the list as is but make it very clear that not all languages may work on their OS. This warning should be at the top of the list in red with a warning symbol so people will read it.

Comment 7

[:Gijs (he/him)]

(In reply to julesmyers2011 from comment #6)
> Hi Gijs and Eitan,
> 
> So, when I see a name and it has a language next to it that tells me that
> the voice will speak that language. Some do and some don't which makes
> understand it's function confusing.

I don't really understand. If you use a Hebrew voice and give it Hebrew text to read, it speaks Hebrew. This is not a question of which voices do/don't work.

> Since you don't have a solid enough technology to have all the listings
> speak each language, I say just make a UI change. You might want to write a
> script that will search the OS (if possible) to see which languages the OS
> can speak. Then only list those languages/options that will work with the
> OS. 

All of these voices are provided by the OS. That is, what you're saying is already what happens. The OS has all the voices we list. You're just using voice X with an article in language Y, and so you still get language Y as output (just with voice X instead of a different one). As I said before, this is not a translation feature. All it does is read an article out loud in the language in which it's written.  Different voices have been optimized for different languages, so you may want to choose a different voice if the article is in a particular language.

Does this make more sense?

Comment 8

[:Gijs (he/him)]

As an alternative question: which voices, according to you, *do* work, when reading https://blog.monotonous.org/2016/03/07/narrate-a-new-feature-in-firefox-nightly/ , that are not labeled as "English" ?

Comment 9

[julesmyers2011]

The main issue is people are going to think that each listing is suppose to be a translation.  When I see a language next to a name, I am expecting to hear a language translation. When I tested the list some of the options are a language translation and some are just a voice accent, which makes it confusing to what it's functionality is suppose to be. 

To me, since I initially expected the list to be all translations the ones that work are the ones that are translations NOT voice accents. But, since you have little control over what translations each OS does I gave you the three suggestions above.

Comment 10

[:Gijs (he/him)]

(In reply to julesmyers2011 from comment #9)
> The main issue is people are going to think that each listing is suppose to
> be a translation.  When I see a language next to a name, I am expecting to
> hear a language translation. When I tested the list some of the options are
> a language translation and some are just a voice accent, which makes it
> confusing to what it's functionality is suppose to be. 
> 
> To me, since I initially expected the list to be all translations the ones
> that work are the ones that are translations NOT voice accents. But, since
> you have little control over what translations each OS does I gave you the
> three suggestions above.

I'm... fairly sure there aren't any voices that provide translations? Which voices do that, on your machine?

Comment 11

[julesmyers2011]

Gijs: I'm just telling you from 15 years of end user experience, people are going to not understand the technical definition of what voice is that you are using. But, do what you want. I don't know how else to explain it to you.

Comment 12

[Jared Wein [:jaws] (please needinfo? me)]

This is a misleading part of the feature that we shouldn't ignore or close as wontfix.

At the very least we can make the option clearer before people use it. Changing the text from "Carmit(Hebrew)" to "Carmit (English with Hebrew accent)" would make a big difference in changing expectations.

Are we able to know what language the voice will speak in? Are we able to know what accent the voice will speak in? What information is missing? I realize it is a very hard problem to know what language the webpage is written in.

Comment 13

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #12)
> This is a misleading part of the feature that we shouldn't ignore or close
> as wontfix.
> 
> At the very least we can make the option clearer before people use it.
> Changing the text from "Carmit(Hebrew)" to "Carmit (English with Hebrew
> accent)" would make a big difference in changing expectations.

But that also makes no sense. Carmit does speak Hebrew when you give it Hebrew text, and most English voices don't, as noted in my earlier comment. It just says "Hebrew character, hebrew character, hebrew character" (I mean, literally, in Enligsh, the words "Hebrew", "character", repeated for as many characters as there are).

The option is already labeled "voice" not "language". We already try to detect the language of the article and pick an appropriate voice (though it seems cld (LanguageDetector.jsm, which we're already using) isn't making us pick hebrew - not sure why at this time, and if you ever change the default from the automatic selection, we will stop doing that, which is a problem for multilingual people). Beyond hiding all the "not applicable" voices (which could be very bad if we mis-identify the language) I just don't really see how we can make this any clearer.

> Are we able to know what language the voice will speak in?

Whatever language the article is written in, if it can. Otherwise, gobbledygook in whatever language the voice is labeled in.

> Are we able to know what accent the voice will speak in?

The language that we currently display ("Hebrew", "Australian English", whatever). Though note that e.g. on Windows, all we get by default is 2 US English voices.

> What information is missing?

It's not so much information that's missing, it's that we give people the option to pick whatever voice they like. Hiding voices would not really make that scenario better - we might as well not give people a choice at all (which would break actual hebrew text even worse, at the current time.

Comment 14

[Eitan Isaacson [:eeejay]]

(In reply to julesmyers2011 from comment #6)
> Hi Gijs and Eitan,
> 
> So, when I see a name and it has a language next to it that tells me that
> the voice will speak that language.

It will speak that language if it is given text in that language. No speech synthesis in any OS translates text in any way whatsoever. 

> Some do and some don't which makes understand it's function confusing.

Whether a voice can read any other language besides its native tongue depends on the voice and the engine.

OSX actually abstracts it in a perfect way: Carmit is an Israeli who reads and speaks fluent Hebrew. She can also read and speak English with a charming Hebrew accent like most Israelis.

Alex, on the other hand is an American who can read and speak in English. Like most Americans, he doesn't understand Hebrew, he only recognizes Hebrew letters and can tell you when he encounters them.

> As a user I have no idea which languages
> my OS will speak or not, that is up to Mozilla to deal with.
> 

We tell you what language each voice can speak.

> Since you don't have a solid enough technology to have all the listings
> speak each language, I say just make a UI change.

The problem is not the technology, but your expectation. Each voice is guaranteed to speak text in its own language. It may manage to read another language badly, but it definitely won't translate any content.

> You might want to write a
> script that will search the OS (if possible) to see which languages the OS
> can speak. Then only list those languages/options that will work with the
> OS. 

We already list which language each voice is meant for.

> A third idea is to keep the list as is but make it very clear that not all
> languages may work on their OS.

All languages work as intended.

> This warning should be at the top of the
> list in red with a warning symbol so people will read it.

If there is a way to make this feature more clear and eliminate any confusion, I'm all for it. 
Reading an article out loud is a non-destructive operation, there is no need for red text and dire warnings.

Comment 15

[Jared Wein [:jaws] (please needinfo? me)]

Eitan, where'd you get that OSX abstraction text from? Can we show that in our dialog too?

Comment 16

[Eitan Isaacson [:eeejay]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #15)
> Eitan, where'd you get that OSX abstraction text from? Can we show that in
> our dialog too?

I made that up :)

What I meant was that each voice is personified with a name that matches the gender and language of that voice. I was just trying to explain what a "voice" is. It is a virtual person who can read text, not a translator.

Comment 17

[Jared Wein [:jaws] (please needinfo? me)]

Are the voices somewhat fixed? If we know what they are and the list is < 100 long, then it wouldn't be that difficult to categorize them.

Comment 18

[Eitan Isaacson [:eeejay]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #17)
> Are the voices somewhat fixed?

Somewhat. People can install new voices on their platform.

> If we know what they are and the list is
> 100 long, then it wouldn't be that difficult to categorize them.

Categorize them how? We have the full bcp47 tag for the voice's supported language. The assumption should be that the voice is exclusive to that language.

Comment 19

[Jared Wein [:jaws] (please needinfo? me)]

The idea would be to categorize them as:
Carmit: ["Hebrew", "English"],
Kanya: ["Thai"],
Lekha: ["Hindi"],
Mei-Jia: ["Chinese"]

And then we could display text next to them like:
Carmet can read Hebrew and also some English
Kanya can read Thai
Lekha can read Hindi
Mei-Jia can read Chinese

Comment 20

[Eitan Isaacson [:eeejay]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #19)
> The idea would be to categorize them as:
> Carmit: ["Hebrew", "English"],

I think each voice should be assumed to speak only *one* language. If it speaks additional languages outside of its intended use, that's great. But I don't think that is something that needs to be communicated to the user, it doesn't add any value.

> Kanya: ["Thai"],
> Lekha: ["Hindi"],
> Mei-Jia: ["Chinese"]
> 
> And then we could display text next to them like:
> Carmet can read Hebrew and also some English
> Kanya can read Thai
> Lekha can read Hindi
> Mei-Jia can read Chinese

Maybe one day the webspeech spec will make the language attribute a list, and then each voice can potentially be multilingual. At that point, we would need to consider having such labels in the UI.

But, back to this bug - I think it is a misunderstanding of the feature that is the trouble. Detailing which languages the voices can speak does not remedy that. The user was surprised Carmit spoke an english article in english and didn't translate it to hebrew.

Comment 21

[Jared Wein [:jaws] (please needinfo? me)]

Okay, my last idea is to then just add some text next to this that says "Reader View can read the text of this page, but cannot translate this page."

If people don't like that then I'm fine with closing this bug as resolved-wontfix (or cantfix but no such option exists). I think we've exhausted all ideas here.

Comment 22

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #21)
> Okay, my last idea is to then just add some text next to this that says
> "Reader View can read the text of this page, but cannot translate this page."

I would prefer not to say "by the way, just so we're clear, X is a thing we can't do". It's negative feedback in UI.

> If people don't like that then I'm fine with closing this bug as
> resolved-wontfix (or cantfix but no such option exists). I think we've
> exhausted all ideas here.

I think we should ask the UX folks. :-)

Markus, do you have other ideas about how we could clarify the distinction between "able to read language X aloud with pronounciation/inflection/accent from language X [potentially regional, eg. british vs. american english]" (which is what the languages next to voice names refer to) and "will translate the article into language X for you", which is how some people are reading it?

Comment 23

[Eitan Isaacson [:eeejay]]

A related issue has come up since this feature has been released. People attempt to use Narrate on content with no supported voices (eg. pressing Narrate on an Italian article with no Italian voices installed on the system). This seems to be a more serious issue because this feature is simply broken on certain locales and platform configurations.

After discussing this with others, I think a good solution would be to hide the Narrate button when there is no intersection between available voices and content language. Further, we can offer a filtered list of voices that would match the language of the content (eg. Only list English voices for English articles). I think that solution would remedy this issue as well.

Markus, what do you think?

Comment 24

[Eitan Isaacson [:eeejay]]

(In reply to :Gijs Kruitbosch from comment #4)
> "Detecting" the language
> is difficult and error-prone, but in principle possible. We could try and
> factor out UI to detect and/or pick a language, and then narrow down the
> list of voices, or something, so we don't offer a Hebrew voice to read an
> English article or vice versa.

Yes, this :)

Comment 25

[Eitan Isaacson [:eeejay]]

Attachment: Bug 1294765 - Refactor content language detection to happen early and always.

Review commit: https://reviewboard.mozilla.org/r/85030/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/85030/

Comment 26

[Eitan Isaacson [:eeejay]]

Attachment: Bug 1294765 - Filter out voices that don't match the content language.

Review commit: https://reviewboard.mozilla.org/r/85032/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/85032/

Comment 27

[:Gijs (he/him)]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

https://reviewboard.mozilla.org/r/85030/#review83746

::: toolkit/components/narrate/NarrateControls.jsm:175
(Diff revision 1)
> -    if (options.length) {
>        options.unshift({
>          label: gStrings.GetStringFromName("defaultvoice"),
>          value: "automatic"
>        });
>        this.voiceSelect.addOptions(options, selectedVoice);

Does it make sense to do this if there are no voices?

::: toolkit/components/narrate/Narrator.jsm:44
(Diff revision 1)
> +    if (win.document.body.classList.contains("loaded")) {
> +      detect();

Isn't this invoked sync from  https://dxr.mozilla.org/mozilla-central/source/toolkit/components/reader/AboutReader.jsm#116-118, which means it's guaranteed to not have loaded content yet?

::: toolkit/components/narrate/test/head.js:28
(Diff revision 1)
> -  ["reader.parse-on-load.enabled", true],
> -  ["media.webspeech.synth.enabled", true],
> -  ["media.webspeech.synth.test", true],
> -  ["narrate.enabled", true],
> -  ["narrate.test", true]
> -];
> +  "reader.parse-on-load.enabled": true,
> +  "media.webspeech.synth.enabled": true,
> +  "media.webspeech.synth.test": true,
> +  "narrate.enabled": true,
> +  "narrate.test": true,
> +  "narrate.voice": null

Nit: add trailing comma

Comment 28

[:Gijs (he/him)]

Comment on attachment 8800011 [details]
Bug 1294765 - Filter out voices that don't match the content language.

https://reviewboard.mozilla.org/r/85032/#review83748

::: toolkit/components/narrate/NarrateControls.jsm:195
(Diff revision 1)
> +  _getVoicePref: function() {
> +    let voicePref = Services.prefs.getCharPref("narrate.voice");
> +    try {
> +      return JSON.parse(voicePref);
> +    } catch (e) {
> +      return { default: voicePref };

I see us writing and falling back to this, but does the language detection ever return "default" as the language? It seems like the code that picks voices also deals with language being the empty string as a fallback for "we don't know what the language is"... so maybe we don't need the explicit default option?

If we do want the explicit default option, I suggest instead of the:
```
      let selectedVoice = voicePrefs[language] || "automatic";
```

We should use:
```
let selectedVoice = voicePrefs[language || "default"];
```

::: toolkit/components/narrate/test/NarrateTestUtils.jsm:122
(Diff revision 1)
> +        if (getter) {
> +          resolve(Services.prefs[getter](pref));
> +        } else {
> -        resolve();
> +          resolve();

Rather than passing the right getter etc., I would just resolve(Preferences.get(pref)) (you might need to Cu.import Preferences.jsm). That'll "just work" for all of these.

::: toolkit/components/narrate/test/browser.ini:6
(Diff revision 1)
>  [DEFAULT]
>  support-files =
>    head.js
>    NarrateTestUtils.jsm
>    moby_dick.html
> +  inferno.html

You can actually add support-files = inferno.html to the individual test that needs it. :-)

::: toolkit/components/narrate/test/head.js:35
(Diff revision 1)
>    "narrate.test": true,
> +  "narrate.filter-voices": false,
>    "narrate.voice": null
>  };
>  
> -function setup(voiceUri = "automatic") {
> +function setup(filterVoices = false, voiceUri = "automatic") {

Wouldn't it be simpler to add the new parameter at the end? Then you don't have to modify existing callsites...

Comment 29

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

https://reviewboard.mozilla.org/r/85030/#review83746

> Isn't this invoked sync from  https://dxr.mozilla.org/mozilla-central/source/toolkit/components/reader/AboutReader.jsm#116-118, which means it's guaranteed to not have loaded content yet?

Yeah.. I thought it would be more robust if we checked anyway. In case the entry point changes and we miss the event.

> Nit: add trailing comma

It's the last entry, there should be no comma. Am I missing something?

Comment 30

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/85030/diff/1-2/

Comment 31

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800011 [details]
Bug 1294765 - Filter out voices that don't match the content language.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/85032/diff/1-2/

Comment 32

[:Gijs (he/him)]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

https://reviewboard.mozilla.org/r/85030/#review84094

::: toolkit/components/narrate/test/head.js:28
(Diff revision 2)
> -  ["reader.parse-on-load.enabled", true],
> -  ["media.webspeech.synth.enabled", true],
> -  ["media.webspeech.synth.test", true],
> -  ["narrate.enabled", true],
> -  ["narrate.test", true]
> -];
> +  "reader.parse-on-load.enabled": true,
> +  "media.webspeech.synth.enabled": true,
> +  "media.webspeech.synth.test": true,
> +  "narrate.enabled": true,
> +  "narrate.test": true,
> +  "narrate.voice": null

Re: trailing comma: It's legal to have the trailing comma even though there are no extra entries following it, and it avoids having to update this line when you add another key:value pair at some point in the future.

Comment 33

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

https://reviewboard.mozilla.org/r/85030/#review84094

> Re: trailing comma: It's legal to have the trailing comma even though there are no extra entries following it, and it avoids having to update this line when you add another key:value pair at some point in the future.

Cool, got it. Had to tweak .eslintrc as well.

Comment 34

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800010 [details]
Bug 1294765 - Refactor content language detection to happen early and always.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/85030/diff/2-3/

Comment 35

[Eitan Isaacson [:eeejay]]

Comment on attachment 8800011 [details]
Bug 1294765 - Filter out voices that don't match the content language.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/85032/diff/2-3/

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/58079f69e082
Refactor content language detection to happen early and always. r=Gijs
https://hg.mozilla.org/integration/autoland/rev/85584e74202b
Filter out voices that don't match the content language. r=Gijs

Comment 37

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/58079f69e082
https://hg.mozilla.org/mozilla-central/rev/85584e74202b

Comment 38

[Markus Jaritz [:markus] (UX)]

(In reply to Eitan Isaacson [:eeejay] from comment #23)
> After discussing this with others, I think a good solution would be to hide
> the Narrate button when there is no intersection between available voices
> and content language. Further, we can offer a filtered list of voices that
> would match the language of the content (eg. Only list English voices for
> English articles). I think that solution would remedy this issue as well.
> 
> Markus, what do you think?

Looks like I am late with feedback. Narrowing down the list sounds great if we can do that well. I however wouldn't go that far and hide the feature completely if no matching voice is available. This might be confusing to people as it will not be obvious to them why the feature is gone in some cases. Instead I would disable the feature, and offer an explanation as to why it is not available for that article: "Not available. No matching voice found for the language of the article."

Comment 39

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Markus Jaritz [:maritz] (UX) from comment #38)
> cases. Instead I would disable the feature, and offer an explanation as to
> why it is not available for that article: "Not available. No matching voice
> found for the language of the article."

Can we use softer language instead? "Narration is disabled since no matching voice was found for the language of the article." (Do we use "article" in other places, or is it "page"?)