1178326 - Allow for dynamically changing speech recognition languages

Status: RESOLVED INVALID

(Core :: Web Speech, defect)

Description

[kdavis]

Allow for dynamically changing speech recognition languages.

Gecko should be modified to allow for dynamically changing the speech recognition language.

Comment 1

[kdavis]

In particular we want to have:

* Initial support for English be shipped on device
* Support for Spanish downloadable

Comment 2

[kdavis]

In more detail, " Allow for dynamically changing speech recognition languages" involves:

1. Modify the Web Speech API implementation to allow for runtime language choice
2. Modify Gecko to download language specific speech recognition models at runtime
3. Host language specific speech recognition models on Marketplace, or other appropriate server

Comment 3

[kdavis]

Now we are adding French too.

Comment 4

[kdavis]

The portion of this bug "Modify Gecko to download language specific speech
recognition models at runtime" will not be necessary as we will piggyback
on this functionality which is already provided by the downloading of lang
packs in Bug 1107341

Comment 5

[kdavis]

The portion of this bug "Host language specific speech recognition models
on Marketplace, or other appropriate server" will not be necessary as we
will piggyback on this functionality which is already provided by hosting
of langpacks in Bug 1107341

Comment 6

[kdavis]

Attachment: Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2 commits for this bug.

This patch enables gecko to download and access speech models that are
now included in language packs.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks.

This commit extends nsIAppsService with the method getSpeechDataResource() 
which is used to obtain speech model resources from such langpacks.

In particular the new method 

  nsIFile getSpeechDataResource(in DOMString language,
                                in DOMString version,
                                in DOMString resourceName);

allows one to obtain a nsIFile referencing a speech model resource for
a given version and resource name.

The try for this commit is running here http://mzl.la/1Vf2rIP

[1] http://bit.ly/1KAEpTB

Comment 7

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

The try for this patch is running here http://mzl.la/1QCWjbU

[1] http://bit.ly/1KAEpTB

Comment 8

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8666083 [details] [diff] [review]
Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

>+  *    speech-data: ["/de/speech-data/your-data-file.res"]
... ok so, the value is an array
(but I think this needs some comment what kind of data speech-data is about.)


>+      if (item["speech-data"]) {
>+        if (item["speech-data"].constructor === Array) {
This checks that the value is Array

>+          debug("Error: languages-provided." + lang + ".speech-data" +
>+                " value must be an array but is " + (typeof item["speech-data"]) +
>+                " : " + item["speech-data"]);
>+          return false;
And if it is, we return early.


>+        }
>+        for (var index = 0; index < item["speech-data"].length; ++index) {
Yet this seems to expect it to be array. So, what?
Should it b !== Array, not ===

>+    for (let lang in this._speechModels) {
>+      if (this._speechModels[lang].from == aApp.manifestURL) {
>+        debug("Speech data unregistered: " + uneval(this._speechModels[lang]));
>+        delete this._speechModels[lang];
>+        Services.obs.notifyObservers(null, "langpack-uninstalled", lang);
Hmm, so what does 'lang' mean here. Is it any language?
PocketSphinxSpeechRecognitionServiceRegistrationService which uses the notification is per process.
How does it know which application we're dealing with here?
Or perhaps I misunderstand the setup here.
Could you explain?
And so we actually need the notification? Couldn't the code here just access PocketSphinxSpeechRecognitionServiceRegistrationService
and call some method on it?
(That would add some flesh to otherwise empty nsISpeechRecognitionServiceRegistrationService ;) )


(stas needs to review most of this patch)

Comment 9

[kdavis]

(In reply to Olli Pettay [:smaug] from comment #8)
> Comment on attachment 8666083 [details] [diff] [review]
> Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService
> 
> >+  *    speech-data: ["/de/speech-data/your-data-file.res"]
> ... ok so, the value is an array
> (but I think this needs some comment what kind of data speech-data is about.)

I can put it there, but it seems odd to as none of the other 10 or so
parameters have any explanation as to what type of data they are and
from the example its should be obvious it's an Array.

> 
> 
> >+      if (item["speech-data"]) {
> >+        if (item["speech-data"].constructor === Array) {
> This checks that the value is Array
> 
> >+          debug("Error: languages-provided." + lang + ".speech-data" +
> >+                " value must be an array but is " + (typeof item["speech-data"]) +
> >+                " : " + item["speech-data"]);
> >+          return false;
> And if it is, we return early.

I agree this is wrong.

The odd thing is that if this caused checkManifest() to fail (which is looks like it did), it didn't stop installation of langpacks with speech models, as installation worked. 

> >+    for (let lang in this._speechModels) {
> >+      if (this._speechModels[lang].from == aApp.manifestURL) {
> >+        debug("Speech data unregistered: " + uneval(this._speechModels[lang]));
> >+        delete this._speechModels[lang];
> >+        Services.obs.notifyObservers(null, "langpack-uninstalled", lang);
> Hmm, so what does 'lang' mean here. Is it any language?

Its a BCP-47 lang code identifying the language and language variation provided by the langpack.

> PocketSphinxSpeechRecognitionServiceRegistrationService which uses the
> notification is per process.

PocketSphinxSpeechRecognitionServiceRegistrationService is created and lives in the main thread. See, for example PocketSphinxSpeechRecognitionServiceRegistrationService::GetInstance() which contains a MOZ_ASSERT(NS_IsMainThread())

> And so we actually need the notification? Couldn't the code here just access
> PocketSphinxSpeechRecognitionServiceRegistrationService
> and call some method on it?

We don't want to block, to keep the experience responsive, as installation may be time consuming.

Comment 10

[Olli Pettay [:smaug][[e-mail]]]

notifyObservers is synchronous, so I don't see how it is any better than a normal method call.

Comment 11

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8666084 [details] [diff] [review]
Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

I wonder if we could somehow simplify the setup here. nfroyd as a xpcom peer might be better person
to look at this.


nsIFactoryWrapper is way too generic name for a class, and concrete classes shouldn't start with nsI*.

Using name mozilla::GetContractID for some speech specific stuff is too generic.


But I'm still trying to understand the whole setup here.
Are (speech) language packs global or per process or per app or what?
I assume they are global.

Comment 12

[kdavis]

Attachment: Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2 commits for this bug.

This patch enables gecko to download and access speech models that are
now included in language packs.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks.

This commit extends nsIAppsService with the method getSpeechDataResource() 
which is used to obtain speech model resources from such langpacks.

In particular the new method 

  nsIFile getSpeechDataResource(in DOMString language,
                                in DOMString version,
                                in DOMString resourceName);

allows one to obtain a nsIFile referencing a speech model resource for
a given version and resource name.

Comment 13

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

Comment 14

[kdavis]

Attachment: Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2 commits for this bug.

This patch enables gecko to download and access speech models that are
now included in language packs.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks.

This commit extends nsIAppsService with the method getSpeechDataResource() 
which is used to obtain speech model resources from such langpacks.

In particular the new method 

  nsIFile getSpeechDataResource(in DOMString language,
                                in DOMString version,
                                in DOMString resourceName);

allows one to obtain a nsIFile referencing a speech model resource for
a given version and resource name.

Comment 15

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

Comment 16

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

Comment 17

[kdavis]

Attachment: Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2: Introduce getSpeechDataResource() to nsIAppsService

Part 1 of 2 commits for this bug.

This patch enables gecko to download and access speech models that are
now included in language packs.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks.

This commit extends nsIAppsService with the method getSpeechDataResource() 
which is used to obtain speech model resources from such langpacks.

In particular the new method 

  nsIFile getSpeechDataResource(in DOMString language,
                                in DOMString version,
                                in DOMString resourceName);

allows one to obtain a nsIFile referencing a speech model resource for
a given version and resource name.

Comment 18

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

Comment 19

[kdavis]

Attachment: Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2: Dynamically register locale specific SpeechRecognitionServices on langpack installs

Part 2 of 2 commits for this bug.

This patch enables gecko to use the speech models that are downloaded
as part of langpacks.

The langpack-builder and langpack manifest were extended[1] to include
speech models in langpacks. 

This patch introduces a service that:

  * Installs a speech model when it is downloaded in a langpack
  * Registers a pocketsphinx service that uses the new speech model
  * Uninstalls the speech model when its langpack is uninstalled
  * Deregisters a pocketsphinx service that used the uninstalled speech model

In addition this patch modifies the pocketsphinx service to deal
with the location of the newly installed speech model, it can not
be in the same location as the pre-installed model as that dir is
not writable at runtime.

Comment 20

[kdavis]

I'm not going to try and rush this in to 2.5. I'll wait and do it calmly

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:anatal, could you have a look please?
For more information, please visit auto_nag documentation.