1178288 - Have the WebSpeech API support downloadable language packs (Piggyback on marketplace)

Status: RESOLVED WONTFIX

(Core :: Web Speech, defect)

Description

[kdavis]

Have the WebSpeech API support downloadable language packs (Piggyback on marketplace)

Comment 1

[kdavis]

Language pack spec https://github.com/stasm/spec/blob/master/webapp-langpacks.markdown

Comment 2

[kdavis]

Current implementation of downloadable language packs is done under Bug 1107341

Comment 3

[kdavis]

Current language pack builder https://github.com/l20n/langpack-builder

Comment 4

[Staś Małolepszy :stas]

The Gecko part was bug 1108096.  Would it be possible to use getLocalizationResource [1] via the Webapps:GetLocalizationResource message to fetch the model files from the langpack?

It looks like pocket sphinx looks for those files in specific places [2], so that might be hard?

Alternatively we could put those files someplace special when a langpack is installed [3].

[1] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#146
[2] http://mxr.mozilla.org/mozilla-central/source/media/pocketsphinx/src/pocketsphinx.c#120
[3] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#331

Comment 5

[kdavis]

(In reply to Staś Małolepszy :stas from comment #4)
> The Gecko part was bug 1108096.  Would it be possible to use
> getLocalizationResource [1] via the Webapps:GetLocalizationResource message
> to fetch the model files from the langpack?
> 
> It looks like pocket sphinx looks for those files in specific places [2], so
> that might be hard?
> 
> Alternatively we could put those files someplace special when a langpack is
> installed [3].
> 
> [1] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#146
> [2]
> http://mxr.mozilla.org/mozilla-central/source/media/pocketsphinx/src/
> pocketsphinx.c#120
> [3] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#331

I think the third option, "put those files someplace special when a langpack is
installed" is the easiest from the speech recognition point of view.

Is this hard from your side of things?

Comment 6

[kdavis]

(In reply to kdavis from comment #5)
> I think the third option, "put those files someplace special when a
> langpack is installed" is the easiest from the speech recognition
> point of view.
> 
> Is this hard from your side of things?

If this is hard from your side of things, then we can change where we
grab the models from. Either way it shouldn't be too bad.

Comment 7

[Staś Małolepszy :stas]

I guess it really depends what exactly is "someplace special" and how SR can access it :)  Would those need to be files?  Database?  An API call?

Comment 8

[kdavis]

It's all files and the current location is...

root@flame:/ # ll /system/b2g/models/                                          
drwxr-xr-x root     root              2015-07-16 07:30 dict
drwxr-xr-x root     root              2015-07-16 07:30 en-US
root@flame:/ # ll /system/b2g/models/dict/                                     
-rw-r--r-- root     root      3245613 2015-07-15 01:45 en-US.dic
root@flame:/ # ll /system/b2g/models/en-US/                                    
feat.params         means               noisedict           transition_matrices
mdef                mixture_weights     sendump             variances

Comment 9

[kdavis]

But, we can move this location if needed.

Comment 10

[Zibi Braniecki [:zbraniecki][:gandalf]]

Is it at all possible to feed those files to the library programmatically in form of Blobs?

Comment 11

[kdavis]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #10)
> Is it at all possible to feed those files to the library programmatically in
> form of Blobs?

Just to clarify what does here "library" mean here? Pocketsphinx?

Comment 12

[Staś Małolepszy :stas]

(In reply to kdavis from comment #11)
> Just to clarify what does here "library" mean here? Pocketsphinx?

Yes.

I'm not familiar with Gecko's C/C++, but would it be possible to use the nsIServiceManager [1] to get nsIMessageBroadcaster [2] and listen to Webapps:GetLocalizationResource messages [3]?

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIServiceManager#getService%28%29
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsIMessageBroadcaster
[3] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#67

Comment 13

[kdavis]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #10)
> Is it at all possible to feed those files to the library programmatically in
> form of Blobs?

Assuming library=pocketsphinx as in Comment 12, the answer is "no" for the current implementation.

These files are not all binary, for example en-US.dic is text. Thus, there would need to be some
glue layer added if they were converted to binary.

Why do you need these files to be binary? I would think from the language pack side of things
they should be uninterpreted data the internal details of which are immaterial.

Comment 14

[kdavis]

(In reply to Staś Małolepszy :stas from comment #12)
> I'm not familiar with Gecko's C/C++, but would it be possible to use the
> nsIServiceManager [1] to get nsIMessageBroadcaster [2] and listen to
> Webapps:GetLocalizationResource messages [3]?
> 
> [1]
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/
> Interface/nsIServiceManager#getService%28%29
> [2]
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/
> Interface/nsIMessageBroadcaster
> [3] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#67

I haven't tried this explicitly, but from my knowledge of the various components
I would say "yes" this is possible.

Comment 16

[kdavis]

(In reply to Staś Małolepszy :stas from comment #12)
> (In reply to kdavis from comment #11)
> > Just to clarify what does here "library" mean here? Pocketsphinx?
> 
> Yes.
> 
> I'm not familiar with Gecko's C/C++, but would it be possible to use the
> nsIServiceManager [1] to get nsIMessageBroadcaster [2] and listen to
> Webapps:GetLocalizationResource messages [3]?
> 
> [1]
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/
> Interface/nsIServiceManager#getService%28%29
> [2]
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/
> Interface/nsIMessageBroadcaster
> [3] http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm#67

Just looked at this in a bit more detail and I think this path of attack will not work.

The idea is fine, but the problem is that one needs to register a nsIMessageListener[1] with the nsIMessageBroadcaster[2], more specifically with its super-class nsIMessageListenerManager[3], to
obtain "Webapps:GetLocalizationResource" messages. However, the interface nsIMessageListener[1]
is specific to JavaScript not C++[4]. So, one can't register a nsIMessageListener[1] from C++.

I'd guess there has to be some way around this, as listening for messages is too simple a use
case not to be available from C++.

[1] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L173
[2] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L296
[3] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L206
[4] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L176

Comment 17

[kdavis]

Staś in light of comment 16, and further research that confirms
nsIMessageListenerManager messages are only consumable in JS[1],
is there any other channel through which C++ can consume such
"Webapps:GetLocalizationResource" messages?

[1] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L15

Comment 18

[Staś Małolepszy :stas]

I don't know Gecko well enough to answer that, unfortunately.

Fabrice, can you chime in here, please?  We're trying to find a way for speech recognition to use Langpacks' getLocalizationResource to retrieve extra speech models.  See my comment 12 and Kelly's comment 16 for some context.  Is there anyway to use this API from C++ code?

Comment 19

[[:fabrice] Fabrice Desré]

If we extract the models from the langpacks to store them on disk (it's a bit sad to basically double the disk space usage), they will have to be somewhere under /data, since /system where the default ones are is readonly. Can pocketsphinx support that?

I think we need to understand first what are the limitations from pocketsphinx. If we can't feed it blobs, the simplest thing is likely to extract them at install time, and not mess with getLocalizationResource. If we can feed pocketsphinx blobs (they can be representing text files), it's easy to expose getLocalizationResource through an xpcom service. Does that need to work both in the parent and child processes?

Comment 20

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #19)
> If we extract the models from the langpacks to store them on disk (it's a
> bit sad to basically double the disk space usage), they will have to be
> somewhere under /data, since /system where the default ones are is readonly.
> Can pocketsphinx support that?

We can have pocketsphinx read the models from whatever directory is easiest
from the langpack installation point of view.
 
> I think we need to understand first what are the limitations from
> pocketsphinx. If we can't feed it blobs, the simplest thing is likely to
> extract them at install time, and not mess with getLocalizationResource.

Pocketsphinx needs to be fed files with a directory structure described in
comment 8.

The actual location of the directory /system/b2g/models can be anywhere we
find is most convenient from the langpack installation point of view.

> If we can feed pocketsphinx blobs (they can be representing text files), it's
> easy to expose getLocalizationResource through an xpcom service.

I am not sure why blobs are relevant here. Why do we need these files to be binary?
I would think from the langpack side of things they should be uninterpreted data
the internal details of which are immaterial.

Could someone please clarify this for me?

> Does that need to work both in the parent and child processes?

My plan was to have a service that listened for Webapps:GetLocalizationResource messages
and when such a message was fired for a language that had an associated speech model,
then this service would use the files on disk installed by the langpack installation
process to create a speech recognition service for the new language. (So, the message
processing is all in the process of the service.)


A quick summary of what the WebSpeech API code requires right now...

Assuming a service model as above, if the models are installed using the current format,
the WebSpeech API code only needs to know three things to create a new speech recognition
service

1. Where the analog of the directory /system/b2g/models is for models installed
   by the langpack installation procedure. (I assume this is known at compile time)
2. When a langpack with a speech recognition model is installed
3. What is the language (en-US, es-ES, zh...) of the newly installed langpack

Comment 21

[Zibi Braniecki [:zbraniecki][:gandalf]]

I think our questions about Blobs can be interpreted in a different way.

Is it possible for pocketsphinx to not get a directory path with localization files, but to feed it with a text file already.

So, in a naive way, instead of this:

PocketSphinx.setL10nPath('./b2g/data/locales/{{locale}}/file.data');

Give it this:

var fileData = // data obtained from Langpack API

PocketSphinx.addLanguageData('en-US', fileData);

-----------

The reason why we are asking is because Langpack API stores langpack data and maintains retrieving them. So in an ideal world we would like to use it for retrieving files for PocketSphinx instead of duplicating data from the langpack zip package to the hard drive so that PocketSphinx can read it back from the hard drive.

Comment 22

[kdavis]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #21)
> I think our questions about Blobs can be interpreted in a different way.
> 
> Is it possible for pocketsphinx to not get a directory path with
> localization files, but to feed it with a text file already.

No this is not currently possible.

> So, in a naive way, instead of this:
> 
> PocketSphinx.setL10nPath('./b2g/data/locales/{{locale}}/file.data');
> 
> Give it this:
> 
> var fileData = // data obtained from Langpack API
> 
> PocketSphinx.addLanguageData('en-US', fileData);

All the pockesphinx code is C++ the JS API for what Pocketsphinx provides
is given by the Web Speech API's SpeechRecognition Interface[1]. This API
has no methods of the form PocketSphinx.setL10nPath() or addLanguageData()
and we do not wish to introduce such as it runs counter to the spirit and
letter of the Web Speech API specification.

Also introducing a JS API such as setL10nPath() or addLanguageData() causes
the JS API to expose details of a particular Web Speech API implementation,
which is something we wish to avoid as it nullifies the purpose for the
existence of the Web Speech API standard.

[1] https://dvcs.w3.org/hg/speech-api/raw-file/tip/webspeechapi.html#speechreco-section

> The reason why we are asking is because Langpack API stores langpack data
> and maintains retrieving them. So in an ideal world we would like to use it
> for retrieving files for PocketSphinx instead of duplicating data from the
> langpack zip package to the hard drive so that PocketSphinx can read it back
> from the hard drive.

I think our miss communication here is with respect to the target programming
language. Everything related to pocketsphinx and loading models occurs in C++.

The C++ simply has to know...

1. Where the analog of the directory /system/b2g/models is for models installed
   by the langpack installation procedure. (I assume this is known at compile time)
2. When a langpack with a speech recognition model is installed
3. What is the language (en-US, es-ES, zh...) of the newly installed langpack

Comment 23

[Staś Małolepszy :stas]

(In reply to kdavis from comment #22)
> The C++ simply has to know...
> 
> 1. Where the analog of the directory /system/b2g/models is for models
> installed by the langpack installation procedure. (I assume this is known at
> compile time)

This is the tricky part :) One possibility is that the files aren't installed anywhere special and just stay in the package of the langpack (/system/b2g/webapps/fr.langpack.gaiamobile.org/application.zip) and can be accessed via a programmatic API. Hence our questions about the possibility of using the "Webapps:GetLocalizationResource" API in C++.

Comment 24

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to kdavis from comment #22)
> I think our miss communication here is with respect to the target programming
> language. Everything related to pocketsphinx and loading models occurs in
> C++.

I don't think that programming language matters that much here. We could expose the same Langpack API to C++ if that would allow us to feed pocketsphinx with data instead of having to store the data on the hard drive and provide paths to it.

I assume that you are saying that the pocketsphinx API cannot work with that and needs to load files from hard drive. That's kind of limiting because it forces us to store them in a particular place a duplicate them (right now they are stored in .zip files that langpacks are).

Comment 25

[kdavis]

Awesome! Not really awesome, but now I think I see the pain points, which is progress.

Staś, with the current C++ support one can not listen for "Webapps:GetLocalizationResource"
messages, see comment 16. So this can not work with the current configuration.

Zibi, I think I was hoping for a simple solution which didn't involve exposing a JS API in
C++ as this seems like either too much work or something I thought was already there :-) But,
it's looking more and more like this is the only way out.

Staś and Zibi, the zip question can be critical. I had the false impression that the lang
pack machine didn't modify files in any way. However, this is not the case. The question
is how to deal with this problem.

We can't duplicate the files for the speech models they are simply too big. (The English
model, for example, is about 20MB.) For example, see Bug 1174041, when I landed the English
model into gecko, the standard partition had to be changed as there was not enough space
to allow shallow flashes on flames. So, I think duplication is out.

So either

1. The speech models stay in the zip
2. The speech models are pulled out of the zip and placed in some special directory

I'd guess that if we are forced to expose the JS API in C++, then we could use solution 1.
However, my gut says this is too much work for such a simple problem. 

Is there an easier way?

Comment 26

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to kdavis from comment #25)
> Zibi, I think I was hoping for a simple solution which didn't involve
> exposing a JS API in
> C++ as this seems like either too much work or something I thought was
> already there :-) But,
> it's looking more and more like this is the only way out.

I don't think it's as much work as you are worried. But I'll let Fabrice speak, he wrote the API :)

> Staś and Zibi, the zip question can be critical. I had the false impression
> that the lang pack machine didn't modify files in any way. However, this is not the case.
> The question is how to deal with this problem.

Langpacks are .zip files installed via webapp installation mechanism and treated a little bit specially (but not really that much).

That means that files that are part of the langpack live in that .zip file and are accessible primarily via Gecko APIs. I can imagine you may just access the zip from your own code, but I'd say that sounds like a bad design.

> We can't duplicate the files for the speech models they are simply too big.
> (The English
> model, for example, is about 20MB.) For example, see Bug 1174041, when I
> landed the English
> model into gecko, the standard partition had to be changed as there was not
> enough space
> to allow shallow flashes on flames. So, I think duplication is out.

How well the speech models compress?

> So either
> 
> 1. The speech models stay in the zip
> 2. The speech models are pulled out of the zip and placed in some special
> directory
> 
> I'd guess that if we are forced to expose the JS API in C++, then we could
> use solution 1.
> However, my gut says this is too much work for such a simple problem. 

If we can use Gecko API to provide you access to those models the same way as other components access files from packaged app zip files, that would probably be the best option.

Is that possible?

Comment 27

[kdavis]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #26)
> (In reply to kdavis from comment #25)
> ...
> I can imagine you may just access the zip from your own code, but I'd say
> that sounds like a bad design.

I agree.

> How well the speech models compress?

localhost:recognition kdavis$ du -ch models
3.1M	models/dict
 18M	models/en-US
 21M	models
 21M	total
localhost:recognition kdavis$ zip -r models.zip models
  adding: models/ (stored 0%)
  ....
  adding: models/en-US/variances (deflated 11%)
localhost:recognition kdavis$ ls -lah models.zip 
-rw-r--r--  1 kdavis  staff    12M Aug  5 08:56 models.zip

> > So either
> > 
> > 1. The speech models stay in the zip
> > 2. The speech models are pulled out of the zip and placed in some special directory
> > 
> > I'd guess that if we are forced to expose the JS API in C++, then we could
> > use solution 1.
> > However, my gut says this is too much work for such a simple problem. 
> 
> If we can use Gecko API to provide you access to those models the same way
> as other components access files from packaged app zip files, that would
> probably be the best option.

It's looking like this is the "best" option.

> Is that possible?

This is definitely possible. However, what is "the same way as other components
access files from packaged app zip files" in C++?

Comment 28

[Staś Małolepszy :stas]

(In reply to kdavis from comment #25)

> Staś, with the current C++ support one can not listen for
> "Webapps:GetLocalizationResource"
> messages, see comment 16. So this can not work with the current
> configuration.

Fabrice, can we somehow work around this?

Comment 29

[kdavis]

(In reply to Staś Małolepszy :stas from comment #28)
> (In reply to kdavis from comment #25)
> 
> > Staś, with the current C++ support one can not listen for
> > "Webapps:GetLocalizationResource"
> > messages, see comment 16. So this can not work with the current
> > configuration.
> 
> Fabrice, can we somehow work around this?

We can definitely "work around" this.

We can extend nsIMessageListener[1] to have it's receiveMessage() method
take a parameter which has the following properties

   target:    %the target of the message. Either an element owning
               the message manager, or message manager itself if no
               element owns it%
   name:      %message name%,
   sync:      %true or false%.
   data:      %structured clone of the sent message data%,
   json:      %same as .data, deprecated%,
   objects:   %named table of jsvals/objects, or null%
   principal: %principal for the window app

mirroring the JS API, then have this method called by the
same code that calls the JS receiveMessage() method.

From a grep of gecko it doesn't seem as if any C++ code
implements nsIMessageManager, which is not surprising as
nsIMessageManager is only for JS; so, nothing should
break, famous last words.

The main thing which gives me pause is it looks like the C++
code that needs to be added to create the parameter to send
to the receiveMessage() method will be ugly and somewhat com-
plicated.

[1] https://github.com/mozilla/gecko-dev/blob/master/dom/base/nsIMessageManager.idl#L173

Comment 30

[[:fabrice] Fabrice Desré]

Touching the message listener directly is not the right way to do that. We can expose whatever we need on the nsIAppsService xpcom for instance, and that will make it easily available from c++.

The questions I see is more:
- does that need to be available both in the parent and child processes? (I guess yes).
- are you fine with an asynchronous load of these resources? Making that synchronous will be hard and/or ugly.

Comment 31

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #30)
> Touching the message listener directly is not the right way to do that. We
> can expose whatever we need on the nsIAppsService xpcom for instance, and
> that will make it easily available from c++.

OK.

However I don't see how this method is able to notify the C++ code when a
langpack is downloaded.

From what I see of nsIAppsService, C++ can, given the manifestURL or the
local id of the app, obtain information about the app. But, I do not see
how using nsIAppsService C++ can be notified when a langpack is downloaded.

If the manifestURL, or local id, of the langpack app for a given language
XX were known at compile time, then the C++ could periodically pole for
the langpack for XX. However, the constant polling seems less than ideal,
and I am not sure the manifestURL and/or local id of the langpack app for
a given language is known at compile time.

> NEED INFO Fabrice: Is the above polling and compile time knowledge of
>                    a manifestURL or local id what you have in mind?

> NEED INFO Stas/Zibi: Are the interfaces nsIAppsService and mozIApplication
>                      sufficient to allow one to get localized resources? If
>                      so, how do you see them being used? (To me they look as
>                      if they must be enhanced, but maybe there is something I
>                      don't see.)


> The questions I see is more:
> - does that need to be available both in the parent and child processes? (I
> guess yes).

This really now depends upon the solution we come up with.

Initially I was hoping to make a service singleton that was always present and was
notified when a new langpack was installed. It would then create a new speech
recognition service that corresponded to the new language.

However, it looks as if this plan will have to change dependent upon our answers to
the questions here. So, I can't truthfully say at this point.

> - are you fine with an asynchronous load of these resources? Making that
> synchronous will be hard and/or ugly.

I am fine with an asynchronous load. However, the asynchronous load would
have to occur "not so long" after the langpack is downloaded.

For example, a user goes through the standard process

Settings > Language > Get More Languages > Marketplace

to download a langpack. They then, rightfully so, think they can use the new
speech recognition language right away. If the asynchronous load takes too
long, we will cause users much consternation.

Comment 32

[[:fabrice] Fabrice Desré]

(In reply to kdavis from comment #31)
> (In reply to [:fabrice] Fabrice Desré from comment #30)
> > Touching the message listener directly is not the right way to do that. We
> > can expose whatever we need on the nsIAppsService xpcom for instance, and
> > that will make it easily available from c++.
> 
> OK.
> 
> However I don't see how this method is able to notify the C++ code when a
> langpack is downloaded.
> 
> From what I see of nsIAppsService, C++ can, given the manifestURL or the
> local id of the app, obtain information about the app. But, I do not see
> how using nsIAppsService C++ can be notified when a langpack is downloaded.
> 
> If the manifestURL, or local id, of the langpack app for a given language
> XX were known at compile time, then the C++ could periodically pole for
> the langpack for XX. However, the constant polling seems less than ideal,
> and I am not sure the manifestURL and/or local id of the langpack app for
> a given language is known at compile time.
> 
> > NEED INFO Fabrice: Is the above polling and compile time knowledge of
> >                    a manifestURL or local id what you have in mind?

No, we don't know anything at compile time. Expect langpacks to come from anywhere ;)
The simplest thing to do is very likely to have the langpack installer dispatch an observer notification that the webspeech component will listen too. Something like notifyObservers(null, "langpack-installed", manifestURL), and one for uninstalling langpacks too. Then the webspeech component will be free to do whatever it needs.
Since this is a private gecko API, we can even send the .zip path in addition or instead of the manifestURL, or any interface that suits your needs.

> > NEED INFO Stas/Zibi: Are the interfaces nsIAppsService and mozIApplication
> >                      sufficient to allow one to get localized resources? If
> >                      so, how do you see them being used? (To me they look as
> >                      if they must be enhanced, but maybe there is something I
> >                      don't see.)
> 
> 
> > The questions I see is more:
> > - does that need to be available both in the parent and child processes? (I
> > guess yes).
> 
> This really now depends upon the solution we come up with.
> 
> Initially I was hoping to make a service singleton that was always present
> and was
> notified when a new langpack was installed. It would then create a new speech
> recognition service that corresponded to the new language.

Right, but is that service instantiated in every process, or only in the parent?


> > - are you fine with an asynchronous load of these resources? Making that
> > synchronous will be hard and/or ugly.
> 
> I am fine with an asynchronous load. However, the asynchronous load would
> have to occur "not so long" after the langpack is downloaded.
> 
> For example, a user goes through the standard process
> 
> Settings > Language > Get More Languages > Marketplace
> 
> to download a langpack. They then, rightfully so, think they can use the new
> speech recognition language right away. If the asynchronous load takes too
> long, we will cause users much consternation.

By async, I'm just talking whether we needed blocking i/o, not some long lasting download process. Once the langpack is installed everything is available in the zip.

I hope this is clearing things up a bit. That should not be so hard ;)

Comment 33

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #32)
> (In reply to kdavis from comment #31)
> > (In reply to [:fabrice] Fabrice Desré from comment #30)
> 
> The simplest thing to do is very likely to have the langpack installer
> dispatch an observer notification that the webspeech component will listen
> too. Something like notifyObservers(null, "langpack-installed",
> manifestURL), and one for uninstalling langpacks too.

Sounds fine to me.
 
> Right, but is that service instantiated in every process, or only in the parent?

To clarify, by service I mean that registered in kLayoutCIDs of nsLayoutModule.cpp,
so it will run in whatever processes these are run in. I assume this is the parent.

> I hope this is clearing things up a bit.

Yes it is, thanks!

> That should not be so hard ;)

I agree, deciding what to do is actually harder than doing it :-)

Comment 34

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #32)
> (In reply to kdavis from comment #31)
> > (In reply to [:fabrice] Fabrice Desré from comment #30)
> 
> The simplest thing to do is very likely to have the langpack installer
> dispatch an observer notification that the webspeech component will listen
> too. Something like notifyObservers(null, "langpack-installed",
> manifestURL), and one for uninstalling langpacks too. Then the webspeech
> component will be free to do whatever it needs.
> 
> Since this is a private gecko API, we can even send the .zip path in
> addition or instead of the manifestURL, or any interface that suits your
> needs.

Stas/Zibi let's see how far we can get with the manifestURL, I do not want
to have to know the internal organization of your zip files.

This brings me back to my needinfo....

Stas or Zibi: Are the interfaces nsIAppsService and mozIApplication sufficient
to allow one to get the speech model data?

Comment 35

[[:fabrice] Fabrice Desré]

(In reply to kdavis from comment #33)
> (In reply to [:fabrice] Fabrice Desré from comment #32)
> > (In reply to kdavis from comment #31)
> > > (In reply to [:fabrice] Fabrice Desré from comment #30)
> > 
> > The simplest thing to do is very likely to have the langpack installer
> > dispatch an observer notification that the webspeech component will listen
> > too. Something like notifyObservers(null, "langpack-installed",
> > manifestURL), and one for uninstalling langpacks too.
> 
> Sounds fine to me.
>  
> > Right, but is that service instantiated in every process, or only in the parent?
> 
> To clarify, by service I mean that registered in kLayoutCIDs of
> nsLayoutModule.cpp,
> so it will run in whatever processes these are run in. I assume this is the
> parent.

Nope, this runs in all processes. There's no explicit remoting anywhere in the webspeech api implementation?

Comment 36

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to kdavis from comment #34)
> (In reply to [:fabrice] Fabrice Desré from comment #32)
> > (In reply to kdavis from comment #31)
> > > (In reply to [:fabrice] Fabrice Desré from comment #30)
> > 
> > The simplest thing to do is very likely to have the langpack installer
> > dispatch an observer notification that the webspeech component will listen
> > too. Something like notifyObservers(null, "langpack-installed",
> > manifestURL), and one for uninstalling langpacks too. Then the webspeech
> > component will be free to do whatever it needs.
> > 
> > Since this is a private gecko API, we can even send the .zip path in
> > addition or instead of the manifestURL, or any interface that suits your
> > needs.
> 
> Stas/Zibi let's see how far we can get with the manifestURL, I do not want
> to have to know the internal organization of your zip files.

Our manifest is described here: https://github.com/stasm/spec/blob/master/webapp-langpacks.markdown

The simplest and most logical thing from our perspective would be to extend 'languages-provided' into sth like this:

"languages-provided": {
  "de": {
    "name": "Deutsch",
    "revision": 201411051234,
    "apps": {
      "app://calendar.gaiamobile.org/manifest.webapp": "/de/apps/calendar",
      "app://email.gaiamobile.org/manifest.webapp": "/de/apps/email"
    },
    "speech-data": "/de/speech-data/your-data-file.res"
  },
}

or an array if you need multiple, or even an object if you need them named.

Then, we need to either provide you a specific C++ API like:

navigator.mozApps.getSpeechDataResources('de', '2.5');

or extend the current one:

navigator.mozApps.getLocalizationResource('de', '2.5', '/de/speech-data/your-data-file.res', 'text', 'speech-data');

(in that scenario the new parameter to getLocalizationResource by default is 'apps')

Fabrice, am I on the right track?

Comment 37

[[:fabrice] Fabrice Desré]

The manifest addition looks fine to me.

c++ doesn't have easy access to navigator.mozApps.getLocalizationResource() so I'm not convinced we need to change our content facing api though. I think adding getSpeechDataResources() to nsIAppsService should be enough.

Comment 38

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #35)
> (In reply to kdavis from comment #33)
> > (In reply to [:fabrice] Fabrice Desré from comment #32)
> > > (In reply to kdavis from comment #31)
> > > > (In reply to [:fabrice] Fabrice Desré from comment #30)
> ...
> There's no explicit remoting anywhere in the webspeech api implementation?

As far as I know, it is only in the speech synthesis, as opposed to speech
recognition, implementation.

Comment 39

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #37)
> ...
> I think adding getSpeechDataResources() to nsIAppsService should be enough.

I can take a swing at this, but I'll need support from Stas and Zibi

Comment 40

[kdavis]

Currently, as far as I can see, one can not deregister a language pack.
If this is the case, will it continue to be the case for FxOS 2.5?

Comment 41

[Staś Małolepszy :stas]

You can uninstall a langpack just like you uninstall other webapps, via Settings > Application Permissions.

Comment 42

[kdavis]

(In reply to Staś Małolepszy :stas from comment #41)
> You can uninstall a langpack just like you uninstall other webapps, via
> Settings > Application Permissions.

Thanks!

I thought, for consistency's sake, it would be under the same menu you
install language packs with

            Settings > Language > "No 'Uninstall Languages' here."

I'd go so far as to say this is a bug, as no normal user will know that
langpacks are implemented as apps.

Comment 43

[kdavis]

Before I start implementation I thought floating a trial balloon for the new method
of nsIAppsService.idl would be wise.

+  /**
+   * Returns requested speech data resource as a nsIInputStream
+   */
+  nsIInputStream getSpeechDataResource(in DOMString language,
+                                       in DOMString version,
+                                       in DOMString resourceName);

where "resourceName" would have a value like "/de/speech-data/your-data-file.res"
from the example above.

Suggestions and comments are more than welcome.

Comment 44

[kdavis]

Another trial balloon for the new method of nsIAppsService.idl:

+  /**
+   * Returns requested speech data resource as a nsIFile
+   */
+  nsIFile getSpeechDataResource(in DOMString language,
+                                in DOMString version,
+                                in DOMString resourceName);

where "resourceName" would have a value like "/de/speech-data/your-data-file.res"

Both this and the trial balloon of comment 43 are motivated by the nsIZipReader.idl
which is able to open a zip entry as either a nsIInputStream or nsIFile.

Again, suggestions and comments are more than welcome.

Comment 45

[[:fabrice] Fabrice Desré]

resourceName should just be "your-data-file.res" no? "de" comes from language and "speech-data" could be constant.

Other than that, I don't mind returning nsIInputStream or nsIFile or a Blob, but anything doing i/o needs to be async - only nsIInputStream seems ok to do synchronously. Whatever best fit the consumer is ok.

Comment 46

[kdavis]

(In reply to Zibi Braniecki [:gandalf][:zibi] from comment #36)
> (In reply to kdavis from comment #34)
> > (In reply to [:fabrice] Fabrice Desré from comment #32)
> > > (In reply to kdavis from comment #31)
> > > > (In reply to [:fabrice] Fabrice Desré from comment #30)
> > > 
> > > The simplest thing to do is very likely to have the langpack installer
> > > dispatch an observer notification that the webspeech component will listen
> > > too. Something like notifyObservers(null, "langpack-installed",
> > > manifestURL), and one for uninstalling langpacks too. Then the webspeech
> > > component will be free to do whatever it needs.
> > > 
> > > Since this is a private gecko API, we can even send the .zip path in
> > > addition or instead of the manifestURL, or any interface that suits your
> > > needs.
> > 
> > Stas/Zibi let's see how far we can get with the manifestURL, I do not want
> > to have to know the internal organization of your zip files.
> 
> Our manifest is described here:
> https://github.com/stasm/spec/blob/master/webapp-langpacks.markdown
> 
> The simplest and most logical thing from our perspective would be to extend
> 'languages-provided' into sth like this:
> 
> "languages-provided": {
>   "de": {
>     "name": "Deutsch",
>     "revision": 201411051234,
>     "apps": {
>       "app://calendar.gaiamobile.org/manifest.webapp": "/de/apps/calendar",
>       "app://email.gaiamobile.org/manifest.webapp": "/de/apps/email"
>     },
>     "speech-data": "/de/speech-data/your-data-file.res"
>   },
> }
> 
> or an array if you need multiple, or even an object if you need them named.

I'll need an array, as I have about 10 files I'll need.

Is anyone willing to take-on the JS side of this task? I'll take on the C++
side of things, and we can meet in the middle.

Comment 47

[Staś Małolepszy :stas]

(In reply to kdavis from comment #46)
 
> Is anyone willing to take-on the JS side of this task? I'll take on the C++
> side of things, and we can meet in the middle.

What's involved on the JS side?  Do you mean the checkManifest and register methods of http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm ?

Comment 48

[kdavis]

(In reply to Staś Małolepszy :stas from comment #47)
> (In reply to kdavis from comment #46)
>  
> > Is anyone willing to take-on the JS side of this task? I'll take on the C++
> > side of things, and we can meet in the middle.
> 
> What's involved on the JS side?  Do you mean the checkManifest and register
> methods of
> http://mxr.mozilla.org/mozilla-central/source/dom/apps/Langpacks.jsm ?

Of the things I know about:

* Notify observers with "langpack-installed" topic when a langpack is installed
* Notify observers with "langpack-uninstalled" topic when a langpack is uninstalled
* Modify langpacks to include the "speech-data" array parameter

likely some other things too that I don't know of, as I don't know the
details of the JS langpack implementation.

Comment 49

[Gregor Wagner [:gwagner]]

This should block the feature bug and not the 2.5 release.

Comment 50

[kdavis]

(In reply to [:fabrice] Fabrice Desré from comment #19)
> If we extract the models from the langpacks to store them on disk (it's a
> bit sad to basically double the disk space usage), they will have to be
> somewhere under /data, since /system where the default ones are is readonly.
> Can pocketsphinx support that?

Dumb implementation question, but how does one obtain /data in a platform
independent manner?

I would have guessed it was something like...

nsCOMPtr<nsIFile> tmpFile;
NS_GetSpecialDirectory(NS_DATA_DIR, getter_AddRefs(tmpFile));

where NS_DATA_DIR is a fictitious property name for the /data directory.

Unfortunately, there is no define in nsDirectoryServiceDefs.h for NS_DATA_DIR,
or any other property name for the /data directory.

Comment 51

[[:fabrice] Fabrice Desré]

You should not try to store them directly under some /data directory. I would use ProfD/speech instead.

Comment 52

[Staś Małolepszy :stas]

Kelly -- if Gaia is configured on buildtime to include translations for one of the speech-enabled languages, would we still ask the user to download the langpack to enable speech recognition, or should we build Gaia with the right model baked in?

Comment 53

[kdavis]

(In reply to Staś Małolepszy :stas from comment #52)
> Kelly -- if Gaia is configured on buildtime to include translations for one
> of the speech-enabled languages, would we still ask the user to download the
> langpack to enable speech recognition, or should we build Gaia with the
> right model baked in?

I would assume that we should "build Gaia with the right model baked in". Then
the user doesn't have to worry about downloading anything. But, I do not know
the official word on how this should work.

Who is responsible for backing in the langpacks into gaia?

Comment 54

[Staś Małolepszy :stas]

That would be me and Zibi.  We don't currently have a way to build langpacks on buildtime.  Instead, we copy localization files into each app's zip file.

How often do we expect the models to change?

Comment 55

[Francesco Lodolo [:flod]]

This looks like a FxOS specific bug. Reopen if it's not the case.