845685 - libchewing + emscripten as a javascript IME engine

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Description

[Ting-Yuan Huang]

This bug is created to track the status of translating libchewing into javascript by emscripten. Currently libchewing can be compiled and run on PC with emcripten's virtual file system. To work properly on Firefox OS, it's needed to:

1. Take care of IO. Emscripten's virtual fs loads all files into memory and eats a lot of memory.
2. Besides vfs's overhead, make sure it'll fit in FirefoxOS's limited amount of memory.
3. Make sure the performance is acceptable.

Comment 1

[Ting-Yuan Huang]

Attachment: WIP -- Steps and files to compile and integrate libchewing into gaia.

Attached are steps and files to compile and integrate libchewing into gaia.

Currently, it takes about 3s to bootstrap and requires about 17.xMB memory on unagi. Refinements, especially to memory usages, are going on.

A more comprehensive document will be released later.

Comment 2

[Justin Lebar (not reading bugmail)]

What is the target memory usage of this module?

Comment 3

[Ting-Yuan Huang]

(In reply to Justin Lebar [:jlebar] from comment #2)
> What is the target memory usage of this module?

I tried to run a native version of libchewing and monitor the memory usage. It looks like that, without exposing the capability of MMU to javascript, what we can do best is a little more than 9452KB: 

VmPeak:     9452 kB
VmHWM:      1212 kB
VmStk:       148 kB

I tunned the emulated native heap to 10MB (it's 16MB by default) and the emulated stack to 256KB (5MB by default, within the heap). Now it requires about 11.10MB which is quite close to the lower bound.

I'm trying to measure other javascript based IMEs and will show the results later.

Comment 4

[Justin Lebar (not reading bugmail)]

OOC what are the RSS numbers there?

Comment 5

[Ting-Yuan Huang]

It's VmHWM, the peak of RSS.

Comment 6

[Justin Lebar (not reading bugmail)]

I see.  I think this is a pretty compelling case for exposing decommit semantics to emscripten, then.

Comment 7

[Ting-Yuan Huang]

Attachment: Steps and files to compile and integrate libchewing into gaia.

* Rename emchewing.sh to example.sh.
* Fine tune stack and total memory size.
* Added testjs.sh for running built-in tests in libchewing.

Comment 8

[Ting-Yuan Huang]

Here is the memory usage (right after gc) of another IME, Zhuyin:

28.25 MB (37.53%) -- window(app://keyboard.gaiamobile.org/index.html#show=312)

and it spent 4s on initialization.

Zhuyin is manually rewritten from its Objective-C version, McBopomofo. The databases are translated into JSON beforehand and loaded into JavaScript arrays at runtime.

Comment 9

[Tim Guan-tin Chien [:timdream] (please needinfo)]

(In reply to Ting-Yuan Huang from comment #8)
> Zhuyin is manually rewritten from its Objective-C version, McBopomofo. The
> databases are translated into JSON beforehand and loaded into JavaScript
> arrays at runtime.

The detail about this statement is incorrect -- JSZhuyin is manually written by me, with a lousy algorithm unrelated to McBopomofo. I just copied the database from that because there are no other phrases database available under compatible license.

(libchewing & libchewing-data are licensed under LGPL, incompatible with Gaia)

Comment 10

[Ting-Yuan Huang]

Attachment: Steps and files to compile and integrate libchewing into gaia.

The patch to emscripten in the first step in the example is no longer needed; It's already landed.

Comment 11

[BMO Automation]

Firefox OS is not being worked on