981789 - Stop using workers in OS.File

Status: RESOLVED WONTFIX

(Toolkit Graveyard :: OS.File, enhancement)

Description

[(no longer active)]

Workers are prohibitively expensive to start up and to keep in memory on b2g.  I think for most if not all of what OS.File tries to accomplish, it should not need a background thread, and can just get away with doing non-blocking IO.  If there _is_ a case where we need a background thread, we should be able to use one of the many existing threads that Gecko currently uses for IO purposes.

Note that I'm not very familiar with the OS.File code so I'm not sure how actionable this bug is in its current state.  For one thing, it will probably mean that we need to stop using JS for the implementation if we don't have a good way to access the platform specific non-blocking IO calls, which would be a nice outcome.  Another aspect of this bug could be making the code much simpler.  The amount of complexity in the current implementation really scares me.

Another point to consider would be to work with the ongoing implementation of the Filesystem API which implements many of the same things that OS.File tries to accomplish and exposes that functionality to web content.  That implementation is currently focused on the Firefox OS device storage API, and Yuan is working on it (CCing him here if you have any questions from him about the implementation.)  It seems like we should be able to use much of the same infrastructure, and we won't even need a chrome-specific API to access the same functionality.

Please let me know how we can make this more actionable.

Comment 1

[David Teller [:Yoric] - still alive but not very active]

We are in the process of providing a native implementation for bits and pieces of OS.File (see bug 961665, bug 975702).

The upside is that startup is much, much, much faster. The downside is the explosion in codesize (if we don't count the scaffolding, that's ~50 loc of JS => ~550 loc of C++, xpidl, webidl) and the much more sophisticated underlying memory management. So far, having the code of OS.File in JS has served us well, with dozens of external contributions, I have no idea whether we are going to have nearly as much success with the C++ implementation.

Now, the specs of the Filesystem API are much, much, much weaker than OS.File. In the long term, it might be possible to rewrite a mozFilesystem API with all the features of OS.File we currently use, but that would require quite a lot of work, even without counting all the security issues, so I wouldn't count on it in any foreseeable future.

Comment 2

[(dormant account)]

~50 loc of JS => ~550 loc of C++, xpidl, webidl) <-- a lot of this is spinup effort. Each additional JS->C++ port will be less jarring unless things are going very wrong :)

Comment 3

[David Teller [:Yoric] - still alive but not very active]

Sadly, I'm not sure about that. I counted the other ~600loc as spinup.

Comment 4

[[:fabrice] Fabrice Desré]

Well, we rewrote a few js workers and a protocol handler in c++ for b2g. And yes, you will have to deal with more code - that's not great obviously but the worse part is really the amount of scaffolding/boilerplate that is needed. It's more frustrating than difficult.

Comment 5

[Marco Castelluccio [:marco]]

Since we are going to remove OS.File and we are already using it less and less, I'm not sure it makes sense to keep this open.