842644 - add ability to mmap files as ArrayBuffers in JS (packaged apps initially)

Status: NEW

(Core :: DOM: Core & HTML, defect)

Description

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

We could use an ability to mmap files as ArrayBuffers, especially in packaged apps for b2g.  Andreas had the idea to let you mmap any Blob object that was created by FileReader (or whatever the appropriate abstraction is).  Alternatively, if it's simpler, just mmap based on a URL that resolves down to direct filesystem access.

Note that since packaged apps are zips, the requirement would be that anything you want to mmap would need to be stored in the zip file with no compression.

Comment 1

[(no longer active)]

I think we don't need this as an emergency fix any more.  In which case I think we should figure out the desired API here.

Alon, what is the desired API from the viewpoint of a generic emscripten app?

Comment 2

[Alon Zakai (:azakai)]

Not sure what the goal here is? Is it to speed up load time?

Comment 3

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Speed up load time and reduce memory usage -- if it's a packaged app, everything is on disk already, so we should be able to just mmap and hand it to emscripten instead of needing to read it into memory.

Comment 4

[Alon Zakai (:azakai)]

I see. Well, the ability to mmap a file into an ArrayBuffer would be good then. Then that ArrayBuffer would be held in the emscripten emulated filesystem with no actual memory taken. But when the file is used, we would still need to copy it into the main memory typed array (so potentially even better would be to mmap into part of an ArrayBuffer, but that sounds very hard, likely not an option).

Comment 5

[(no longer active)]

(In reply to comment #4)
> I see. Well, the ability to mmap a file into an ArrayBuffer would be good then.
> Then that ArrayBuffer would be held in the emscripten emulated filesystem with
> no actual memory taken. But when the file is used, we would still need to copy
> it into the main memory typed array (so potentially even better would be to
> mmap into part of an ArrayBuffer, but that sounds very hard, likely not an
> option).

Ah yeah that's a good point.  :(  I wonder how much of a win this would be in practice given the additional memcpy that we'd have to go through with such an API.

Comment 6

[Alon Zakai (:azakai)]

It would be strictly better than now, where we create a copy of the file in the emscripten filesystem. With mmap that would just be mapped and not actually taken.

I wonder though how feasible it is to do this to a typed array. If it slows down accessing elements compared to a normal typed array it seems problematic. And that seems likely since my guess is that we can't keep using all the normal typed array optimizations with an mmaped one.

Comment 7

[(no longer active)]

(In reply to comment #6)
> It would be strictly better than now, where we create a copy of the file in the
> emscripten filesystem. With mmap that would just be mapped and not actually
> taken.

From a memory usage standpoint or from a performance standpoint?

> I wonder though how feasible it is to do this to a typed array. If it slows
> down accessing elements compared to a normal typed array it seems problematic.
> And that seems likely since my guess is that we can't keep using all the normal
> typed array optimizations with an mmaped one.

Another complication would be supporting all of the existing js engine typed array APIs...

Comment 8

[Tim Guan-tin Chien [:timdream] (please needinfo)]

:vlad, is this feature ended up being implemented in bug 945152? If they are talking about the same thing we can close this one :)