add ability to mmap files as ArrayBuffers in JS (packaged apps initially)

NEW
Unassigned

Status

()

6 years ago
3 years ago

People

(Reporter: vlad, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

We could use an ability to mmap files as ArrayBuffers, especially in packaged apps for b2g.  Andreas had the idea to let you mmap any Blob object that was created by FileReader (or whatever the appropriate abstraction is).  Alternatively, if it's simpler, just mmap based on a URL that resolves down to direct filesystem access.

Note that since packaged apps are zips, the requirement would be that anything you want to mmap would need to be stored in the zip file with no compression.

Comment 1

6 years ago
I think we don't need this as an emergency fix any more.  In which case I think we should figure out the desired API here.

Alon, what is the desired API from the viewpoint of a generic emscripten app?

Comment 2

6 years ago
Not sure what the goal here is? Is it to speed up load time?
Speed up load time and reduce memory usage -- if it's a packaged app, everything is on disk already, so we should be able to just mmap and hand it to emscripten instead of needing to read it into memory.

Comment 4

6 years ago
I see. Well, the ability to mmap a file into an ArrayBuffer would be good then. Then that ArrayBuffer would be held in the emscripten emulated filesystem with no actual memory taken. But when the file is used, we would still need to copy it into the main memory typed array (so potentially even better would be to mmap into part of an ArrayBuffer, but that sounds very hard, likely not an option).

Comment 5

6 years ago
(In reply to comment #4)
> I see. Well, the ability to mmap a file into an ArrayBuffer would be good then.
> Then that ArrayBuffer would be held in the emscripten emulated filesystem with
> no actual memory taken. But when the file is used, we would still need to copy
> it into the main memory typed array (so potentially even better would be to
> mmap into part of an ArrayBuffer, but that sounds very hard, likely not an
> option).

Ah yeah that's a good point.  :(  I wonder how much of a win this would be in practice given the additional memcpy that we'd have to go through with such an API.

Comment 6

6 years ago
It would be strictly better than now, where we create a copy of the file in the emscripten filesystem. With mmap that would just be mapped and not actually taken.

I wonder though how feasible it is to do this to a typed array. If it slows down accessing elements compared to a normal typed array it seems problematic. And that seems likely since my guess is that we can't keep using all the normal typed array optimizations with an mmaped one.

Comment 7

6 years ago
(In reply to comment #6)
> It would be strictly better than now, where we create a copy of the file in the
> emscripten filesystem. With mmap that would just be mapped and not actually
> taken.

From a memory usage standpoint or from a performance standpoint?

> I wonder though how feasible it is to do this to a typed array. If it slows
> down accessing elements compared to a normal typed array it seems problematic.
> And that seems likely since my guess is that we can't keep using all the normal
> typed array optimizations with an mmaped one.

Another complication would be supporting all of the existing js engine typed array APIs...
(Assignee)

Updated

6 years ago
Component: DOM: Mozilla Extensions → DOM
Product: Core → Core
:vlad, is this feature ended up being implemented in bug 945152? If they are talking about the same thing we can close this one :)
Flags: needinfo?(vladimir)
Flags: needinfo?(vladimir)
You need to log in before you can comment on or make changes to this bug.