Closed Bug 951710 Opened 11 years ago Closed 6 years ago

Consider using emscripten'd libjpeg for the gallery

Categories

(Firefox OS Graveyard :: Gaia::Gallery, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::Gallery
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: jrmuizel, Assigned: jrmuizel)

References

Details

Using libjpeg directly gives the ability to do:
- streaming decodes
- low res decodes
- streaming encodes
- control encoding parameters
- embed exif data
- embed color profiles

If were doing all of this work on a worker I don't imagine the performance hit from using JS will be that bad.
Having something like this would make bug 854795 non-blocking for the reduction of the gallery memory usage.
See Also: → 854795
The ultimate goal is to reduce memory consumption.  Somehow I doubt this would help.  Please prove me wrong :)
(In reply to Kyle Huey [:khuey] (khuey@mozilla.com) from comment #2)
> The ultimate goal is to reduce memory consumption.  Somehow I doubt this
> would help.  Please prove me wrong :)

Currently the gallery app causes a bunch of full-size image buffers. i.e to encode the image we need to have the whole image in memory. Using libjpeg directly would let us use a fixed amount of memory and operate on images regardless of their size.
Sure, but emscriptening libjpeg and running that in a worker is going to cost memory too.
Cc-ing Alon that has things to say about that!
You'll need at least a few MB of overhead just to load and compile the worker code, and for the heap, on top of the usual worker overhead. And I worry about performance because I assume libjpeg can use nonportable inline asm, which this approach couldn't use. Also, each image would need to be copied from the emscripten heap into an imageData (and earlier, the input data copied into the heap), which is additional overhead.

So unless using specific libjpeg features gives a very big win over the default way that gecko uses libjpeg, I am skeptical of this. However, it wouldn't be too hard to try, people have ported libjpeg in the past as well as smaller image decompressors like stb_image, this should definitely work, it's just performance I am worried about.
I think I made a pretty strong case that the web platform needs a memory-efficient way to decode big images for display at small size in my WHATWG proposal at http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2013-December/041802.html

But having said that, I'm willing to consider this approach if we can't get something better baked into Gecko in time.  A few megabytes of static overhead for emscripten is better than having background apps killed every time the user zooms in on an image!

I've never used emscripten and from my reading, it is not clear to me how to use it to compile just a library rather than an entire program. So if someone could provide a compiled jpeg decoder, I'd be happy to put it through its paces and see what performance is like. 

The Gaia changes required to adapt to a new image decoding approach are significant, and I wouldn't really want to have to change Gaia once to do things this way only to have to change them all again to do things the way proposed in bug 854795.

So I'd want to wait and see whether 854795 can be implemented successfully before starting to convert the Gallery and other Gaia apps to use libjpeg directly.  But that doesn't mean that we can't start experimenting in this bug at the same time.
If someone wants to try this, this is a good overview of porting a library using emscripten is at,

http://mozakai.blogspot.com/2012/03/howto-port-cc-library-to-javascript.html

and an example of a ported library with useful makefiles etc. to compare against is

https://github.com/kripken/zee.js

Most of the work is in writing a C API for a C++ library (which might not be a problem here), and then a JS API that connects to the C API. Feel free to ping me if any problems are hit.
(In reply to Alon Zakai (:azakai) from comment #8)
> ...
> 
> Most of the work is in writing a C API for a C++ library (which might not be
> a problem here), and then a JS API that connects to the C API. Feel free to
> ping me if any problems are hit.

How convenient that libjpeg is a C library!

Bug 854795 deals with some issues, when we want to display a large image in downscaled resolution.  That is awesome, and something we are looking at doing.  We have other issues with memory consumption, and hit them often when we end up having the whole decoded image in memory (gets especially bad when we stick it into canvas.)  In some of those case, we actually need the full resolution image (e.g, we want to save it), so if our only interface is toBlob, we get in trouble.

The idea is to have a fallback emscripten/libjpeg available, because we can decouple the "we need it now" and "get the standard to add this to the web api and use it".  Slippery slope, to be sure, but let's discuss whether there is any merit to the idea at all.
I'll try to put a prototype together over the next two weeks.
Assignee: nobody → jmuizelaar
I put together something at https://github.com/jrmuizel/cjpeg.js

It will encode a typed array of data to a Blob.

The current approach that it takes to buffer data to the worker is probably quite slow however my goal was to figure out if this approach is workable from a memory point of view.

This code should only use memory proportional to the width of the image instead of the full size.

The emscriptened code is around 400k but around 250k of that is the emscripten runtime which can probably be reduced for this use case.
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.