968707 - Cavas.ToBlob is slow on B2G

Status: RESOLVED WONTFIX

(Core :: Graphics: Canvas2D, defect)

Description

[u459114]

There is HW jpg encoder on most platforms, but we didn't really take advantage of it. On B2G, we still use software jpg encoder. 

Use case:
Gallery calls Canvas.ToBlob to encode a image after image process, which may take more then 3 seconds before encoded image callback.

By leverage to power of HW encode, we can get better encoding performance and save power consumption.

Comment 1

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

This bug is trying to use HW encoder for encoding acceleration on mobile devices.
Try to improve |toBlob| in HTMLCanvasElement.
See http://mxr.mozilla.org/mozilla-central/source/dom/webidl/HTMLCanvasElement.webidl#31

Comment 2

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

Do we have an idea of what the API for the hw encoder looks like?  We can definitely do this.

Comment 3

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

Some findings so far:
1. It takes about 0.078s for GetImageBuffer and takes 1.329742s for ProcessImageData(compress to JPEG) on 5 Mega-pixel pictures in Unagi phone.
2. In Android Nexus 4, the stock photo edit APP also takes around 2 seconds for 8 Mega-pixel pictures.
3. There are no suitable API in camera HAL for jpeg encoder. Be aware that the format in camera is YUV and we use RGB in HTMLCanvasElement. No HAL define in Android means that it might be a mess even in the same vendor if we directly use the way in Camera BSP. Also we need a effort for colour converting from RGB to YUV.

Both cases(Unagi and Nexus 4) are use software jpeg encoder.

It comes 3 solutions so far.
1. Define our own HAL APIs for JPEG hw encoder/decoder.
2. Try OpenMax.
3. Improve libjpeg-turbo by multi-threading or OpenCL.

I am checking OpenMax now. Will keep update.

Comment 4

[Chia-hung Tai [:ctai :ctai_mozilla :cht]]

About solution 2, OpenMax, it looks like not workable now.
I found the test codes for OpenMAX decoder in frameworks/base/media/tests/omxjpegdecoder.
It basically uses an OMXClient to get an decoder from image source. I feed a jpeg to this binary, but I get a null decoder from Nexus 4 and Unagi.
The format of image source is MEDIA_MIMETYPE_IMAGE_JPEG. John Lin tell me I can get the runtime media codecs list by executing "adb shell cat /system/etc/media_codecs.xml" in Nexus 4. After check the file, it doesn't have the value MEDIA_MIMETYPE_IMAGE_JPEG("image/jpeg").
Looks like the support of OpenMax jpeg decoder/encoder is not very common.
So I will try solution 3, improve libjpeg-turbo by multi-threading or OpenCL.

Comment 5

[Chiajung Hung [:chiajung]]

ToBlob involves a Snapshot call and a SurfaceToPackedBGRA, both of them basically memcpy/color convert the buffer in SharedSurface(skiaGL)/gfxSurface(cairo). I think we can avoid at least 1 memcpy here. 

Snapshot may not needed, too. Since all these function should be called on main thread, if we can make sure the draw operation is committed and finished, we should be able to find a way to extract the underlying buffer for encode directly. 

Though it may somewhat offtopic here, since this bug seems handling the lower level jpeg encoding problem.

Comment 6

[Sylvestre Ledru [:Sylvestre]]

Mass closing as we are no longer working on b2g/firefox os.

Comment 7

[Sylvestre Ledru [:Sylvestre]]

Mass closing as we are no longer working on b2g/firefox os.