888177 - Implement ImageCapture API

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P4)

Description

[u459114]

ImageCapture is a W3C spec to enable web client to take a photo and set camera setting.
With getUserMedia/ MediaRecording/ MediaCapture support, we can remove mozCamera API.

Spec:
https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/ImageCapture.html

Comment 1

[Shelly Lin [:shelly]]

Attachment: [WIP] WebIDL of ImageCapture

Implemented the webidl of ImageCapture, but excluded the part of PhotoSettingsOptions for now.

Comment 2

[Shelly Lin [:shelly]]

Attachment: [WIP] WebIDL of ImageCapture

Comment 3

[Shelly Lin [:shelly]]

Update:

-From w3c-
Giri will put up a new version soon.

- From Rob -
http://lists.w3.org/Archives/Public/public-media-capture/2013Mar/0118.html

Comment 4

[Shelly Lin [:shelly]]

Attachment: [WIP] WebIDL and function takePhoto()

A actually workable patch for the function takePhoto(), but the webidl is not yet updated to the latest version of ImageCapture.

Comment 5

[u459114]

Comment on attachment 803497 [details] [diff] [review]
[WIP] WebIDL and function takePhoto()

Review of attachment 803497 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/ImageCapture.cpp
@@ +46,5 @@
> +    {
> +      MOZ_ASSERT(NS_IsMainThread());
> +      mCapture->mWorkerThread->Shutdown();
> +      mCapture->mWorkerThread = nullptr;
> +

Why don't just keep use the same mCaptureThread to execute TakePhotoTask::Run?

@@ +180,5 @@
> +      aResult.Throw(rv);
> +      return;
> +    }
> +    nsRefPtr<TakePhotoTask> event = new TakePhotoTask(this);
> +    mWorkerThread->Dispatch(event, NS_DISPATCH_NORMAL);

Move these two lines out of if (!mWorkerThread) statement?

@@ +204,5 @@
> +  nsTArray<uint8_t> argbImgBuf;
> +  gfxIntSize size;
> +  int stride;
> +  mVideoFrameAdapter->GetARGBImage(&argbImgBuf, &size, &stride);
> +

RGB or ARGB? I don't think the image from camera composed alpha channel

@@ +206,5 @@
> +  int stride;
> +  mVideoFrameAdapter->GetARGBImage(&argbImgBuf, &size, &stride);
> +
> +  nsCOMPtr<imgIEncoder> encoder =
> +      do_CreateInstance("@mozilla.org/image/encoder;2?type=image/png");

Make encoder as a data member of ImageCapture take more memory, but good for performance.

@@ +223,5 @@
> +  uint32_t outputBufLen;
> +  encoder->GetImageBufferUsed(&outputBufLen);
> +  outputBuf.SetLength(outputBufLen);
> +
> +  char* encodedImgBuf;

char* encodedImgBuf = nullptr

Comment 6

[u459114]

Encode image from camera preview or record callback can not get high quality image.

Comment 7

[Florian Bender]

Is there a bug to enable this in unpriviliged content by default?

Comment 8

[Alfredo Yang (:alfredo)]

(In reply to Florian Bender from comment #7)
> Is there a bug to enable this in unpriviliged content by default?

No.
ImageCapture spec could be changed a lot in W3C discussion; for example, using Promise instead of event callback. It'd be better to wait for spec is more mature.

Comment 9

[[:fabrice] Fabrice Desré]

Alfredo, is the new spec the one at https://w3c.github.io/mediacapture-image/ now?

Comment 10

[Alfredo Yang (:alfredo)]

Yes.

Comment 11

[Maire Reavy [:mreavy]]

alfredo - where do you think this spec stands, regarding being ready for implementation?

Comment 12

[Alfredo Yang (:alfredo)]

IMHO, the spec status is not good. Only mozilla has implemented it so far. Questions raised by us are unsolved. So the implementation is pending.

Comment 13

[ddascalescu+github]

Chrome on Android has implemented ImageCapture behind a flag, according to https://bugs.chromium.org/p/chromium/issues/detail?id=603328#c6

Comment 14

[ddascalescu+github]

grabFrame() works in the current Chrome Dev on Linux and Mac.

Comment 15

[beaufort.francois]

Attachment: s/videoStreamTrack/track/g in ImageCapture API

Here's a patch that renames videoStreamTrack to track in the ImageCapture API as discussed in https://github.com/w3c/mediacapture-image/issues/87

Comment 16

[Mounir Lamouri (:mounir)]

François, I think it would be better to create a new bug with this patch. Feel free to CC me and I can do a first round of review for you.

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Comment on attachment 8800544 [details] [diff] [review]
s/videoStreamTrack/track/g in ImageCapture API

Review of attachment 8800544 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/webidl/ImageCapture.webidl
@@ +13,4 @@
>  [Pref="dom.imagecapture.enabled", Constructor(VideoStreamTrack track)]
>  interface ImageCapture : EventTarget {
>    // readonly attribute PhotoSettingsOptions photoSettingsOptions;
> +  readonly attribute VideoStreamTrack track;

Just a quick note, we should align with the spec and use MediaStreamTrack here, and remove VideoStreamTrack entirely [1].

[1] https://dxr.mozilla.org/mozilla-central/source/dom/webidl/VideoStreamTrack.webidl#14

Comment 18

[Bulk Bug Changes for mreavy's org]

Mass change P3->P4 to align with new Mozilla triage process.

Comment 19

[reda.housnialaoui]

Since Chrome 62, ImageCapture is active without the need of a flag.

Are you still waiting for the spec to stabilize?
I think that many of us want to use this feature even in the current state.

Comment 20

[Blake Wu [:bwu][:blakewu]]

Unassigned Alfredo since he is not working on this. If anyone is interested in fixing this bug, please feel free to take it. :-)

Comment 21

[Blake Wu [:bwu][:blakewu]]

(In reply to reda.housnialaoui from comment #19)
> Since Chrome 62, ImageCapture is active without the need of a flag.
> 
> Are you still waiting for the spec to stabilize?
> I think that many of us want to use this feature even in the current state.
Can you tell us what kind of use cases you like to apply this to? 
I believe if there are many strong requests/demands, people in Firefox would reconsider its priority.

Comment 22

[reda.housnialaoui]

Hello blakewu,

We have a computer vision use case requiring to capture a very high resolution photo on an end consumer Camera.
Many cameras are able to capture a video stream using a maximal resolution of 1920*1080.
But some of them are also able to capture a still image at much higher resolution (~3200*2400 for example).

Today in Firefox, we are only able to grab a frame from the video stream. 
Therefore we cannot capture very high quality image using Firefox.

I mentioned the resolution, but I think that abilities to control the zoom, the exposure, the light, etc ... are expected by those who work in computer vision.

Chrome is not yet ok on its windows media foundation implementation (which is the one that I am interested in).
I proposed a patch in https://bugs.chromium.org/p/chromium/issues/detail?id=730068 to fix it.

I didn't take a look at Firefox source code.
Do W3C image capture abstractions already exist in the code?

Comment 23

[Munro Mengjue Chiang [:mchiang] Be Mozillian since 1/19/2018]

It has been implemented but hidden with ther pref dom.imagecapture.enabled
Unfortunately, current implementation only provides video resolution (the same as what you have via grabbing a frame from video)

Comment 24

[reda.housnialaoui]

I looked quickly at the source code.

From what I understand, the first thing to do is to implement takePhoto defined in MediaStreamTrack.h [1] into dom/media/MediaManager.cpp/LocalTrackSource [2].

[1] https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaStreamTrack.h#117
[2] https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaManager.cpp#1080

Comment 25

[reda.housnialaoui]

But I didn't find any abstraction allowing to:
- retrieve photo capabilities
- pass photo settings to the takePhoto

Am I missing something?

Comment 26

[Andreas Pehrson [:pehrsons]]

We used to support it better for B2G (Firefox OS), but it has since been removed from the tree. You can see the latest version at https://hg.mozilla.org/mozilla-central/file/0a5cf829f56e/dom/media/webrtc/MediaEngineGonkVideoSource.cpp.

Comment 27

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

ImageCapture is a camera api, which is WebRTC.

Comment 28

[support]

ImageCapture is really useful take a quick photo and control the camera settings:

https://developers.google.com/web/updates/2016/12/imagecapture

Our HTML5 RDP client also need this to grab frames/control camera settings for the RDP server (redirect user camera to RDP server).

After setting dom.imagecapture.enabled as true, we do can access ImageCapture, and I have this takePhoto() (not promise) method, and this onphoto event.

It should be easy to make this available.

Comment 29

[Sebastian Zartner [:sebo]]

Might be worth mentioning at https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features. Maybe it should be even added to the experimental features page in the settings.

Sebastian

Comment 30

[[e-mail]]

We're building an app which works on openCV which requires user's camera focal length and I found out that this information is saved inside EXIF metadata of the image. But right now only chrome provides this metadata because it supports ImageCapture API.
I would like to request for implementation of this API.

Comment 31

[Will Morgan]

It would be fantastic if we could apply white balance and focus mode settings in Firefox like we can on Chrome for our biometrics app.