Open Bug 888177 Opened 11 years ago Updated 10 months ago

Implement ImageCapture API

Categories

(Core :: WebRTC: Audio/Video, defect, P4)

ARM
Gonk (Firefox OS)
defect

Tracking

()

People

(Reporter: u459114, Unassigned)

References

(Depends on 3 open bugs)

Details

(Keywords: dev-doc-needed)

Attachments

(1 file, 3 obsolete files)

ImageCapture is a W3C spec to enable web client to take a photo and set camera setting.
With getUserMedia/ MediaRecording/ MediaCapture support, we can remove mozCamera API.

Spec:
https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/ImageCapture.html
Assignee: nobody → slin
Attached patch [WIP] WebIDL of ImageCapture (obsolete) — Splinter Review
Implemented the webidl of ImageCapture, but excluded the part of PhotoSettingsOptions for now.
Attached patch [WIP] WebIDL of ImageCapture (obsolete) — Splinter Review
Attachment #792693 - Attachment is obsolete: true
Update:

-From w3c-
Giri will put up a new version soon.

- From Rob -
http://lists.w3.org/Archives/Public/public-media-capture/2013Mar/0118.html
A actually workable patch for the function takePhoto(), but the webidl is not yet updated to the latest version of ImageCapture.
Attachment #792697 - Attachment is obsolete: true
Depends on: 916643
Attachment #803497 - Attachment is obsolete: true
Comment on attachment 803497 [details] [diff] [review]
[WIP] WebIDL and function takePhoto()

Review of attachment 803497 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/ImageCapture.cpp
@@ +46,5 @@
> +    {
> +      MOZ_ASSERT(NS_IsMainThread());
> +      mCapture->mWorkerThread->Shutdown();
> +      mCapture->mWorkerThread = nullptr;
> +

Why don't just keep use the same mCaptureThread to execute TakePhotoTask::Run?

@@ +180,5 @@
> +      aResult.Throw(rv);
> +      return;
> +    }
> +    nsRefPtr<TakePhotoTask> event = new TakePhotoTask(this);
> +    mWorkerThread->Dispatch(event, NS_DISPATCH_NORMAL);

Move these two lines out of if (!mWorkerThread) statement?

@@ +204,5 @@
> +  nsTArray<uint8_t> argbImgBuf;
> +  gfxIntSize size;
> +  int stride;
> +  mVideoFrameAdapter->GetARGBImage(&argbImgBuf, &size, &stride);
> +

RGB or ARGB? I don't think the image from camera composed alpha channel

@@ +206,5 @@
> +  int stride;
> +  mVideoFrameAdapter->GetARGBImage(&argbImgBuf, &size, &stride);
> +
> +  nsCOMPtr<imgIEncoder> encoder =
> +      do_CreateInstance("@mozilla.org/image/encoder;2?type=image/png");

Make encoder as a data member of ImageCapture take more memory, but good for performance.

@@ +223,5 @@
> +  uint32_t outputBufLen;
> +  encoder->GetImageBufferUsed(&outputBufLen);
> +  outputBuf.SetLength(outputBufLen);
> +
> +  char* encodedImgBuf;

char* encodedImgBuf = nullptr
Encode image from camera preview or record callback can not get high quality image.
Depends on: 1054905
Assignee: slin → ayang
Depends on: 1064738
Depends on: 1065957
Depends on: 1077299
Depends on: 1069222
No longer depends on: 1069222
Depends on: 1069222
Is there a bug to enable this in unpriviliged content by default?
(In reply to Florian Bender from comment #7)
> Is there a bug to enable this in unpriviliged content by default?

No.
ImageCapture spec could be changed a lot in W3C discussion; for example, using Promise instead of event callback. It'd be better to wait for spec is more mature.
Alfredo, is the new spec the one at https://w3c.github.io/mediacapture-image/ now?
Flags: needinfo?(ayang)
Yes.
Flags: needinfo?(ayang)
Component: Audio/Video → Audio/Video: Recording
alfredo - where do you think this spec stands, regarding being ready for implementation?
Flags: needinfo?(ayang)
Priority: -- → P3
Summary: [meta] Implement ImageCapture API → Implement ImageCapture API
IMHO, the spec status is not good. Only mozilla has implemented it so far. Questions raised by us are unsolved. So the implementation is pending.
Flags: needinfo?(ayang)
Chrome on Android has implemented ImageCapture behind a flag, according to https://bugs.chromium.org/p/chromium/issues/detail?id=603328#c6
grabFrame() works in the current Chrome Dev on Linux and Mac.
Here's a patch that renames videoStreamTrack to track in the ImageCapture API as discussed in https://github.com/w3c/mediacapture-image/issues/87
François, I think it would be better to create a new bug with this patch. Feel free to CC me and I can do a first round of review for you.
Comment on attachment 8800544 [details] [diff] [review]
s/videoStreamTrack/track/g in ImageCapture API

Review of attachment 8800544 [details] [diff] [review]:
-----------------------------------------------------------------

::: dom/webidl/ImageCapture.webidl
@@ +13,4 @@
>  [Pref="dom.imagecapture.enabled", Constructor(VideoStreamTrack track)]
>  interface ImageCapture : EventTarget {
>    // readonly attribute PhotoSettingsOptions photoSettingsOptions;
> +  readonly attribute VideoStreamTrack track;

Just a quick note, we should align with the spec and use MediaStreamTrack here, and remove VideoStreamTrack entirely [1].

[1] https://dxr.mozilla.org/mozilla-central/source/dom/webidl/VideoStreamTrack.webidl#14
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
Since Chrome 62, ImageCapture is active without the need of a flag.

Are you still waiting for the spec to stabilize?
I think that many of us want to use this feature even in the current state.
Unassigned Alfredo since he is not working on this. If anyone is interested in fixing this bug, please feel free to take it. :-)
Assignee: ayang → nobody
(In reply to reda.housnialaoui from comment #19)
> Since Chrome 62, ImageCapture is active without the need of a flag.
> 
> Are you still waiting for the spec to stabilize?
> I think that many of us want to use this feature even in the current state.
Can you tell us what kind of use cases you like to apply this to? 
I believe if there are many strong requests/demands, people in Firefox would reconsider its priority.
Hello blakewu,

We have a computer vision use case requiring to capture a very high resolution photo on an end consumer Camera.
Many cameras are able to capture a video stream using a maximal resolution of 1920*1080.
But some of them are also able to capture a still image at much higher resolution (~3200*2400 for example).

Today in Firefox, we are only able to grab a frame from the video stream. 
Therefore we cannot capture very high quality image using Firefox.

I mentioned the resolution, but I think that abilities to control the zoom, the exposure, the light, etc ... are expected by those who work in computer vision.

Chrome is not yet ok on its windows media foundation implementation (which is the one that I am interested in).
I proposed a patch in https://bugs.chromium.org/p/chromium/issues/detail?id=730068 to fix it.

I didn't take a look at Firefox source code.
Do W3C image capture abstractions already exist in the code?
It has been implemented but hidden with ther pref dom.imagecapture.enabled
Unfortunately, current implementation only provides video resolution (the same as what you have via grabbing a frame from video)
I looked quickly at the source code.

From what I understand, the first thing to do is to implement takePhoto defined in MediaStreamTrack.h [1] into dom/media/MediaManager.cpp/LocalTrackSource [2].

[1] https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaStreamTrack.h#117
[2] https://dxr.mozilla.org/mozilla-central/source/dom/media/MediaManager.cpp#1080
But I didn't find any abstraction allowing to:
- retrieve photo capabilities
- pass photo settings to the takePhoto

Am I missing something?
We used to support it better for B2G (Firefox OS), but it has since been removed from the tree. You can see the latest version at https://hg.mozilla.org/mozilla-central/file/0a5cf829f56e/dom/media/webrtc/MediaEngineGonkVideoSource.cpp.

ImageCapture is a camera api, which is WebRTC.

Component: Audio/Video: Recording → WebRTC: Audio/Video

ImageCapture is really useful take a quick photo and control the camera settings:

https://developers.google.com/web/updates/2016/12/imagecapture

Our HTML5 RDP client also need this to grab frames/control camera settings for the RDP server (redirect user camera to RDP server).

After setting dom.imagecapture.enabled as true, we do can access ImageCapture, and I have this takePhoto() (not promise) method, and this onphoto event.

It should be easy to make this available.

Depends on: 1676192
Depends on: 1686145

Might be worth mentioning at https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Experimental_features. Maybe it should be even added to the experimental features page in the settings.

Sebastian

Keywords: dev-doc-needed

We're building an app which works on openCV which requires user's camera focal length and I found out that this information is saved inside EXIF metadata of the image. But right now only chrome provides this metadata because it supports ImageCapture API.
I would like to request for implementation of this API.

It would be fantastic if we could apply white balance and focus mode settings in Firefox like we can on Chrome for our biometrics app.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.