748835 - Desktop UI implementation for Camera API feature

Status: RESOLVED INCOMPLETE

(Firefox :: Site Permissions, defect)

Description

[Maire Reavy [:mreavy]]

Desktop will need to implement the Camera API UX/UI design.  The UI spec is currently being created (with the Mobile design and still image capture getting higher priority on the schedule) and will be attached when available. 

Feature page: https://wiki.mozilla.org/Platform/Features/Camera_API_-_Phase_2_%28getUserMedia%29

Comment 1

[Asa Dotzler [:asa]]

Gavin, Dolske, Dietrich, this is another Firefox k9o blocker that I mentioned last week at our planning session but didn't follow-up with that same day. Anant is working on back-end.

Comment 2

[Justin Dolske [:Dolske]]

Erp. This fell through the cracks. Dao, can you take a crack at this?

Asa/Maire: Can you clarify what the scope is for this work? It sounds like the first step is a UI like http://people.mozilla.com/~jboriss/specs/first_spec.png (is that current?), used for taking still-images only. And that later on there may be some additional UI (eg for video)?

Presumably we'll need a patch in bug 749873 for actually making this functional, but I suppose we could start creating some of the UI bits until then. Will comment there too.

Comment 3

[Justin Dolske [:Dolske]]

So many meta/tracking bugs... Looks like bug 749889 is actually where the backend work is happening?

Comment 4

[Anant Narayanan [:anant]]

Too many bugs! The specific "downstream" bug for the getUserMedia backend implementation on Desktop is bug 691234. There are separate bugs for Android & B2G back-ends but this bug deals only with the desktop UI I believe.

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

(update from email a few days ago):

> What's the best path for getting familiar with API(s) involved and a build that implements them? Patches? Repo? Imminent m-c landing?

Core backend is currently alder (default branch), with getUserMedia code in a patch.  Depending on khuey's re-review, the core of the backend (media/webrtc/trunk)  may land on m-c (preffed off) early next week.  All other reviews are in place.  Then we just need to land the desktop getUserMedia patch with snapshot capability.   This exists in video form; I don't know if the desktop snapshot code is ready, but with video support snapshot should be easy.

> And who's the right person to talk to about any bugs/questions/issues that might come up as this gets going (wrt to the back-end)?

Myself (webrtc backend), anant and crypt (Suhas from Cisco) for the getUserMedia/snapshot implementation.  mreavy on Product issues. 


(Update since then):  review of makesystem patches is done; landing RSN for media/webrtc/trunk

Comment 6

[Dão Gottwald [:dao]]

Randell, does the backend provide any hooks for the UI? I.e. how is the UI going to be triggered, how should the captured image be handed back?

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

Desktop backend for getUserMedia() needs to throw up UI, and the UI should be different (at least in detail, and maybe significantly) between a snapshot request and a video or audio request.  It will need to have hooks for invoking UI even if the patch does not have them today.

getUserMedia() for audio or video return MediaStreams via callbacks; snapshot returns an nsIFile (IIRC) per the current spec proposal and same as implemented in the Android backend in FF15 and m-c, and in the "fallback" dummy backend.

I believe the current patch for bug 691234 implements video, audio and camera API/snapshot.

Comment 8

[Dão Gottwald [:dao]]

(In reply to Randell Jesup [:jesup] from comment #7)
> Desktop backend for getUserMedia() needs to throw up UI, and the UI should
> be different (at least in detail, and maybe significantly) between a
> snapshot request and a video or audio request.  It will need to have hooks
> for invoking UI even if the patch does not have them today.

Are these hooks in place now?

Comment 9

[Anant Narayanan [:anant]]

(In reply to Dão Gottwald [:dao] from comment #8)
> (In reply to Randell Jesup [:jesup] from comment #7)
> > Desktop backend for getUserMedia() needs to throw up UI, and the UI should
> > be different (at least in detail, and maybe significantly) between a
> > snapshot request and a video or audio request.  It will need to have hooks
> > for invoking UI even if the patch does not have them today.
> 
> Are these hooks in place now?

The code is in place, but we'll need to talk about how to hook the UI in. Are you going to be working on this, Dão? If so, it might be beneficial to have a quick chat on IRC/Vidyo so we can discuss the best way of going about this.

Comment 10

[Anant Narayanan [:anant]]

I have had a chance to connect with Dão in the past week (damn time zones!), so I'll briefly describe what needs to be done here. Hopefully that will unblock him and we can move forward with this bug.

My understanding is that for this bug we will focus only on the still picture API, and that a UI for video and audio will be done in bug 729522. I'm hoping large chucks of the work done in this bug can be reused there.

The still picture API looks like this:
navigator.getUserMedia({picture:true}, function(blob) { ... });

Whenever the API call is made, the user must be presented with a preview window that will allow them to choose between various available cameras. Once a user clicks a picture, we should allow them to crop and resize the snapshot. If a user accepts, it is handed back to the caller as a DOMFile via the success callback.

The code flow is as follows:
- https://mxr.mozilla.org/mozilla-central/source/dom/base/Navigator.cpp#960
- https://mxr.mozilla.org/mozilla-central/source/dom/media/MediaManager.cpp#361 (checking if the window is allowed to open a popup, do we need this for desktop?)
- https://mxr.mozilla.org/mozilla-central/source/dom/media/MediaManager.cpp#241 (aSource->Snapshot is where control is handed over the actual snapshot code, off the main thread)

In https://mxr.mozilla.org/mozilla-central/source/content/media/webrtc/MediaEngineWebRTCVideo.cpp#221 we simply take a snapshot directly from the camera and return a JPEG image. Instead, we should do whatever you guys need in order to instantiate the UI; and then figure out a way for the UI to communicate back the location of the picture (or the raw picture itself in a buffer, whichever is more convenient) to the Snapshot function.

As for the implementation of the preview window itself, I suspect you'll need the following functionality from the backend:
- Enumerate all available cameras, with their names
- Obtain a media stream from any of these cameras to display the preview

Both of the above is currently possible to do if the UI is built in C++, but if you're going to build it in JS, we'll need to talk about how to do it. The best way is probably to re-use getUserMedia itself, with some extra sauce when called from chrome code.

To summarize, open questions for Dão:

- How would you like for the UI to be instantiated? Direct function call? Observer event? What parameters do you need?
- What is the best way to return an image from the UI code to the backend?
- How are you going to build the UI, and what kind of interfaces will you need from the backend in order to build it?

Happy to continue the conversation on this bug, IRC, email, vidyo, hangout, brain-to-brain transfer, pigeons, or messengers :)

Comment 11

[Dão Gottwald [:dao]]

(In reply to Anant Narayanan [:anant] from comment #10)
> - How would you like for the UI to be instantiated? Direct function call?
> Observer event?

A custom DOM event would probably be preferable to an observer event, as the former would automatically be associated with a chrome and content window.

I guess a direct call could happen via a new nsIBrowserDOMWindow or nsIXULBrowserWindow method.

> What parameters do you need?

I'll at least need the content window requesting the image and probably some handle or ID that allows me to hand the data back such that the backend can associate it with the original request.

> - What is the best way to return an image from the UI code to the backend?

XPCOM?

> - How are you going to build the UI, and what kind of interfaces will you
> need from the backend in order to build it?

The UI is going to be implemented in JS (with some markup and CSS). I was going to use getUserMedia in some privileged way, but if a separate XPCOM API is simpler or cleaner from your perspective, that works for me as well.

Comment 12

[Dão Gottwald [:dao]]

Anant, ping? I assume you're going to take care of the backend stuff. Is this going to happen soon? I really just need someone to tell me what APIs to use. Some options are going to be less convenient for me, some more, but anything works as long as I can get and hand back the data.

Comment 13

[Anant Narayanan [:anant]]

Hey Dão, most of the backend work is already in place - what's missing is access to getUserMedia from chrome code and the possibility to enumerate devices and obtain a preview from JS (though this is possible in C++). I'll track that work in bug 752352.

Comment 14

[Jason Smith [:jsmith]]

Talking with Anant, this actually does block. Putting this back on triage radar.

Comment 15

[Maire Reavy [:mreavy]]

Is this bug meant to capture the support needed for still-image gUM on desktop?

Comment 16

[Anant Narayanan [:anant]]

Yup! We can't pref gUM on without this UI because we don't ask permission for still images (the preview window would be implied consent). The other option is to disable {picture:true} support and go ahead with pref'ing on gUM.

Comment 18

[Firefox Bug Husbandry Bot]

Reverting spam.