1265609 - getUserMedia has no efficient access to Syphon and Spout

Status: NEW

(Core :: WebRTC: Audio/Video, enhancement, P5)

Description

[Jesse Ricke]

getUserMedia has no efficient access to OpenGL textures as applied by the Syphon and Spout frameworks. Browser users want the ability to access any video on their systems. The most efficient way to access video across applications are the OpenGL frameworks called Syphon for Mac and Spout for Windows. There do exist virtual webcam applications that can access these textures but the video must be processed on the CPU. Native support for Syphon and Spout would be far more efficient.

links to frameworks:
http://syphon.v002.info/
http://spout.zeal.co/

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

You can use WebGL and canvases to process the captured video.  You can see some examples of canvas.captureStream() at http://mozilla.github.io/webrtc-landing/

Would that work for your usecases?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

I believe this bug is about adding support to Firefox for having Syphon and Spout input sources in addition to our current paths for getting video into the browser.

Comment 3

[Jesse Ricke]

The canvas.captureStream() example is very interesting but not what I’m pursuing. My use case has visual artists in mind, those working in VJ software and creative coding, though broadcasters are relevant as well. These users build visuals using several applications, often routing the output of one app into another through some GL based framework. I believe efficiently incorporating a browser into this chain would require native support for Syphon in OS X and Spout in Windows. Currently, the bast way to bring a Syphon texture into the browser is through a virtual webcam like CamTwist <http://camtwiststudio.com/> but this requires added CPU processing. Wouldn’t it be better to keep it all on the GPU?

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

This is an enhancement request; I doubt we'll prioritize it high enough to get action, but we're willing to entertain patches.  Alternatively, you can write an extension, or use (as I think you may be) a fake camera driver as an intermediary.

Comment 5

[Jesse Ricke]

CamTwist is the fake camera driver I've used. There's also screen share, which I find has similar performance. I've recently begun sending UDP encoded texture frames over dataChannels, but have not benchmarked it's performance against CamTwist and screen share. While too demanding for most users, texture over DataChannel might be the best solution for my case.

Comment 6

[Bulk Bug Changes for mreavy's org]

Mass change P4->P5 to align with new Mozilla triage process.