Closed Bug 854094 Opened 7 years ago Closed 7 years ago

drawImage doesn't work on remote WebRTC media stream assigned to <video>

Categories

(Core :: WebRTC: Audio/Video, defect)

x86_64
Linux
defect
Not set

Tracking

()

VERIFIED FIXED
mozilla23
Tracking Status
firefox22 --- fixed
firefox23 --- verified

People

(Reporter: michel.gutierrez, Assigned: roc)

Details

(Whiteboard: [WebRTC] [blocking-webrtc-])

Attachments

(3 files)

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130307122903

Steps to reproduce:

Establish a WebRTC video call between 2 browsers instances (Firefox/Firefox from 2 different profiles, or Firefox/Chrome). 
Run local video elements in the browser: 1 with local stream attached + 1 with remote stream attached.
Run an animation loop to calling repeatedly canvas drawImage from the each of the video elements: 
		animate: function() {
			try {
				if(!this.videoConnected) {
					return;
				}
				if ( this.video.readyState === this.video.HAVE_ENOUGH_DATA ) {
					this.videoImageContext.drawImage( this.video, 0, 0, this.videoImage.width, this.videoImage.height );
				}
			} catch(e) {
				console.log("animate error",e);
			}
		},
where:
- this.video is the <video> element
- this.videoImage is a <canvas> element
- this.videoImageContext is a canvas 2D context from this.videoImage



Actual results:

The code works fine for the <video> element displaying the local stream (getUserMedia).
The drawImage call for the <video> element attached to the remote stream generates an exception: 
"""
animate error: [Exception... "Component is not available" nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)" location: "JS frame :: http://xxx/sites/all/modules/jocly/xd-view.js :: Gadget3DVideo<.animate :: line 1172" data: no] { message="Component is not available", result=2147746065, name=
"NS_ERROR_NOT_AVAILABLE", more...}



Expected results:

the drawImage should have worked as it does for the local video copy on Firefox and for both local and remote streams on chromium.
Test case: 

- need 2 computers with cams or 1 computer with 2 cams
- run Chrome (last stable version will be ok) on http://www.jocly.com/jocly/hub?splash=0
- run Firefox nightly on http://www.jocly.com/jocly/hub?splash=0
- in one of the browsers, pick a game in the "Choose you game" list: Chess, Draughts, Brazilian Draughts or Yohoho (the only games that currently integrates video into a 3D view)
- a block appears showing a demo of chosen game, with buttons "Play", "New turnbased game", "New live game"
- click "New live game"
- in the new block that appears, click "Start"
- in the "Waiting live" block of both browsers, a new entry appears for the game offer. This entry should display a camera icon to indicate the game initiator has WebRTC capabilities
- from the other browser, click the game entry
- a block opens, click the "Join" button
- the game starts on both browsers (using the whole screen)
- you are requested to allow the camera. If from the same computer, make sure you pick different cameras
- once both cameras are up, the WebRTC call starts automatically and you soon get both video streams at the top of each game
- in the 3D scene, video objects appear for each of the connected video streams (you may have to use your mouse to turn around the scene to see both videos)
- on Chrome both video objects are displayed, on Firefox only the local video is shown because of this bug

This youtube video display screencasts (made from chrome) of what should be seen: http://www.youtube.com/watch?v=KWuSnwD2gJ8

This article shows the source code for the WebRTC and WebRTC/WebGL integration parts: http://www.jocly.com/node/338

This article shows the overall project context: http://www.jocly.com/node/339
Roc: don't I remember some patches you did around issues about HAVE_ENOUGH_DATA?

Probably blocking- for release but something we'd very much want, since it affects integration with webgl/etc.
Flags: needinfo?(roc)
Whiteboard: [WebRTC] [blocking-webrtc-]
This is broken because no-one's assigning a principal to the remote MediaStream.
Flags: needinfo?(roc)
Something in the WebRTC stack needs to assign a principal to the DOMMediaStream we create for a remote MediaStream. PeerConnectionImpl::CreateRemoteSourceStreamInfo looks like a good place to do that.

The question is, what should that principal be? That's a pretty deep question because it invokes the WebRTC security model.

I'm told that for now calls are "unsecured" and the data should always be available to the receiving page. I'll implement that.
Attached patch fixSplinter Review
Assignee: nobody → roc
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #734962 - Flags: review?(rjesup)
Bonus points if you get a mochitest here. 

Or I'll look into this after this lands to get an automated test here.
Flags: in-testsuite?
Attachment #734962 - Flags: review?(rjesup) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/7dd53c10924e

(In reply to Jason Smith [:jsmith] from comment #7)
> Or I'll look into this after this lands to get an automated test here.

If you could ... I haven't figured out how to write a PeerConnection test yet :-)
https://hg.mozilla.org/mozilla-central/rev/7dd53c10924e
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla23
Keywords: verifyme
Might want to consider uplifting this to Aurora. It will calm app developer complaints that we're getting.
Whiteboard: [WebRTC] [blocking-webrtc-] → [WebRTC] [blocking-webrtc-] [webrtc-uplift]
Comment on attachment 734962 [details] [diff] [review]
fix

[Approval Request Comment]
Bug caused by (feature/regressing bug #): none
User impact if declined: applications using drawImage on WebRTC video won't work
Testing completed (on m-c, etc.): just landed
Risk to taking this patch (and alternatives if risky): very low risk. Seldom used feature, simple patch.
String or IDL/UUID changes made by this patch:none
Attachment #734962 - Flags: approval-mozilla-aurora?
I'll note that this is likely used by any application wanting to mix webrtc and webgl, which isn't an improbable mix.
Comment on attachment 734962 [details] [diff] [review]
fix

The minimal user impact means we won't track, but we will approve for uplift.
Attachment #734962 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Per comment 14, removing webrtc-uplift flag
Whiteboard: [WebRTC] [blocking-webrtc-] [webrtc-uplift] → [WebRTC] [blocking-webrtc-]
Michel - Can you verify this patch fixes the drawImage issue on your WebRTC-based app?
Flags: needinfo?(michel.gutierrez)
Yes :) 

using nightly 23.0a1 (2013-04-11) it now shows both local and remote video streams.

Good job !
Flags: needinfo?(michel.gutierrez)
Verified per comment 17
Status: RESOLVED → VERIFIED
Keywords: verifyme
You need to log in before you can comment on or make changes to this bug.