854094 - drawImage doesn't work on remote WebRTC media stream assigned to <video>

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect)

Description

[Michel Gutierrez]

Attachment: WebRTC call between Chromium (left) and Firefox (right). On Firefox, the remote video is not displayed due to the drawImage issue.

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:19.0) Gecko/20100101 Firefox/19.0
Build ID: 20130307122903

Steps to reproduce:

Establish a WebRTC video call between 2 browsers instances (Firefox/Firefox from 2 different profiles, or Firefox/Chrome). 
Run local video elements in the browser: 1 with local stream attached + 1 with remote stream attached.
Run an animation loop to calling repeatedly canvas drawImage from the each of the video elements: 
		animate: function() {
			try {
				if(!this.videoConnected) {
					return;
				}
				if ( this.video.readyState === this.video.HAVE_ENOUGH_DATA ) {
					this.videoImageContext.drawImage( this.video, 0, 0, this.videoImage.width, this.videoImage.height );
				}
			} catch(e) {
				console.log("animate error",e);
			}
		},
where:
- this.video is the <video> element
- this.videoImage is a <canvas> element
- this.videoImageContext is a canvas 2D context from this.videoImage



Actual results:

The code works fine for the <video> element displaying the local stream (getUserMedia).
The drawImage call for the <video> element attached to the remote stream generates an exception: 
"""
animate error: [Exception... "Component is not available" nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)" location: "JS frame :: http://xxx/sites/all/modules/jocly/xd-view.js :: Gadget3DVideo<.animate :: line 1172" data: no] { message="Component is not available", result=2147746065, name=
"NS_ERROR_NOT_AVAILABLE", more...}



Expected results:

the drawImage should have worked as it does for the local video copy on Firefox and for both local and remote streams on chromium.

Comment 1

[Michel Gutierrez]

Test case: 

- need 2 computers with cams or 1 computer with 2 cams
- run Chrome (last stable version will be ok) on http://www.jocly.com/jocly/hub?splash=0
- run Firefox nightly on http://www.jocly.com/jocly/hub?splash=0
- in one of the browsers, pick a game in the "Choose you game" list: Chess, Draughts, Brazilian Draughts or Yohoho (the only games that currently integrates video into a 3D view)
- a block appears showing a demo of chosen game, with buttons "Play", "New turnbased game", "New live game"
- click "New live game"
- in the new block that appears, click "Start"
- in the "Waiting live" block of both browsers, a new entry appears for the game offer. This entry should display a camera icon to indicate the game initiator has WebRTC capabilities
- from the other browser, click the game entry
- a block opens, click the "Join" button
- the game starts on both browsers (using the whole screen)
- you are requested to allow the camera. If from the same computer, make sure you pick different cameras
- once both cameras are up, the WebRTC call starts automatically and you soon get both video streams at the top of each game
- in the 3D scene, video objects appear for each of the connected video streams (you may have to use your mouse to turn around the scene to see both videos)
- on Chrome both video objects are displayed, on Firefox only the local video is shown because of this bug

This youtube video display screencasts (made from chrome) of what should be seen: http://www.youtube.com/watch?v=KWuSnwD2gJ8

This article shows the source code for the WebRTC and WebRTC/WebGL integration parts: http://www.jocly.com/node/338

This article shows the overall project context: http://www.jocly.com/node/339

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

Roc: don't I remember some patches you did around issues about HAVE_ENOUGH_DATA?

Probably blocking- for release but something we'd very much want, since it affects integration with webgl/etc.

Comment 3

[Michel Gutierrez]

A quicker test case:

Simply visit http://www.jocly.com/jocly/hubquick/yohoho?autolive=1 from both browsers

(or 
- http://www.jocly.com/jocly/hubquick/draughts?autolive=1
- http://www.jocly.com/jocly/hubquick/chess?autolive=1
- http://www.jocly.com/jocly/hubquick/brazilian-draughts?autolive=1)

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

This is broken because no-one's assigning a principal to the remote MediaStream.

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Something in the WebRTC stack needs to assign a principal to the DOMMediaStream we create for a remote MediaStream. PeerConnectionImpl::CreateRemoteSourceStreamInfo looks like a good place to do that.

The question is, what should that principal be? That's a pretty deep question because it invokes the WebRTC security model.

I'm told that for now calls are "unsecured" and the data should always be available to the receiving page. I'll implement that.

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Attachment: fix

Comment 7

[Jason Smith [:jsmith]]

Bonus points if you get a mochitest here. 

Or I'll look into this after this lands to get an automated test here.

Comment 8

[Robert O'Callahan (:roc) (email my personal email if necessary)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/7dd53c10924e

(In reply to Jason Smith [:jsmith] from comment #7)
> Or I'll look into this after this lands to get an automated test here.

If you could ... I haven't figured out how to write a PeerConnection test yet :-)

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/7dd53c10924e

Comment 10

[Jason Smith [:jsmith]]

Might want to consider uplifting this to Aurora. It will calm app developer complaints that we're getting.

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 734962 [details] [diff] [review]
fix

[Approval Request Comment]
Bug caused by (feature/regressing bug #): none
User impact if declined: applications using drawImage on WebRTC video won't work
Testing completed (on m-c, etc.): just landed
Risk to taking this patch (and alternatives if risky): very low risk. Seldom used feature, simple patch.
String or IDL/UUID changes made by this patch:none

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

I'll note that this is likely used by any application wanting to mix webrtc and webgl, which isn't an improbable mix.

Comment 13

[Alex Keybl [:akeybl]]

Comment on attachment 734962 [details] [diff] [review]
fix

The minimal user impact means we won't track, but we will approve for uplift.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/e7b49daa6e28

Comment 15

[Maire Reavy [:mreavy]]

Per comment 14, removing webrtc-uplift flag

Comment 16

[Jason Smith [:jsmith]]

Michel - Can you verify this patch fixes the drawImage issue on your WebRTC-based app?

Comment 17

[Michel Gutierrez]

Yes :) 

using nightly 23.0a1 (2013-04-11) it now shows both local and remote video streams.

Good job !

Comment 18

[Jason Smith [:jsmith]]

Verified per comment 17

Comment 19

[Michel Gutierrez]

Attachment: WebRTC call with WebGL video integration running on nightly 23.0a1 (2013-04-11)