1248054 - <video /> Contents of Adaptive Stream Video Does Not Upload to a Texture

Status: RESOLVED INVALID

(Core :: Audio/Video: Playback, defect)

Description

[jonobr1]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36

Steps to reproduce:

1) Go to either:

Example using canvas via texImage2D in WebGL (with Three.js):
https://dl.dropboxusercontent.com/u/1595444/shaka-player/stream-texture.html

Example using canvas via drawImage directly:
https://dl.dropboxusercontent.com/u/1595444/shaka-player/shaka-canvas.html

2) Click the "Play Button" on the <video /> element.


Actual results:

The <video /> element plays and you can see a moving image. However, below that there is a <canvas /> element. The <canvas /> element remains empty. There are no errors.


Expected results:

I expect to see 2 moving images. One for the <video /> element and one for the <canvas /> element.

Comment 1

[jonobr1]

User Agent tested on examples: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:44.0) Gecko/20100101 Firefox/44.0"

Comment 2

[Michelle Funches - QA]

Verified: The first time I ran this the 2nd element remained empty. The 2nd time the video could be seen.
Version 	47.0a1
Build ID 	20160212030242
User Agent 	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0

Comment 3

[Michelle Funches - QA]

2nd element would not play video at all, could only display/view end credits
 Version 	47.0a1
Build ID 	20160215030213
User Agent 	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:47.0) Gecko/20100101 Firefox/47.0

Comment 4

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Blake - do you have someone who wants to work on this?

Comment 5

[Blake Wu [:bwu][:blakewu]]

After discussing with Daosheng offline, he will help check it.
Thanks!

Comment 6

[Daosheng Mu[:daoshengmu]]

In my investigation, it is a HTMLMediaElement issue neither WebGL nor Canvas. We can check this WebGL video example first, http://threejs.org/examples/webgl_video_panorama_equirectangular.html. It playbacks correctly. 

While using VideoElement, we will check the readyState of this videoElement to be HAVE_ENOUGH_DATA and then go to play. In these two attachment examples, their readyState are always 3(HAVE_FUTURE_DATA). It should be 4(HAVE_ENOUGH_DATA)

I have confirmed this in Google Chrome. They are always HAVE_ENOUGH_DATA in the VideoElement.

Comment 7

[Blake Wu [:bwu][:blakewu]]

Daosheng,
Thanks for your analysis!

Comment 8

[Blake Wu [:bwu][:blakewu]]

kilik,
Hopefully you still have some bandwidth to help this bug.:-)
Please put this in your queue.
Thanks!

Comment 9

[Chris Pearce [:cpearce (Not reading bugmail)]]

Chrome do not implement the HAVE_ENOUGH_DATA readyState as required by the specification, which we we do. So the behaviour difference is causing the test case to work in Chrome but not Firefox.

We switch to HAVE_ENOUGH_DATA when we estimate that at the rate at which we're downloading the file we will have enough data buffered for for playback to reach the end of the video without playback stopping to wait for buffering.

IIRC, Chrome switch to HAVE_ENOUGH_DATA immediately, regardless of the rate of download.

The test case here uses MSE. So the resource isn't downloaded as fast as possible, it goes in fits and starts, and even stops if playback stops. So our readyState HAVE_ENOUGH_DATA calculation isn't mapping well to MediaSource case where the download isn't controlled by us.

Comment 10

[Kilik Kuo [:kikuo] (inactive)]

(In reply to Chris Pearce (:cpearce) from comment #9)
> Chrome do not implement the HAVE_ENOUGH_DATA readyState as required by the
> specification, which we we do. So the behaviour difference is causing the
> test case to work in Chrome but not Firefox.
> 
> We switch to HAVE_ENOUGH_DATA when we estimate that at the rate at which
> we're downloading the file we will have enough data buffered for for
> playback to reach the end of the video without playback stopping to wait for
> buffering.
> 
> IIRC, Chrome switch to HAVE_ENOUGH_DATA immediately, regardless of the rate
> of download.
> 
> The test case here uses MSE. So the resource isn't downloaded as fast as
> possible, it goes in fits and starts, and even stops if playback stops. So
> our readyState HAVE_ENOUGH_DATA calculation isn't mapping well to
> MediaSource case where the download isn't controlled by us.

Checked js code in the test page (https://dl.dropboxusercontent.com/u/1595444/shaka-player/shaka-canvas.html)
...
      var canvas = document.createElement('canvas');
      var ctx = canvas.getContext('2d');
      canvas.width = 640;
      canvas.height = 480;
      document.body.appendChild(canvas);
      var loop = function() {
        requestAnimationFrame(loop);
        if (video.readyState !== video.HAVE_ENOUGH_DATA) {
          return;
        }
        ctx.drawImage(video, 0, 0, 640, 480);
      };
      loop();
...

drawImage() is only called when readyState is HAVE_ENOUGH_DATA.
Modify

if (video.readyState !== video.HAVE_ENOUGH_DATA)

to

if (video.readyState < video.HAVE_CURRENT_DATA)

the video is uploaded to texture.

Canvas Spec only mentioned the behaviour when readyState is HAVE_NOTHING or HAVE_METADATA [1].
[1] https://www.w3.org/TR/2dcontext/#dom-context-2d-drawimage
I don't think this should be a issue, or am I misunderstanding something ?

Comment 11

[Blake Wu [:bwu][:blakewu]]

Kilik, 
Good finding! 
A quote from spec in [1] in comment 10 explains clearly, "If the image argument is an HTMLImageElement object that is not fully decodable, or if the image argument is an HTMLVideoElement object whose readyState attribute is either HAVE_NOTHING or HAVE_METADATA, then return bad and abort these steps."
Therefore, this bug should not be a problem.

Comment 12

[jonobr1]

So great! Thanks so much for the thorough follow up and explanation. I'll make sure to update Three.js so that this is fixed upstream for others as well. Thanks!