1526207 - Calling `drawImage()` fails when drawing a video on android

Status: NEW

(Core :: Graphics: Canvas2D, defect, P2)

Description

[seanplwong]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36

Steps to reproduce:

https://codepen.io/aertmann/pen/mAVaPx?editors=1010

Actual results:

it throws NS_ERROR_NOT_AVAILABLE

Expected results:

it should draw the frame from the video to the canvas

Comment 1

[seanplwong]

I tried this on stable (65), beta (65), nightly (67), all are affected

Comment 2

[Laurentiu Apahidean]

I have managed to reproduce the issue on the latest version of Nightly (67.0a1) and Beta (66.0b6) using the device Huawei MediaPad M3 Lite 10 (Android 7.0), after a video is uploaded no preview is generated.

Based on the description and my comment I will set this issue as new.

Comment 3

[seanplwong]

I discover this originally on Firefox 65, is it necessary to make Firefox 65 as affected as well?

Comment 4

[seanplwong]

*mark

Comment 5

[Andrei Bodea[:andrei]]

Hello Seanplwong, I was able to reproduce it on FF 65 too, based on Comment 3 and my comment I will mark as affected 65 too.

Comment 6

[Jamie Nicol [:jnicol]]

It's the canvas drawImage with a video element bug.

Snorp, is this finally fixable now that bug 1486659 has landed? I gather we now make an extra copy of the image for the compositor? Does that we're allowed to bind the SurfaceTexture for readback, so can implement SurfaceTextureImage::GetAsSourceSurface()?

Comment 7

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Jamie Nicol [:jnicol] from comment #6)

It's the canvas drawImage with a video element bug.

Snorp, is this finally fixable now that bug 1486659 has landed? I gather we now make an extra copy of the image for the compositor? Does that we're allowed to bind the SurfaceTexture for readback, so can implement SurfaceTextureImage::GetAsSourceSurface()?

Hmm. Maybe? We could at least ask the Compositor to get the bytes for us.

Comment 9

[Paavo Happonen]

Issue still exists with 68.
This bug is surprisingly not triggered with every codec.
It is with h264, but not with theora codec, example; http://jsfiddle.net/mL0kzoc7/
Video colorspace doesn't seem to have any effect.

Comment 10

[naktinis]

I can confirm this on Firefox for Android v68.3.0 using this example https://codepen.io/kus/pen/aOqvWP/.

Comment 12

[naktinis]

I can confirm this is still happening on Firefox Preview Nightly for Android 200324 06:00 (Build #20840606). GV: 76.0a1. Can I provide any additional information to help narrow this down?

Comment 13

[Lee Salzman [:lsalzman]]

Jamie, are you able to take a look at this?

Comment 14

[Wessel Kroos]

I can confirm this also happens on Firefox 74 for Windows 10. But I don't have a test case.
I've caught 10 exceptions over the last 30 days. (And more exceptions pre version 74, but that data has been lost.)
Based on that data I can extrapolate the following findings/remarks:

1. 8/10 of the exceptions happened at the start of the video when the video.currentTime is 0.
2. When the video.currentTime is 0, at least one or both of video.mozPaintedFrames or video.mozPresentedFrames is 0. And video.mozParsedFrames and video.mozDecodedFrames are always higher than 0.
3. It seems like 2/10 of the exceptions happened midway when the video.currentTime is not 0.
4. When video.currentTime is 0 then video.paused is false. But when video.currentTime is higher than 0 then video.paused is true.
5. The video.networkState is always 2 (NETWORK_LOADING)
6. video.videoWidth and video.videoHeight are always set to a value.

Regarding video codecs I've seen both the mimetypes video/mp4 (1 time) and video/webm (5 times).
The video/webm mimetype has always the vp9 codec.
The video/mp4 mimetype could have been one of the following codecs (it was not captured exactly):

avc1.42001E, mp4a.40.2
avc1.64001F, mp4a.40.2
avc1.4d401f
avc1.4d4014
avc1.4d401e
avc1.4d400c
avc1.4d400b

Comment 15

[Jamie Nicol [:jnicol]]

This is expected on Android, because of our use of SurfaceTextures. It is not expected on windows, and presumably has a different cause. could you please file a separate bug with that information?

Comment 16

[Wessel Kroos]

I've filed a separate bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1629381

Comment 17

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Repros easily with https://jsfiddle.net/jib1/0dnyfab2/show (I had trouble operating the codepen)

Comment 18

[Jamie Nicol [:jnicol]]

We might be able to fix this by using ImageReader and AHardwareBuffer for video data instead of SurfaceTextures.

Comment 20

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Fyi something funny is going on in bug 1667532 comment 3 where things worked in previous beta, but now not in latest beta. Any idea why this would work in previous beta?

Comment 22

[Brad Isbell]

I can confirm this is still broken on Firefox v93.1.0 on Android.

Testing with: https://www.w3schools.com/tags/tryit.asp?filename=tryhtml5_canvas_drawimage_video

Comment 23

[Sotaro Ikeda [:sotaro]]

gecko now support only WebRender as compositor. Then this bug could be addressed without AHardwareBuffer usage.

Comment 24

[Sotaro Ikeda [:sotaro]]

Attachment: WIP: Bug 1526207 - Support SurfaceTextureImage::AsSurfaceTextureImage()

Comment 25

[Darkspirit]

These tests want to have their HW decoding re-enabled by this bug:
https://hg.mozilla.org/integration/autoland/rev/5976eb8e6be6
https://searchfox.org/mozilla-central/search?q=bug+1526207&path=&case=false&regexp=false

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 27

[Suhaib Mujahid [:suhaib]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.

Comment 28

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 4 duplicates and 5 See Also bugs.
:sotaro, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 31

[Kelsey Gilbert [:jgilbert]]

If true, this is definitely S2, and fixing this would likely fix bug 1655101 and friends.

@jnicol: Is this still a problem on Android? I no longer have an easy way to test.

Comment 32

[Jamie Nicol [:jnicol]]

This is still true. I'm unconvinced it's an S2 though. It's a poor fix for bug 1655101, even if this worked it'd be unbearably slow and we'd want to implement a fast path for that. As for canvas2d, every so often we get a duplicate filed with a codepen that shows the bug, but I'm unaware of any real world content impacted.

Comment 33

[Jim Mathies [:jimm]]

downgrading per Jamie's comment.

Comment 34

[Karl Tomlinson (:karlt)]

drawImage() from a video element was disabled for SurfaceTextureImage (Android) in https://hg.mozilla.org/mozilla-central/rev/00c7ee68e3a8

The NS_ERROR_NOT_AVAILABLE changed to a 'Passed-in image is "broken"' InvalidStateError in https://hg.mozilla.org/mozilla-central/rev/1debf14a4832#l1.12

The failure is silent since https://hg.mozilla.org/mozilla-central/rev/4a58aec917b0#l1.13
(See bug 1876601.)

Comment 35

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1413500

Comment 38

[Alban Rochel]

(In reply to Jamie Nicol [:jnicol] from comment #32)

This is still true. I'm unconvinced it's an S2 though. It's a poor fix for bug 1655101, even if this worked it'd be unbearably slow and we'd want to implement a fast path for that. As for canvas2d, every so often we get a duplicate filed with a codepen that shows the bug, but I'm unaware of any real world content impacted.

Well, I work on a real world AR solution that relies on drawImage as the way to grab an image from a video element. And we will probably drop support for Firefox on Android because of this very specific bug. Strangely, the whole grabbing stuff works fine on live (camera) streams, and even on VP8/WebM sources grabbed by Chrome, but not on VP8/WebM sources grabbed by Firefox itself.

Comment 39

[Raul Bucata]

Verified this issue and still reproduces on Firefox Nightly

Tested with:

Browser / Version: Firefox Nightly 128.0a1 (2016022983-🦎128.0a1-20240525201917🦎)
Operating System: Google Pixel 3 (Android 12) -1080 x 2160 pixels, 18:9 ratio (~443 ppi density)
Operating System: Oppo Find X5 (Android 13) - 1080 x 2400 pixels, 20:9 ratio (~402 ppi density)

Comment 40

[j.jamart]

Any reason this has been closed as Won't fix + obsolete? Could/should it be re-opened?

I'm currently relying on this feature to grab a screenshot from a video but it's not working (on Firefox mobile only.)
Reproduced on Firefox 127.0.2 using Samsung S23 (but not limited to.)

Comment 41

[Jamie Nicol [:jnicol]]

The bug is still open, it's just been marked as wontfix for specific past releases of firefox (as it was not fixed for them)

Comment 42

[R. N. West [:rnwst]]

A while ago I reported this as bug 1872508, which was marked as a duplicate of this report. I provided a test case there and some additional info. It seems only codecs VP9 and AVC/H.264 are affected, but not the more recent AV1. This was tested on Firefox for Android 123.0a1.

Comment 43

[Bob Hood [:bhood]]

Jamie says that Paul's work on 1934009 is something of a predicate to getting this one addressed, as it will make it a lot easier by allowing video decoding in the GPU process on Android.

Comment 44

[Dennis Friedl]

(In reply to R. N. West [:rnwst] from comment #42)

A while ago I reported this as bug 1872508, which was marked as a duplicate of this report. I provided a test case there and some additional info. It seems only codecs VP9 and AVC/H.264 are affected, but not the more recent AV1. This was tested on Firefox for Android 123.0a1.

Hi, does AV1 really work for you? I can't get an AV1 encoded .webm video to work either.

Comment 45

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Bob Hood [:bhood] from comment #43)

Jamie says that Paul's work on 1934009 is something of a predicate to getting this one addressed, as it will make it a lot easier by allowing video decoding in the GPU process on Android.

Bob - bug 1934009 has landed (3 weeks ago). Can we move this forward? It blocks some webcompat buts (and a bunch of others)

Comment 46

[Sotaro Ikeda [:sotaro]]

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #45)

(In reply to Bob Hood [:bhood] from comment #43)

Jamie says that Paul's work on 1934009 is something of a predicate to getting this one addressed, as it will make it a lot easier by allowing video decoding in the GPU process on Android.

Bob - bug 1934009 has landed (3 weeks ago). Can we move this forward? It blocks some webcompat buts (and a bunch of others)

Next, we need to address Bug 1649110.