1320017 - Wrong Orientation of the screenshot snapped to Canvas from HTML5 video

Status: RESOLVED DUPLICATE of bug 1423850

(Core :: Graphics: Canvas2D, defect)

Description

[longman]

Attachment: Снимок экрана от 2016-11-24 09-44-27.png

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36

Steps to reproduce:

example sources: http://jsfiddle.net/longman/umjkv6eg/
description:
1. I have video with Rotation 180-degree. Firefox's player shows it with correct automatically fixed Rotation (as 0-degree)
2. i try to take screenshot from it to HTML5 Canvas




Actual results:

i got screenshot with different Rotation than in video player, so:
1. in player i see correct automatically fixed 0-degree Rotation
2. in canvas i see screenshot with WRONG 180-degree Rotation (upside down)


Expected results:

i must to see in canvas screenshot with the same Rotation, which i see in video player (in my example both must be 0-degree). 

FYI. you can run jsfiddle-example in Chrome and it works correctly, Safari works as Firefox in wrong way. please make behavior the same as in Chrome, it is more logically applicated. thanks!

Comment 1

[Vincent Liu[:vliu]]

I tried to dump mBufferProvider before [1] by the following WIP to observe snapshotted data.

   RefPtr<DrawTarget> target = mBufferProvider->BorrowDrawTarget(IntRect(0, 0, mBounds.width, mBounds.height));
  if (target) {
           DumpString = gfxUtils::GetAsDataURI(target);
           printf_stderr("%s", DumpString.get());
  }

At this point, this snapshotted data stored in mBufferProvider had orientation issue.

[1]: https://dxr.mozilla.org/mozilla-central/source/gfx/layers/client/ClientCanvasLayer.cpp#128

Comment 2

[longman]

@Vincent
Thanks for reply. i am not sure that i fully understood your thoughts, because i am not c++ specialist). 

cite:
> At this point, this snapshotted data stored in mBufferProvider had orientation issue.
does it means that exists some issue in sources and developers potentially can fix it? and in result we will get the same rotation in video playing and in canvas with screenshot? 
thanks!

Comment 3

[Vincent Liu[:vliu]]

(In reply to longman from comment #2)
> @Vincent
> Thanks for reply. i am not sure that i fully understood your thoughts,
> because i am not c++ specialist). 
> 
> cite:
> > At this point, this snapshotted data stored in mBufferProvider had orientation issue.
> does it means that exists some issue in sources and developers potentially
> can fix it? and in result we will get the same rotation in video playing and
> in canvas with screenshot? 
> thanks!

Currently I tried to dump image data from different aspect of code flow to make sure what causes flip happens. After did the same thing in [1] and [2], I think image was flip from the source side. I'd discussed with media team and they said it is reasonable in this case. Anyway, I will keep tracking to find out the possible way to solve it.

[1]: https://dxr.mozilla.org/mozilla-central/source/gfx/layers/ipc/ImageBridgeChild.cpp#485
[2]: https://dxr.mozilla.org/mozilla-central/source/dom/canvas/CanvasRenderingContext2D.cpp#4816

Comment 4

[Vincent Liu[:vliu]]

More clues in [3] to shows the source was flip. VideoInfo::Rotation in HTMLVideoElement was kDegree_180.


[3]: https://dxr.mozilla.org/mozilla-central/source/dom/html/HTMLVideoElement.cpp#67

Comment 5

[longman]

the reason of the flip i understood and described in my first post. there is more abstract main question: how do you think (and your team) what result we much to have in canvas image when source video-image was flipped to 180 degree. please note that logically result in canvas and in video player must be the same: so we must have in both correct 0-degree rotation OR both wrong 180-degree rotation. :)
what way team will choose?

Comment 6

[longman]

i have tried to find some JS-solution to detect degree-info from canvas or from video file but it is unsuccessfull, we don't know what way choose browser in own sources: does it fixed rotation or not (as you seen Chrome do this by default and Firefox/Safari - no)

Comment 7

[Vincent Liu[:vliu]]

Firefox dealt with image rotation for playback in [1]. But for drawImage() in canvas, FF didn't. I checked the whatwg spec about drawImage but it doesn't define what Gecko should do if the degree of VideoElement are not zero. One concern may consider that this image rotation may consumes performance because it contains Matrix calculation. Nical, may I have your input here for any suggestion?  Really thanks.

[1]: https://dxr.mozilla.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#237-265

Comment 8

[Nicolas Silva [:nical]]

We should apply the rotation. If applying this rotation adds the cost of an extra transformation we can always special-case the common path where no rotation is applied and avoid paying that cost when it's not needed.

Comment 9

[Vincent Liu[:vliu]]

Attachment: WIP-1.patch

Here to record the WIP for the current status. This WIP only works if the video rotates 180 degree. It also contains a problem that the second snapshot turns whole block.

Comment 10

[Vincent Liu[:vliu]]

(In reply to longman from comment #6)
> i have tried to find some JS-solution to detect degree-info from canvas or
> from video file but it is unsuccessfull, we don't know what way choose
> browser in own sources: does it fixed rotation or not (as you seen Chrome do
> this by default and Firefox/Safari - no)

Hi longman,
I would like to test clips which contain 90 and 270 degree. Do you have any clips for them? Or do you know how to make a clip with desired degree? Thanks

Comment 11

[longman]

Hi Vincent,
it is very simple to capture video with smartphone. i have very helpful link for this https://addpipe.com/blog/mp4-rotation-metadata-in-mobile-video-files/. 
if you will have some questions please inform me. thanks!

Comment 12

[longman]

Hi Vincent, what is the status of this ticket? more than a month ago from my last post.. thanks

Comment 13

[Vincent Liu[:vliu]]

(In reply to longman from comment #12)
> Hi Vincent, what is the status of this ticket? more than a month ago from my
> last post.. thanks

Sorry too late to reply. I still stuck on other stability issues and can't come back to this one.

Comment 14

[Hani Yacoub, Desktop QA]

I can't manage to reproduce this issue on my end, "No video with supported format and MIME type found" message received from accessing the link from the description.
longman, is this issue still reproducible?
Thanks.

Comment 15

[longman]

(In reply to Hani Yacoub from comment #14)

I can't manage to reproduce this issue on my end, "No video with supported format and MIME type found" message received from accessing the link from the description.
longman, is this issue still reproducible?
Thanks.

Hello, i added permissions to source file. please check updated fiddle: http://jsfiddle.net/longman/umjkv6eg/11/
and YES, problem is still here.

what is your plan after 5 years of this issue? :)

Thanks!

Comment 16

[Hani Yacoub, Desktop QA]

Thanks for updating fiddle.
I wanted to reconfirm if this issue is still reproducible and might be find if there is any intentions to fix this.
The issue is reproducible on Windows 10 x64, macOS 11.6 and on Ubuntu 20.04 x64.

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last 7 months.
:lsalzman, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:lsalzman, could you have a look please?

For more information, please visit BugBot documentation.

Comment 20

[Lee Salzman [:lsalzman]]

Timothy, any idea if this is still an issue?

Comment 21

[Timothy Nikkel (:tnikkel)]

Testcase from comment 15 looks to be fixed by bug 1423850 for a while now.