889625 - Trying to view a video from a blob URL derived from the video app renders videos 90 degrees off the correct orientation

Status: NEW

(Core :: Audio/Video: Playback, defect)

Description

[Jason Smith [:jsmith]]

Build: B2G 18 7/2/2013
Device: Unagi

STR

1. Go to http://mozilla.github.io/qa-testcase-data/webapi/filemanagement/fileUploadTest.html in the browser
2. Select browse on the "Upload Single File with No Filter"
3. Select video
4. Pick a video you have on your sdcard
5. Select submit query
6. Go to the bottom of the page and select the resulting URL

Expected

The video should start playing in alignment with the orientation of the phone.

Actual

See screenshots. The video renders 90 degrees off the correct orientation.

Comment 1

[Jason Smith [:jsmith]]

Attachment: Portrait View of Video from Video App

Comment 2

[Jason Smith [:jsmith]]

Attachment: Landscape Video from Video App

Comment 3

[Jason Smith [:jsmith]]

Attachment: Logcat

Comment 4

[Jason Smith [:jsmith]]

Interesting logcat points:

07-02 15:36:46.214: E/GeckoConsole(7066): [JavaScript Warning: "HTTP "Content-Type" of "text/html" is not supported. Load of media resource about:blank failed." {file: "app://video.gaiamobile.org/index.html#pick" line: 0}]
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser.so' not found 
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LITE_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser_lite.so' not found
07-02 15:36:48.176: E/GeckoConsole(7066): [JavaScript Error: "NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIDOMMozURLProperty.createObjectURL]" {file: "app://video.gaiamobile.org/gaia_build_defer_index.js" line: 449}]
07-02 15:36:48.176: I/Gecko(7066): ###!!! [Child][RPCChannel] Error: Channel closing: too late to send/recv, messages will be lost
07-02 15:36:54.292: E/GeckoConsole(109): [JavaScript Error: "NS_ERROR_DOM_BAD_URI: Access to restricted URI denied" {file: "app://browser.gaiamobile.org/gaia_build_defer_index.js" line: 134}]

Comment 5

[Jason Smith [:jsmith]]

Hmm...there's a lot of interesting logcat points being shown across the Video app, Browser app, and core Gecko. The video used here btw was 3GP as well.

cc-ing folks across those functional areas.

Comment 6

[Jason Smith [:jsmith]]

I'm debating on nominating this for leo, but I really need developer input here to figure out how often this bug could happen. My use case specifically happens if I take a video from the video app via DOM input file and allow a user to view a video from the blob URL. I'm wondering if the general problem here could have deeper impact.

Chris - Do you have any ideas on probability here for this bug?

Comment 7

[cajbir (:cajbir)]

I don't have any ideas on the probability for the bug, sorry. If you take the video off the device and try it in a desktop browser does the same thing happen? If you play the video from a file:// URL (is that possible in FxOS?) or an HTTP URL does it happen?

BTW, these two logcat notes are normal and to be expected:

07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser.so' not found 
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LITE_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser_lite.so' not found

Comment 8

[Jason Smith [:jsmith]]

Opps, didn't mean to nom this.

Comment 9

[Jason Smith [:jsmith]]

I pulled the file directly off my device and analyzed some of the videos directly in windows media player. It looks like all of the videos are being recorded in a landscape orientation, even if the camera viewfinder shows that the video is being recorded in a portrait orientation. That doesn't sound like the correct behavior - what I'd expect to happen is something like the following:

When a camera starts recording a video, the orientation of the video should center itself around the current orientation of the phone on start. So if it starts portrait, then the video's resulting orientation should be portrait. If it started in landscape, it should have an orientation of landscape.

Given that this isn't a core video bug anymore, I'm sending this over to B2G General and putting needinfo on mikeh for input.

Comment 10

[Mike Habicher [:mikeh] (high bugzilla latency)]

jsmith, a number of devices have their camera sensors mounted rotationally offset from the natural orientation of the handset. In other words, what you would consider to be the "top" of the phone and what the camera/video encoder consider to be the "top" of the video frame are different.

We account for this by setting a property in the recording API that indicates a combination of this offset and the overall orientation of the phone as determined by the orientation sensor.

The video encoder takes this property and writes a transform matrix into the header of the video. This matrix is intended to be applied to the video during rendering/playback, and while it can be used for some pretty arbitrary transforms, in our case we use it to apply the correct { 0, 90, 180, 270 }-degree rotation to the video container.

The Video and Gallery apps extract and respect this matrix, as would any other apps that need to playback video with the correct orientation.

Two things: one, this is how Android solves this problem; two, most desktop video players do not respect it, including VLC. QuickTime -does- respect the matrix, and IIRC, YouTube can handle it with uploaded videos as well.

This was all hashed out in bug 812924 (see bug 812924 comment 22 for tech ref links) and djf wrote the matrix extraction code for JS.

Back to your blob issue: provided the transform matrix survives the blob-ification, you should be able to get the transform matrix and extract the rotation information from it.

Comment 11

[Jason Smith [:jsmith]]

So I guess what you are implying then is that we're providing the metadata, but the browser isn't respecting it, so the video being rendered in the browser isn't following the orientation matrix provided. In that case, that actually does make this a core video/audio issue for not respecting a piece of metadata provided when rendering videos.

Comment 12

[cajbir (:cajbir)]

Sotoro, do you know how to get this rotational data from Stagefright?

Comment 13

[Sotaro Ikeda [:sotaro]]

In android, rotation is handled within platform. An application do not need to care about the video's rotation.
- [1] notify already rotated video size
- [2] generate thumbnail of already rotated video frame
- [3] video rendering with rotation

Within stagefright, the rotation information can be get from video track by calling following.
> videoTrack->getFormat()->findInt32(kKeyRotation, &rotationDegrees)


[1] notify already rotated video size
AwesomePlayer and MediaMetadataRetriever notify the already rotated video frame size.
http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/AwesomePlayer.cpp#1027
http://androidxref.com/4.0.4/xref/frameworks/base/media/jni/android_media_MediaMetadataRetriever.cpp#257

[2] generate thumbnail of already rotated video frame
MediaMetadataRetriever generates the already rotated thumbnail.
http://androidxref.com/4.0.4/xref/frameworks/base/media/jni/android_media_MediaMetadataRetriever.cpp#277

[3] video rendering with rotation
The rotation is set to ANativeWindow. In android ANativeWindow is video buffer sournce and rendering target. It is delivered to SurfaceTexture and SurfaceTexture and rendered correctly.
http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/OMXCodec.cpp#1819

Comment 14

[Sotaro Ikeda [:sotaro]]

I feel it should be handled in platform layer not in application.

Comment 15

[Sotaro Ikeda [:sotaro]]

It seems that rotation can be set to a layer of video rendeirng at nsVideoFrame::BuildLayer().

http://mxr.mozilla.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#179

Comment 16

[Sotaro Ikeda [:sotaro]]

The rotation could also affect to a MediaStream made from <video> tag.

Comment 17

[Sotaro Ikeda [:sotaro]]

I am not sure when mp4's "video orientation metadata" is handled at gecko, how gecko expose this capability to gaia/web content side. EXIF metadata's case, it is exposed by using css like ` image-orientation: from-image` 

https://developer.mozilla.org/en-US/docs/Web/CSS/image-orientation

Comment 18

[Sotaro Ikeda [:sotaro]]

But if css setting in web site is necessary, usecomment #0 is not rescued.

Comment 19

[Sotaro Ikeda [:sotaro]]

It is too late for v1.2. Change to "1.3?"

Comment 20

[Maire Reavy [:mreavy]]

This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO

Comment 21

[Jason Smith [:jsmith]]

(In reply to Maire Reavy [:mreavy] from comment #20)
> This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO

Usually the guideline we use for triage is that if the past release already ships with the bug, then the bug isn't a blocker unless there's some special reason (e.g. partner compliant, cert waiver, support problem, top crash, sec-high/critical bug). Given that criteria, I actually don't think this is a blocker.

Comment 22

[Ivan Tsay (:ITsay)]

(In reply to Jason Smith [:jsmith] from comment #21)
> (In reply to Maire Reavy [:mreavy] from comment #20)
> > This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO
> 
> Usually the guideline we use for triage is that if the past release already
> ships with the bug, then the bug isn't a blocker unless there's some special
> reason (e.g. partner compliant, cert waiver, support problem, top crash,
> sec-high/critical bug). Given that criteria, I actually don't think this is
> a blocker.

Hi Hema,

May I have your comment on this bug triage? I agree with Jason that this one should not be a blocker. But since it goes to media, your comment is appreciated.

Comment 23

[Hema Koka [:hema]]

I don't think this should block based on the triage criteria - also highlighted in Jason's comment. For better experience, we should consider addressing this soon at least by 1.4 (for 1.3, we are currently working on orientation correction for images)