Trying to view a video from a blob URL derived from the video app renders videos 90 degrees off the correct orientation

NEW
Unassigned

Status

()

Core
Audio/Video: Playback
5 years ago
3 months ago

People

(Reporter: jsmith, Unassigned)

Tracking

Trunk
ARM
Gonk (Firefox OS)
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fxos:media])

Attachments

(3 attachments)

(Reporter)

Description

5 years ago
Build: B2G 18 7/2/2013
Device: Unagi

STR

1. Go to http://mozilla.github.io/qa-testcase-data/webapi/filemanagement/fileUploadTest.html in the browser
2. Select browse on the "Upload Single File with No Filter"
3. Select video
4. Pick a video you have on your sdcard
5. Select submit query
6. Go to the bottom of the page and select the resulting URL

Expected

The video should start playing in alignment with the orientation of the phone.

Actual

See screenshots. The video renders 90 degrees off the correct orientation.
(Reporter)

Comment 1

5 years ago
Created attachment 770474 [details]
Portrait View of Video from Video App
(Reporter)

Comment 2

5 years ago
Created attachment 770476 [details]
Landscape Video from Video App
(Reporter)

Comment 3

5 years ago
Created attachment 770478 [details]
Logcat
(Reporter)

Comment 4

5 years ago
Interesting logcat points:

07-02 15:36:46.214: E/GeckoConsole(7066): [JavaScript Warning: "HTTP "Content-Type" of "text/html" is not supported. Load of media resource about:blank failed." {file: "app://video.gaiamobile.org/index.html#pick" line: 0}]
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser.so' not found 
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LITE_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser_lite.so' not found
07-02 15:36:48.176: E/GeckoConsole(7066): [JavaScript Error: "NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIDOMMozURLProperty.createObjectURL]" {file: "app://video.gaiamobile.org/gaia_build_defer_index.js" line: 449}]
07-02 15:36:48.176: I/Gecko(7066): ###!!! [Child][RPCChannel] Error: Channel closing: too late to send/recv, messages will be lost
07-02 15:36:54.292: E/GeckoConsole(109): [JavaScript Error: "NS_ERROR_DOM_BAD_URI: Access to restricted URI denied" {file: "app://browser.gaiamobile.org/gaia_build_defer_index.js" line: 134}]
(Reporter)

Comment 5

5 years ago
Hmm...there's a lot of interesting logcat points being shown across the Video app, Browser app, and core Gecko. The video used here btw was 3GP as well.

cc-ing folks across those functional areas.
(Reporter)

Comment 6

5 years ago
I'm debating on nominating this for leo, but I really need developer input here to figure out how often this bug could happen. My use case specifically happens if I take a video from the video app via DOM input file and allow a user to view a video from the blob URL. I'm wondering if the general problem here could have deeper impact.

Chris - Do you have any ideas on probability here for this bug?
blocking-b2g: --- → leo?
Flags: needinfo?(chris.double)

Comment 7

5 years ago
I don't have any ideas on the probability for the bug, sorry. If you take the video off the device and try it in a desktop browser does the same thing happen? If you play the video from a file:// URL (is that possible in FxOS?) or an HTTP URL does it happen?

BTW, these two logcat notes are normal and to be expected:

07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser.so' not found 
07-02 15:36:46.274: E/ExtendedExtractor(7066): Failed to open MM_PARSER_LITE_LIB, dlerror = Cannot load library: load_library[1091]: Library 'libmmparser_lite.so' not found
Flags: needinfo?(chris.double)
(Reporter)

Comment 8

5 years ago
Opps, didn't mean to nom this.
blocking-b2g: leo? → ---
(Reporter)

Comment 9

5 years ago
I pulled the file directly off my device and analyzed some of the videos directly in windows media player. It looks like all of the videos are being recorded in a landscape orientation, even if the camera viewfinder shows that the video is being recorded in a portrait orientation. That doesn't sound like the correct behavior - what I'd expect to happen is something like the following:

When a camera starts recording a video, the orientation of the video should center itself around the current orientation of the phone on start. So if it starts portrait, then the video's resulting orientation should be portrait. If it started in landscape, it should have an orientation of landscape.

Given that this isn't a core video bug anymore, I'm sending this over to B2G General and putting needinfo on mikeh for input.
Component: Video/Audio → General
Flags: needinfo?(mhabicher)
Product: Core → Boot2Gecko
Version: Trunk → unspecified
jsmith, a number of devices have their camera sensors mounted rotationally offset from the natural orientation of the handset. In other words, what you would consider to be the "top" of the phone and what the camera/video encoder consider to be the "top" of the video frame are different.

We account for this by setting a property in the recording API that indicates a combination of this offset and the overall orientation of the phone as determined by the orientation sensor.

The video encoder takes this property and writes a transform matrix into the header of the video. This matrix is intended to be applied to the video during rendering/playback, and while it can be used for some pretty arbitrary transforms, in our case we use it to apply the correct { 0, 90, 180, 270 }-degree rotation to the video container.

The Video and Gallery apps extract and respect this matrix, as would any other apps that need to playback video with the correct orientation.

Two things: one, this is how Android solves this problem; two, most desktop video players do not respect it, including VLC. QuickTime -does- respect the matrix, and IIRC, YouTube can handle it with uploaded videos as well.

This was all hashed out in bug 812924 (see bug 812924 comment 22 for tech ref links) and djf wrote the matrix extraction code for JS.

Back to your blob issue: provided the transform matrix survives the blob-ification, you should be able to get the transform matrix and extract the rotation information from it.
Flags: needinfo?(mhabicher)
(Reporter)

Comment 11

5 years ago
So I guess what you are implying then is that we're providing the metadata, but the browser isn't respecting it, so the video being rendered in the browser isn't following the orientation matrix provided. In that case, that actually does make this a core video/audio issue for not respecting a piece of metadata provided when rendering videos.
Component: General → Video/Audio
Product: Boot2Gecko → Core
Version: unspecified → Trunk

Comment 12

5 years ago
Sotoro, do you know how to get this rotational data from Stagefright?
Flags: needinfo?(sotaro.ikeda.g)
In android, rotation is handled within platform. An application do not need to care about the video's rotation.
- [1] notify already rotated video size
- [2] generate thumbnail of already rotated video frame
- [3] video rendering with rotation

Within stagefright, the rotation information can be get from video track by calling following.
> videoTrack->getFormat()->findInt32(kKeyRotation, &rotationDegrees)


[1] notify already rotated video size
AwesomePlayer and MediaMetadataRetriever notify the already rotated video frame size.
http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/AwesomePlayer.cpp#1027
http://androidxref.com/4.0.4/xref/frameworks/base/media/jni/android_media_MediaMetadataRetriever.cpp#257

[2] generate thumbnail of already rotated video frame
MediaMetadataRetriever generates the already rotated thumbnail.
http://androidxref.com/4.0.4/xref/frameworks/base/media/jni/android_media_MediaMetadataRetriever.cpp#277

[3] video rendering with rotation
The rotation is set to ANativeWindow. In android ANativeWindow is video buffer sournce and rendering target. It is delivered to SurfaceTexture and SurfaceTexture and rendered correctly.
http://androidxref.com/4.0.4/xref/frameworks/base/media/libstagefright/OMXCodec.cpp#1819
Flags: needinfo?(sotaro.ikeda.g)
I feel it should be handled in platform layer not in application.
It seems that rotation can be set to a layer of video rendeirng at nsVideoFrame::BuildLayer().

http://mxr.mozilla.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#179
The rotation could also affect to a MediaStream made from <video> tag.

Updated

5 years ago
blocking-b2g: --- → koi?

Updated

5 years ago
Whiteboard: [fxos:media]
I am not sure when mp4's "video orientation metadata" is handled at gecko, how gecko expose this capability to gaia/web content side. EXIF metadata's case, it is exposed by using css like ` image-orientation: from-image` 

https://developer.mozilla.org/en-US/docs/Web/CSS/image-orientation
But if css setting in web site is necessary, usecomment #0 is not rescued.
It is too late for v1.2. Change to "1.3?"
blocking-b2g: koi? → 1.3?
This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO
(Reporter)

Comment 21

5 years ago
(In reply to Maire Reavy [:mreavy] from comment #20)
> This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO

Usually the guideline we use for triage is that if the past release already ships with the bug, then the bug isn't a blocker unless there's some special reason (e.g. partner compliant, cert waiver, support problem, top crash, sec-high/critical bug). Given that criteria, I actually don't think this is a blocker.

Comment 22

5 years ago
(In reply to Jason Smith [:jsmith] from comment #21)
> (In reply to Maire Reavy [:mreavy] from comment #20)
> > This isn't in my area (WebRTC or Web Audio), but this should block 1.3 IMO
> 
> Usually the guideline we use for triage is that if the past release already
> ships with the bug, then the bug isn't a blocker unless there's some special
> reason (e.g. partner compliant, cert waiver, support problem, top crash,
> sec-high/critical bug). Given that criteria, I actually don't think this is
> a blocker.

Hi Hema,

May I have your comment on this bug triage? I agree with Jason that this one should not be a blocker. But since it goes to media, your comment is appreciated.
Flags: needinfo?(hkoka)

Comment 23

5 years ago
I don't think this should block based on the triage criteria - also highlighted in Jason's comment. For better experience, we should consider addressing this soon at least by 1.4 (for 1.3, we are currently working on orientation correction for images)
blocking-b2g: 1.3? → ---
Flags: needinfo?(hkoka)
(Reporter)

Updated

4 years ago
Duplicate of this bug: 1036897
Component: Audio/Video → Audio/Video: Playback
You need to log in before you can comment on or make changes to this bug.