Last Comment Bug 759945 - Android ICS/JB Software decoding ("software platform decoders") support for H.264/AAC/MP3 video/audio playback
: Android ICS/JB Software decoding ("software platform decoders") support for H...
Status: VERIFIED FIXED
:
Product: Core
Classification: Components
Component: Audio/Video (show other bugs)
: Trunk
: ARM Android
: -- normal with 1 vote (vote)
: mozilla17
Assigned To: cajbir (:cajbir)
:
:
Mentors:
http://cd.pn/b2
Depends on: 714408 783927
Blocks: 748351 755364 787226 781831 782508
  Show dependency treegraph
 
Reported: 2012-05-30 16:41 PDT by cajbir (:cajbir)
Modified: 2013-12-27 14:21 PST (History)
21 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
verified


Attachments
Software decoding on ICS/JB (491.52 KB, patch)
2012-07-24 22:30 PDT, cajbir (:cajbir)
no flags Details | Diff | Splinter Review
Software decoding on ICS/JB (1010.71 KB, patch)
2012-07-25 21:45 PDT, cajbir (:cajbir)
no flags Details | Diff | Splinter Review
Software decoding on ICS/JB (543.19 KB, patch)
2012-08-06 20:38 PDT, cajbir (:cajbir)
no flags Details | Diff | Splinter Review
Part1: Configure and build changes (4.93 KB, patch)
2012-08-08 19:43 PDT, cajbir (:cajbir)
khuey: review+
Details | Diff | Splinter Review
Part 2: Android specific changes (1.45 KB, patch)
2012-08-08 19:46 PDT, cajbir (:cajbir)
mark.finkle: review+
Details | Diff | Splinter Review
Part 3: Changes needed to OmxPlugin (549.31 KB, patch)
2012-08-08 19:53 PDT, cajbir (:cajbir)
cpearce: review+
Details | Diff | Splinter Review
Part 4: MP4 test file (479.51 KB, patch)
2012-08-08 20:57 PDT, cajbir (:cajbir)
cpearce: review+
Details | Diff | Splinter Review
Part 5: Fix build errors on B2G (2.86 KB, patch)
2012-08-08 22:52 PDT, cajbir (:cajbir)
cpearce: review+
Details | Diff | Splinter Review
Part 6: Respect media.plugins.enabled flag (2.37 KB, patch)
2012-08-09 21:37 PDT, cajbir (:cajbir)
cpearce: review+
Details | Diff | Splinter Review
Part 3: Changes needed to OmxPlugin (549.39 KB, patch)
2012-08-09 21:39 PDT, cajbir (:cajbir)
cajbir.bugzilla: review+
Details | Diff | Splinter Review

Description cajbir (:cajbir) 2012-05-30 16:41:30 PDT
Bug 714408 implements H.264/AAC/MP3 playback on B2G using Android libstagefright libraries. This bug is to make that work on consumer android devices.

Steps to reproduce:

1. Visit http://cd.pn/b2 in Firefox on Android
2. Press Play

What happens:

The controls disappear and no playback occurs.

What should happen:

Video should play.
Comment 1 cajbir (:cajbir) 2012-07-24 22:30:36 PDT
Created attachment 645651 [details] [diff] [review]
Software decoding on ICS/JB

This patch adds support for H.264 decoding using libstagefright software decoders on some ICS and Jellybean Android devices. Tested on:

Galaxy Note running ICS
Nexus S running ICS
Nexus S running Jellybean
HTC One X (International) running ICS

To build you'll need to have the Android OS source code (not just the SDK) for ICS. You'll also need the following shared libraries from an android device for linking:

libstagefright_omx.so
libstagefright.so
libutils.so

Add the following .mozconfig options:

--enable-media-plugins
--enable-omx-plugin
--with-android-source=/path/to/android/source
--with-stagefright-libs=/path/to/android/shared/libraries

The "with-stagefright-libs" should be a path to a directory containing the shared libraries mentioned above. These can be retrieved from a device with "adb pull /system/lib/libraryname.so".

A future iteration of the patch will hopefully remove the need for the binaries and the requirement for the android OS source.

Test videos:

http://cd.pn/b
http://cd.pn/b2

Patch includes a test video in content/media/test for running mochitests on a device.
Comment 2 cajbir (:cajbir) 2012-07-25 21:45:26 PDT
Created attachment 646020 [details] [diff] [review]
Software decoding on ICS/JB

Removes "--with-android-source" configure option and instead includes necessary Android OS headers directly. It's no longer needed to have the Android OS source around.
Comment 3 Maire Reavy [:mreavy] 2012-07-26 09:42:36 PDT
We're morphing bug to follow software platform decoder support for Android.
Comment 4 cajbir (:cajbir) 2012-08-06 20:38:15 PDT
Created attachment 649532 [details] [diff] [review]
Software decoding on ICS/JB

Removes need for binary files to link against. This patch builds stub versions of the libstagefright shared libraries we need and links against them. These stubs are not installed on the phone so at runtime we end up using the real libstagefright libraries.
Comment 5 cajbir (:cajbir) 2012-08-08 19:43:47 PDT
Created attachment 650423 [details] [diff] [review]
Part1: Configure and build changes

Configure and build changes for software decoding of H.264/AAC/MP3 content on Android ICS/JB using libstagefright.
Comment 6 cajbir (:cajbir) 2012-08-08 19:46:35 PDT
Created attachment 650425 [details] [diff] [review]
Part 2: Android specific changes

Changes to mobile/android code to get the libomxplugin installed in the correct place and add the preference so that media plugins are disabled by default.
Comment 7 cajbir (:cajbir) 2012-08-08 19:53:37 PDT
Created attachment 650426 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Imports Android OS headers needed for building. Adds code to build stub shared libraries to link against. These shared libraries aren't distributed - they are only used during the build to link against. On the device Fennec will link against the installed ICS/JB android libraries.

Media plugins are disabled via a pref by default as tests fail and playback is not yet performant.

Minor changes to the OmxPlugin code to get playback working.
Comment 8 cajbir (:cajbir) 2012-08-08 20:57:49 PDT
Created attachment 650434 [details] [diff] [review]
Part 4: MP4 test file

Adds an MP4 file for testing in the media mochitests that was missing from part 3.
Comment 9 Chris Pearce (:cpearce) 2012-08-08 21:11:35 PDT
Comment on attachment 650426 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Review of attachment 650426 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/plugins/nsMediaPluginReader.cpp
@@ +142,5 @@
>      }
>      mVideoSeekTimeUs = -1;
>  
>      if (aKeyframeSkip) {
> +      // Disable keyframe skipping for now as

In Edwin's patches we're using some function he found that seeks the decoders to the next keyframe, maybe we can use that here too?

::: content/media/test/manifest.js
@@ +149,5 @@
>  
>    // Opus data in an ogg container
>    { name:"detodos.opus", type:"audio/ogg; codecs=opus", duration:2.9135 },
>  
> +  { name:"gizmo.mp4", type:"video/mp4", duration:5.0 } ,

nit: Remove space between } and ,

::: media/omx-plugin/OmxPlugin.cpp
@@ +216,1 @@
>      }

Indentation is off here.

@@ +523,5 @@
>    else if (err == INFO_FORMAT_CHANGED) {
>      // If the format changed, update our cached info.
> +    if (!SetVideoFormat())
> +      return false;
> +    else return ReadVideo(aFrame, aSeekTimeUs);

Break between else and return to be consistent please.

@@ +579,2 @@
>  
>    return true;

You can't reach this |return true|, so you don't need it now, right?
Comment 10 cajbir (:cajbir) 2012-08-08 22:52:06 PDT
Created attachment 650443 [details] [diff] [review]
Part 5: Fix build errors on B2G

Building on B2G requires conditionals in the makefiles to stop building the stub shared libraries as the real versions are built as part of B2G already.
Comment 11 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-08-09 08:21:42 PDT
Comment on attachment 650443 [details] [diff] [review]
Part 5: Fix build errors on B2G

Review of attachment 650443 [details] [diff] [review]:
-----------------------------------------------------------------

This would be best reviewed by someone familiar with the code.  The build changes are trivial.
Comment 12 cajbir (:cajbir) 2012-08-09 21:37:13 PDT
Created attachment 650786 [details] [diff] [review]
Part 6: Respect media.plugins.enabled flag

media.plugins.enabled was being ignored. This patch fixes that.
Comment 13 cajbir (:cajbir) 2012-08-09 21:39:53 PDT
Created attachment 650787 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Address review comments, carrying r+ forward. For this comment I'll address in a followup bug to fix playback issues that come up during testing:

>In Edwin's patches we're using some function he
> found that seeks the decoders to the next keyframe,
> maybe we can use that here too?
Comment 15 Aaron Train [:aaronmt] 2012-08-10 07:37:03 PDT
Using mozilla-inbound (17) (3bead0b0dd75), with the preference enabled there is still no available support (e.g, the test-case in comment #0) will not play (Video format or MIME type is not supported).
Comment 16 cajbir (:cajbir) 2012-08-10 08:14:31 PDT
(In reply to Aaron Train [:aaronmt] from comment #15)
> Using mozilla-inbound (17) (3bead0b0dd75), with the preference enabled there
> is still no available support (e.g, the test-case in comment #0) will not
> play (Video format or MIME type is not supported).

Do any of the following URL's play for you:

http://cd.pn/a
http://cd.pn/b
http://cd.pn/b2

What device are you using and what Android version does it have? Have you enabled the pref media.plugins.enabled?
Comment 17 Aaron Train [:aaronmt] 2012-08-10 08:18:13 PDT
(In reply to Chris Double (:doublec) from comment #16)
> http://cd.pn/a

E/GeckoConsole( 5500): [JavaScript Warning: "HTTP "Content-Type" of "audio/mpeg" is not supported. Load of media resource http://cd.pn/a/joy.mp3 failed." {file: "http://cd.pn/a/" line: 0}]

> http://cd.pn/b

E/GeckoConsole( 5871): [JavaScript Warning: "HTTP "Content-Type" of "video/mp4" is not supported. Load of media resource http://cd.pn/b/bounty.mp4 failed." {file: "http://cd.pn/b/" line: 0}]

> http://cd.pn/b2

E/GeckoConsole( 5871): [JavaScript Warning: "HTTP "Content-Type" of "video/mp4" is not supported. Load of media resource http://cd.pn/b2/story.mp4 failed." {file: "http://cd.pn/b2/" line: 0}]
 
> What device are you using and what Android version does it have?

Samsung Galaxy Nexus (Android 4.1.1).

>Have you enabled the pref media.plugins.enabled?

Yes.
Comment 18 cajbir (:cajbir) 2012-08-10 08:29:42 PDT
Did you build with "--enable-media-plugins" and "--enable-omx-plugin"? They're off by default.
Comment 19 Aaron Train [:aaronmt] 2012-08-10 08:39:51 PDT
(In reply to Chris Double (:doublec) from comment #18)
> "--enable-omx-plugin"?
Missed that one.
Comment 20 cajbir (:cajbir) 2012-08-10 08:41:40 PDT
They should be the default on Android since it's pref'd off. I'll submit a bug/patch.
Comment 21 cajbir (:cajbir) 2012-08-10 08:45:04 PDT
Bug 781831 for enabling the configure options on android to build by default.
Comment 23 Lawrence Mandel [:lmandel] (use needinfo) 2012-08-13 09:18:38 PDT
Comment 1 lists 4 devices on H.264 decoding is supported with this patch:

Galaxy Note running ICS
Nexus S running ICS
Nexus S running Jellybean
HTC One X (International) running ICS

1. Is there a complete list of devices for which this patch adds H.264 decoding? 
2. Does this patch require ICS or Jellybean? Does it work with previous versions of Android?
Comment 24 cajbir (:cajbir) 2012-08-13 16:17:29 PDT
(In reply to Lawrence Mandel [:lmandel] from comment #23)
> 1. Is there a complete list of devices for which this patch adds H.264
> decoding? 

There isn't a complete list, only those I listed previously which I've tested on. It should work on any ICS or JB device though.

> 2. Does this patch require ICS or Jellybean? 

Either works.

> Does it work with previous versions of Android? 

I have not yet tested on previous versions but will do soon.
Comment 25 Maire Reavy [:mreavy] 2012-08-30 13:58:00 PDT
I'm changing the title to reflect that this is only for ICS/JB.  For tracking purposes I'm opening a new bug for GB support which will be added soon.
Comment 26 Maire Reavy [:mreavy] 2012-08-30 15:17:16 PDT
Moving dependencies on content-specific bugs against libstagefright to Bug 787226, which will be the meta bug tracking such issues.
Comment 27 Cristian Nicolae (:xti) 2012-09-19 06:50:18 PDT
Changes were applied on the latest Nightly. However, the video content is not rendered, but this is a known issue already filled. Instead the audio is played correctly.
Closing bug as verified fixed.

--
Firefox 18.0a1 (2012-09-19)
Device: Galaxy Note
OS: Android 4.0.4

Note You need to log in before you can comment on or make changes to this bug.