Android ICS/JB Software decoding ("software platform decoders") support for H.264/AAC/MP3 video/audio playback

VERIFIED FIXED in Firefox 18

Status

()

Core
Audio/Video
VERIFIED FIXED
5 years ago
4 years ago

People

(Reporter: cajbir, Assigned: cajbir)

Tracking

(Blocks: 3 bugs)

Trunk
mozilla17
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(blocking-kilimanjaro:+, firefox18 verified)

Details

(URL)

Attachments

(6 attachments, 4 obsolete attachments)

(Assignee)

Description

5 years ago
Bug 714408 implements H.264/AAC/MP3 playback on B2G using Android libstagefright libraries. This bug is to make that work on consumer android devices.

Steps to reproduce:

1. Visit http://cd.pn/b2 in Firefox on Android
2. Press Play

What happens:

The controls disappear and no playback occurs.

What should happen:

Video should play.
(Assignee)

Updated

5 years ago
blocking-kilimanjaro: --- → ?
Depends on: 714408, 755364
(Assignee)

Updated

5 years ago
Blocks: 755364
No longer depends on: 755364
Blocks: 748351
Depends on: 761762
Depends on: 761814
Depends on: 761786
No longer depends on: 761762
No longer depends on: 761786
No longer depends on: 761814
blocking-kilimanjaro: ? → +
(Assignee)

Comment 1

5 years ago
Created attachment 645651 [details] [diff] [review]
Software decoding on ICS/JB

This patch adds support for H.264 decoding using libstagefright software decoders on some ICS and Jellybean Android devices. Tested on:

Galaxy Note running ICS
Nexus S running ICS
Nexus S running Jellybean
HTC One X (International) running ICS

To build you'll need to have the Android OS source code (not just the SDK) for ICS. You'll also need the following shared libraries from an android device for linking:

libstagefright_omx.so
libstagefright.so
libutils.so

Add the following .mozconfig options:

--enable-media-plugins
--enable-omx-plugin
--with-android-source=/path/to/android/source
--with-stagefright-libs=/path/to/android/shared/libraries

The "with-stagefright-libs" should be a path to a directory containing the shared libraries mentioned above. These can be retrieved from a device with "adb pull /system/lib/libraryname.so".

A future iteration of the patch will hopefully remove the need for the binaries and the requirement for the android OS source.

Test videos:

http://cd.pn/b
http://cd.pn/b2

Patch includes a test video in content/media/test for running mochitests on a device.
Assignee: nobody → chris.double
Status: NEW → ASSIGNED
(Assignee)

Comment 2

5 years ago
Created attachment 646020 [details] [diff] [review]
Software decoding on ICS/JB

Removes "--with-android-source" configure option and instead includes necessary Android OS headers directly. It's no longer needed to have the Android OS source around.
Attachment #645651 - Attachment is obsolete: true
We're morphing bug to follow software platform decoder support for Android.
Summary: Support H.264/AAC/MP3 video/audio playback on Android → Software decoding ("software platform decoders") support for H.264/AAC/MP3 video/audio playback on Android
(Assignee)

Comment 4

5 years ago
Created attachment 649532 [details] [diff] [review]
Software decoding on ICS/JB

Removes need for binary files to link against. This patch builds stub versions of the libstagefright shared libraries we need and links against them. These stubs are not installed on the phone so at runtime we end up using the real libstagefright libraries.
Attachment #646020 - Attachment is obsolete: true
(Assignee)

Comment 5

5 years ago
Created attachment 650423 [details] [diff] [review]
Part1: Configure and build changes

Configure and build changes for software decoding of H.264/AAC/MP3 content on Android ICS/JB using libstagefright.
Attachment #649532 - Attachment is obsolete: true
Attachment #650423 - Flags: review?(khuey)
(Assignee)

Comment 6

5 years ago
Created attachment 650425 [details] [diff] [review]
Part 2: Android specific changes

Changes to mobile/android code to get the libomxplugin installed in the correct place and add the preference so that media plugins are disabled by default.
Attachment #650425 - Flags: review?(mark.finkle)
(Assignee)

Comment 7

5 years ago
Created attachment 650426 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Imports Android OS headers needed for building. Adds code to build stub shared libraries to link against. These shared libraries aren't distributed - they are only used during the build to link against. On the device Fennec will link against the installed ICS/JB android libraries.

Media plugins are disabled via a pref by default as tests fail and playback is not yet performant.

Minor changes to the OmxPlugin code to get playback working.
Attachment #650426 - Flags: review?(cpearce)
Attachment #650425 - Flags: review?(mark.finkle) → review+
(Assignee)

Comment 8

5 years ago
Created attachment 650434 [details] [diff] [review]
Part 4: MP4 test file

Adds an MP4 file for testing in the media mochitests that was missing from part 3.
Attachment #650434 - Flags: review?(cpearce)
Comment on attachment 650426 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Review of attachment 650426 [details] [diff] [review]:
-----------------------------------------------------------------

::: content/media/plugins/nsMediaPluginReader.cpp
@@ +142,5 @@
>      }
>      mVideoSeekTimeUs = -1;
>  
>      if (aKeyframeSkip) {
> +      // Disable keyframe skipping for now as

In Edwin's patches we're using some function he found that seeks the decoders to the next keyframe, maybe we can use that here too?

::: content/media/test/manifest.js
@@ +149,5 @@
>  
>    // Opus data in an ogg container
>    { name:"detodos.opus", type:"audio/ogg; codecs=opus", duration:2.9135 },
>  
> +  { name:"gizmo.mp4", type:"video/mp4", duration:5.0 } ,

nit: Remove space between } and ,

::: media/omx-plugin/OmxPlugin.cpp
@@ +216,1 @@
>      }

Indentation is off here.

@@ +523,5 @@
>    else if (err == INFO_FORMAT_CHANGED) {
>      // If the format changed, update our cached info.
> +    if (!SetVideoFormat())
> +      return false;
> +    else return ReadVideo(aFrame, aSeekTimeUs);

Break between else and return to be consistent please.

@@ +579,2 @@
>  
>    return true;

You can't reach this |return true|, so you don't need it now, right?
Attachment #650426 - Flags: review?(cpearce) → review+
Attachment #650434 - Flags: review?(cpearce) → review+
(Assignee)

Comment 10

5 years ago
Created attachment 650443 [details] [diff] [review]
Part 5: Fix build errors on B2G

Building on B2G requires conditionals in the makefiles to stop building the stub shared libraries as the real versions are built as part of B2G already.
Attachment #650443 - Flags: review?(cpearce)
Attachment #650443 - Flags: review?(khuey)
Attachment #650423 - Flags: review?(khuey) → review+
Comment on attachment 650443 [details] [diff] [review]
Part 5: Fix build errors on B2G

Review of attachment 650443 [details] [diff] [review]:
-----------------------------------------------------------------

This would be best reviewed by someone familiar with the code.  The build changes are trivial.
Attachment #650443 - Flags: review?(khuey)
Attachment #650443 - Flags: review?(cpearce) → review+
(Assignee)

Comment 12

5 years ago
Created attachment 650786 [details] [diff] [review]
Part 6: Respect media.plugins.enabled flag

media.plugins.enabled was being ignored. This patch fixes that.
Attachment #650786 - Flags: review?(cpearce)
(Assignee)

Comment 13

5 years ago
Created attachment 650787 [details] [diff] [review]
Part 3: Changes needed to OmxPlugin

Address review comments, carrying r+ forward. For this comment I'll address in a followup bug to fix playback issues that come up during testing:

>In Edwin's patches we're using some function he
> found that seeks the decoders to the next keyframe,
> maybe we can use that here too?
Attachment #650426 - Attachment is obsolete: true
Attachment #650787 - Flags: review+
Attachment #650786 - Flags: review?(cpearce) → review+
(Assignee)

Comment 14

5 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/26f6849a88c7
https://hg.mozilla.org/integration/mozilla-inbound/rev/4676db63406f
https://hg.mozilla.org/integration/mozilla-inbound/rev/b3da99fa9720
https://hg.mozilla.org/integration/mozilla-inbound/rev/5ab960f1879e
https://hg.mozilla.org/integration/mozilla-inbound/rev/46972b7d90c3
https://hg.mozilla.org/integration/mozilla-inbound/rev/c47012dda861
Using mozilla-inbound (17) (3bead0b0dd75), with the preference enabled there is still no available support (e.g, the test-case in comment #0) will not play (Video format or MIME type is not supported).
(Assignee)

Comment 16

5 years ago
(In reply to Aaron Train [:aaronmt] from comment #15)
> Using mozilla-inbound (17) (3bead0b0dd75), with the preference enabled there
> is still no available support (e.g, the test-case in comment #0) will not
> play (Video format or MIME type is not supported).

Do any of the following URL's play for you:

http://cd.pn/a
http://cd.pn/b
http://cd.pn/b2

What device are you using and what Android version does it have? Have you enabled the pref media.plugins.enabled?
(In reply to Chris Double (:doublec) from comment #16)
> http://cd.pn/a

E/GeckoConsole( 5500): [JavaScript Warning: "HTTP "Content-Type" of "audio/mpeg" is not supported. Load of media resource http://cd.pn/a/joy.mp3 failed." {file: "http://cd.pn/a/" line: 0}]

> http://cd.pn/b

E/GeckoConsole( 5871): [JavaScript Warning: "HTTP "Content-Type" of "video/mp4" is not supported. Load of media resource http://cd.pn/b/bounty.mp4 failed." {file: "http://cd.pn/b/" line: 0}]

> http://cd.pn/b2

E/GeckoConsole( 5871): [JavaScript Warning: "HTTP "Content-Type" of "video/mp4" is not supported. Load of media resource http://cd.pn/b2/story.mp4 failed." {file: "http://cd.pn/b2/" line: 0}]
 
> What device are you using and what Android version does it have?

Samsung Galaxy Nexus (Android 4.1.1).

>Have you enabled the pref media.plugins.enabled?

Yes.
(Assignee)

Comment 18

5 years ago
Did you build with "--enable-media-plugins" and "--enable-omx-plugin"? They're off by default.
(In reply to Chris Double (:doublec) from comment #18)
> "--enable-omx-plugin"?
Missed that one.
(Assignee)

Comment 20

5 years ago
They should be the default on Android since it's pref'd off. I'll submit a bug/patch.
(Assignee)

Updated

5 years ago
Blocks: 781831
(Assignee)

Comment 21

5 years ago
Bug 781831 for enabling the configure options on android to build by default.
https://hg.mozilla.org/mozilla-central/rev/26f6849a88c7
https://hg.mozilla.org/mozilla-central/rev/4676db63406f
https://hg.mozilla.org/mozilla-central/rev/b3da99fa9720
https://hg.mozilla.org/mozilla-central/rev/5ab960f1879e
https://hg.mozilla.org/mozilla-central/rev/46972b7d90c3
https://hg.mozilla.org/mozilla-central/rev/c47012dda861
Status: ASSIGNED → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla17
Comment 1 lists 4 devices on H.264 decoding is supported with this patch:

Galaxy Note running ICS
Nexus S running ICS
Nexus S running Jellybean
HTC One X (International) running ICS

1. Is there a complete list of devices for which this patch adds H.264 decoding? 
2. Does this patch require ICS or Jellybean? Does it work with previous versions of Android?
(Assignee)

Comment 24

5 years ago
(In reply to Lawrence Mandel [:lmandel] from comment #23)
> 1. Is there a complete list of devices for which this patch adds H.264
> decoding? 

There isn't a complete list, only those I listed previously which I've tested on. It should work on any ICS or JB device though.

> 2. Does this patch require ICS or Jellybean? 

Either works.

> Does it work with previous versions of Android? 

I have not yet tested on previous versions but will do soon.
(Assignee)

Updated

5 years ago
Blocks: 782508

Updated

5 years ago
Depends on: 783927

Updated

5 years ago
Depends on: 784329
Depends on: 786071
Depends on: 786112
Depends on: 786117
Depends on: 784019
Depends on: 786248
I'm changing the title to reflect that this is only for ICS/JB.  For tracking purposes I'm opening a new bug for GB support which will be added soon.
Summary: Software decoding ("software platform decoders") support for H.264/AAC/MP3 video/audio playback on Android → Android ICS/JB Software decoding ("software platform decoders") support for H.264/AAC/MP3 video/audio playback
Blocks: 787226
Moving dependencies on content-specific bugs against libstagefright to Bug 787226, which will be the meta bug tracking such issues.
No longer depends on: 784019, 784329, 786071, 786112, 786117, 786248
Changes were applied on the latest Nightly. However, the video content is not rendered, but this is a known issue already filled. Instead the audio is played correctly.
Closing bug as verified fixed.

--
Firefox 18.0a1 (2012-09-19)
Device: Galaxy Note
OS: Android 4.0.4
Status: RESOLVED → VERIFIED
status-firefox18: --- → verified
You need to log in before you can comment on or make changes to this bug.