755364 - [meta] Platform decoders for Android

Status: RESOLVED INACTIVE

(Core :: Audio/Video: Playback, defect)

Description

[Maire Reavy [:mreavy]]

Attachment: Top 80% of Android devices in our target market

This feature gives gecko the ability to use a device's hardware decoder to play back encoded video or audio (in particular H.264, AAC, and MP3) on that device.  This basis for this work was started in Bug 714408.

The Platform Decoder feature used to be called MPAPI support.

Feature page: https://wiki.mozilla.org/Platform/Features/MPAPI

We will target this feature to specific hardware devices and specific Android versions.  The list of devices and versions that will be supported in the first release of this feature will be determined largely based on popularity.

The attachment is a compilation of the top 80% of Android devices in our target market.  They are grouped by chipset.  You'll note that there are only 6 major chipsets in the market based on this data.  We can use this info to pick a good cross-section of devices amongst the 6 chipsets.  

I tried to carefully review for any typos.  Please let me know if you find any.  If anyone wants an editable form of this doc, please just let me know.

Regarding what Android versions to target: over 85% of the market uses Android version 2.2 or 2.3. (So those are the versions to target.)

Specifically:
Android 2.2 (Froyo) - 20.9%
Android 2.3 (Gingerbread) - 64.4%%
Android 3.0 (Honeycomb) - 3.3%
Android 4.0 (Ice Cream Sandwich) - 4.9%

NOTE: Mozilla has made a decision not to support Android versions older than 2.2 (which is about 6.5% of the market).

(Android version data collected from http://developer.android.com/resources/dashboard/platform-versions.html during a 14-day period ending on May 1, 2012.)

Comment 1

[Maire Reavy [:mreavy]]

Below is the list of first phase Android phones to support and where it comes from.  The list is currently based on popularity, and I expect this list to change as we continue to better understand our market, do our development, and test the phones we're targeting. 

I'm starting with 9 phones.  Targeting 9 phones during Q3 may be too optimistic or we may be able to add to the list.  I believe we won't know if it's "too many, too few, or just right" until we get further into development.   We also may reshuffle the order of phones.  Right now they are in priority order based on what we know about the market and what we think we can support.

SIMPLE LIST OF PHONES FOR PHASE 1 (the chipset that the phone uses is in parentheses):

    1)Sony Ericsson Xperia Arc (Qualcomm – Snapdragon)
    2)Samsung Nexus S (Hummingbird - Exynos 3110)
    3)Samsung I9000 Galaxy S (Hummingbird - Exynos 3110) [This and the Nexus S are both really popular!]
    4)Samsung I9003 Galaxy SL (TI OMAP 36xx)
    5)Motorola Atrix (NVidia Tegra 2)
    6)Samsung I9100G Galaxy S2 (TI OMAP 44xx)
    7)Samsung I9100 Galaxy S2 (Exynos 4210)
    8)Samsung I9300 Galaxy S III  (Exynos 4212 Quad)
    9)Samsung Galaxy S III T999, also called "Samsung Galaxy S III T-Mobile" (Qualcomm MSM8960 Snapdragon)  

NOTE: We can substitute T-Mobile's Samsung Galaxy S III with Verizon's or AT&T's Samsung Galaxy S III if the one from Verizon or AT&T is easier to get.  See more details at the end of this post.


HAVE QA TEST: (more for press and demos than popularity at this point)

    * Galaxy Nexus (TI OMAP 44xx) [Same chipset as the Samsung I9100G Galaxy S2]
    * Samsung P1000 Galaxy Tab [Same chipset as the Nexus S]

I'll work with QA and development to determine what phones to test and in what order.  The spreadsheet in this bug (added in May) is a good starting point for what phones to buy. 


IMPORTANT NOTES ON ANDROID OS VERSIONS:

    * We will focus initial development and testing on Gingerbread (version 2.3) and later because libstagefright (which the platform decoders rely on) didn't really become stable until Gingerbread. 
    * We may try to go back later and try get Froyo (2.2) to work; this decision is still TBD.  
    * Versions earlier than Froyo won't be supported. 
    * When we're ready to develop for Jelly Bean phones (which are just starting to come out), we probably want to upgrade a Galaxy Nexus or a Nexus S to JB.  I believe some carriers (AT&T and T-Mobile) are rolling JB out to Galaxy Nexus devices now and to Nexus S devices "soon."

____________________________________________________________________________________

Here's where the SIMPLE LIST (above) comes from:

At this point we're testing the theory that doing development by chipset helps us cover the market faster.  If this turns out to be incorrect, we should know pretty soon, and we'll make adjustments to our strategy and to the list.  If the platform decoder feature turns out to be phone specific, we could be looking at a true brute force approach (no short cuts) with a long development time to support 80% of the market.  I'm very much hoping that's not the case.

There are 6 chipsets that are used by 80% of the market.  I've listed the 2 most popular phones that use them:  (I gave 2 in case the first one was problematic to find or use, but I didn't want to complicate the simple list by giving the "backup phone" there.  So the simple list contains one phone from each set, plus the Galaxy S III because it's so popular.)

    1) Qualcomm – Snapdragon:
        * Sony Ericsson Xperia Arc - Features OS Android OS, v2.3 (Gingerbread), planned upgrade to v4.0
        * HTC Incredible S – Features OS Android OS, v2.2, v2.3, upgradable to v4.x
    2) Hummingbird (Exynos 3110):
        * Samsung Nexus S – Features OS Android OS, v2.3 (Gingerbread), upgradable to v4.0.4
        * Samsung I9000 Galaxy S - Features OS Android OS, v2.1 (Eclair), upgradable to v2.3
            ** NOTE: Samsung M110S Galaxy S is not the same as the I9000 Galaxy S and can't be upgraded to v2.3 
    3) TI OMAP 36xx:
        * Samsung I9003 Galaxy SL – Features OS Android OS, v2.2 (Froyo), upgradable to v2.3
        * Motorola Defy+ (plus) – Features OS Android OS, v2.3 (Gingerbread)
    4) NVidia Tegra 2:
        * Motorola Atrix – Features OS Android OS, v2.2 (Froyo), upgradable to v2.3 
        * LG Optimus 2X – Features OS Android OS, v2.2 (Froyo), upgradable to v2.3
    5) TI OMAP 44xx:
        * Samsung I9100G Galaxy S2 - Features OS Android OS, v2.3.4 (Gingerbread)
        * Motorola Milestone 3 XT860 – Features OS Android OS, v2.3.4 (Gingerbread)
    6) Exynos 4210:
        * Samsung I9100 Galaxy S2 - Features OS Android OS, v2.3.4 (Gingerbread), upgradable to v4.x 
        * Samsung Galaxy Note N7000 (not popular yet, but up-and-coming)


The Samsung Galaxy S III is a hot new phone, and there are 4 varieties, depending on the carrier:

    * Samsung I9300 Galaxy S III  - Chipset: Exynos 4212 Quad, released with ICS
    * Samsung Galaxy S III T999 ("Samsung Galaxy S III T-Mobile")  - Chipset: Qualcomm MSM8960 Snapdragon, released with ICS
    * Samsung Galaxy S III I535  ("Samsung Galaxy S III Verizon") -  Chipset:  Qualcomm MSM8960 Snapdragon, released with ICS
    * Samsung Galaxy S III I747  ("Samsung Galaxy S III AT&T") - Chipset: Qualcomm MSM8960 Snapdragon, released with ICS

They are all using newer chipsets: the Exynos 4212 and the latest Snapdragon MSM8960, which are follow-ons to the Exynos 4210 and the previous Snapdragon chipsets in our list.   There are no devices in the current market that use these, so we would be doing this work solely for the press and demos.  For that reason I've put the Samsung Galaxy S III phones at the end of the list at this point.  If we're sensing that a lot of reviewers will be using this phone to evaluate this feature, we may need to move it up on the list.


TWO PHONES WORTH TESTING (even though they aren't popular yet):

    * Samsung P1000 Galaxy Tab uses the Hummingbird chipset - our product displays well on this. (Good for demos)  
    * Galaxy Nexus uses the TI OMAP 44xx – Lots of press people have this.

Hopefully, we can piggy-back these two phones on the work done for the Hummingbird and TI OMAP 44xx chipsets.

Comment 2

[cajbir (:cajbir)]

(In reply to Maire Reavy [:mreavy] from comment #1)
> There are no devices in the current market that use
> these, so we would be doing this work solely for the press and demos.  

Is the S3 not available in the US yet? They've been available to consumers in NZ for a while who I wouldn't describe it as there being no devices in the current market using them. It's the Exynos 4212 version available here.

I have an ICS Nexus S, Gingerbread I9000 Galaxy S, and a Gingerbread  (or ICS depending on whether I upgrade) Galaxy Note I'm using for testing. I don't have any non-Samsung phones to develop with but getting it going on these will be a good start.

Comment 3

[Maire Reavy [:mreavy]]

(In reply to Chris Double (:doublec) from comment #2)
> (In reply to Maire Reavy [:mreavy] from comment #1)
> > There are no devices in the current market that use
> > these, so we would be doing this work solely for the press and demos.  
> 
> Is the S3 not available in the US yet? They've been available to consumers
> in NZ for a while who I wouldn't describe it as there being no devices in
> the current market using them. It's the Exynos 4212 version available here.

Sorry, I wrote that sentence too quickly.  I should have said, "relatively few devices in the current market" -- simply because it's so new and this list here is based on popularity (number of phones) in the existing Android phone market.  So at launch, relatively few users in the field would have this -- mostly press people, those doing demos, early adopters, etc.  I (and others) expect the S3 to be among the most popular in the market later on (once it's been out a little longer).

Comment 4

[cajbir (:cajbir)]

Some quick testing with the existing MPAPI code on mozilla-central ported to Android gives a binary that does software H.264 decoding using libstagefright successfully on:

Galaxy Note running ICS
Nexus S running ICS
Nexus S running Jellybean
HTC One X (International) running ICS

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Chris Double (:doublec) from comment #4)
> Some quick testing with the existing MPAPI code on mozilla-central ported to
> Android gives a binary that does software H.264 decoding using
> libstagefright successfully on:
> 
> Galaxy Note running ICS
> Nexus S running ICS
> Nexus S running Jellybean
> HTC One X (International) running ICS

Are there patches? Would you expect these work on all ICS and JB?

Comment 6

[Maire Reavy [:mreavy]]

(In reply to Brad Lassey [:blassey] from comment #5)
> (In reply to Chris Double (:doublec) from comment #4)
> > Some quick testing with the existing MPAPI code on mozilla-central ported to
> > Android gives a binary that does software H.264 decoding using
> > libstagefright successfully on:
> > 
> > Galaxy Note running ICS
> > Nexus S running ICS
> > Nexus S running Jellybean
> > HTC One X (International) running ICS
> 
> Are there patches? Would you expect these work on all ICS and JB?

And do we have a sense of how tough it will be to get GB working?

This is great news and great progress, Chris!  Thanks!!

Comment 7

[cajbir (:cajbir)]

(In reply to Brad Lassey [:blassey] from comment #5)
> 
> Are there patches? Would you expect these work on all ICS and JB?

I'm working on getting a landable patch up. Currently it needs the binary .so files to link against and a path to android OS source. I'd like to remove this need so I can run try server mochitests. If you want the patch as it is currently though I can get it to you. Yes, I'd expect it to work on all ICS and JB. 

> And do we have a sense of how tough it will be to get GB working?

The existing code for software decoding should be relatively easy to get working on GB. We really want to be using Edwin's new code though so we can get better performance and hardware decoding working. I believe this is a bit more ICS specific and might take more work to get going on GB.

Comment 8

[Maire Reavy [:mreavy]]

(In reply to Chris Double (:doublec) from comment #7)
> (In reply to Brad Lassey [:blassey] from comment #5)
> 
> > And do we have a sense of how tough it will be to get GB working?
> 
> The existing code for software decoding should be relatively easy to get
> working on GB. We really want to be using Edwin's new code though so we can
> get better performance and hardware decoding working. I believe this is a
> bit more ICS specific and might take more work to get going on GB.

Agreed, we want Edwin's new code which takes advantage of hardware decoding, but it's good to know we have a software fallback that works across ICS/JB and probably GB with relatively little effort.  

With Android Flash going away from the Google Play store next month (Aug 15), we (Mozilla) need to decide if software decoding using libstagefright could be used as an interim solution until the hardware decoding solution is ready. (BTW, I was reminded today that Android doesn't run on the train system yet, and even if it did, we'd fast-track an interim solution to this problem.)  

So it's really comes down to 
(1) How quickly do we think we could land a patch (or series of patches) that supports software decoding across GB, ICS, and JB?  
(2) Do we think the performance would be acceptable (I realize "acceptable" is not a metric, but I'm just looking for a thoughtful, educated guess) until hardware decoding support is ready?
(3) How much testing would it need? Or is it the kind of feature or code that either works or it doesn't across all phones running a particular OS?

Secondarily
(1) Is it also likely to be easy to get software decoding working for Honeycomb (since we'll be supporting tablets soon)?   If so, when can we have a patch for that support?
(2) Does it make sense to try to get software decoding working for Froyo?  I imagine that's trickier since libstagefright didn't stabilize until GB.  Do we have a sense of how much work it would be?
(3) Do we have a sense of how long it will take to get hardware decoding support across most of the market?  Do we know yet if hardware decoding support is a per phone effort or if phones running the same chipset are likely to work once we get one phone using that chipset working?

Sorry for all the questions!  There's a lot of interest in this right now.  The Android team is trying to figure out what it needs to do (if anything) in order to get an interim solution for H.264 video playback once Flash is no longer available for download, and I know you guys did some super work during the work week last week.  (Wish I could have been there!)

We (Mozilla) need to decide if the Android team needs to be brainstorming a "Plan B" interim solution or if software decoding will likely be sufficient given the time we have available.

We also may want to open a separate bug on "software platform decoders," especially if we will be releasing software decoding support separately from hardware decoding support (seems likely) and most especially if software platform decoders become the interim solution for Flash disappearing from the Google Play Store.

Thanks!

Comment 9

[cajbir (:cajbir)]

Brad, patch uploaded to bug 759945. Maire, I'm working on a response to your questions and will reply later tonight.

Comment 10

[cajbir (:cajbir)]

(In reply to Maire Reavy [:mreavy] from comment #8)
> (1) How quickly do we think we could land a patch (or series of patches)
> that supports software decoding across GB, ICS, and JB?  

From the existing MPAPI code and patch in bug 759945 we have ICS and JB covered for software decoding. It needs more work to get better performance and bug fixes - the mochitests hang in places - but I don't think it's far off. Days not weeks.

Gingerbread shouldn't be too different. Code based on Edwin's work will likely be different for GB though so I'm basing what I know from our existing MPAPI code.

> (2) Do we think the performance would be acceptable (I realize "acceptable"
> is not a metric, but I'm just looking for a thoughtful, educated guess)
> until hardware decoding support is ready?

WebM playback on the devices is good for mobile sized videos and that's done in software so I'd expect H.264 to be on par. This will hopefully mean videos like http://cd.pn/b2 (an example of what Mozilla uses on their sites) which is 640x360 should play on modern smartphones (Nexus S, etc). 

We won't have 720P, 1080P or anything like that with software decoding.

> (3) How much testing would it need? Or is it the kind of feature or code
> that either works or it doesn't across all phones running a particular OS?

If Mochitests pass I'd be confident the code is has been exercised enough to say it's ok. I've run the tests on my Nexus S and there are failures at the moment.

> (1) Is it also likely to be easy to get software decoding working for
> Honeycomb (since we'll be supporting tablets soon)?   If so, when can we
> have a patch for that support?

When I last looked at video playback on Honeycomb it was similar to ICS so I think we can get it working there without too much trouble. I'll test on the Galaxy Tab.

> (2) Does it make sense to try to get software decoding working for Froyo?  I
> imagine that's trickier since libstagefright didn't stabilize until GB.  Do
> we have a sense of how much work it would be?

I have not looked at Froyo at all. Finding a device running it can be problematic. I'd prefer to concentrate on Gingerbread and above for now. If someone has a test device I can do builds for them to try until we get one ourselves.

> (3) Do we have a sense of how long it will take to get hardware decoding
> support across most of the market?  Do we know yet if hardware decoding
> support is a per phone effort or if phones running the same chipset are
> likely to work once we get one phone using that chipset working?

At the moment we don't have reliable hardware decoding working on B2G, where we control everything. Once Edwin's new code can work against mozilla central we can look at what's needed to get that working on Android. Failing that, it's a matter of investigating why the hardware decoder is not working in the current code. Edwin did some work here a while back on getting the SGS2 to do hardware decoding so we should check with him. 

Edwin, what was the state of that code you did to extract the data returned by the hardware decoder?

> We (Mozilla) need to decide if the Android team needs to be brainstorming a
> "Plan B" interim solution or if software decoding will likely be sufficient
> given the time we have available.

A "Plan B" of "spawn the built in media player when user clicks a video" might be wise. We could use this on Froyo and older in any case.

> We also may want to open a separate bug on "software platform decoders,"
> especially if we will be releasing software decoding support separately from
> hardware decoding support (seems likely) and most especially if software
> platform decoders become the interim solution for Flash disappearing from
> the Google Play Store.

I thought I'd morph bug 759945 into software decoding. Get that landed so people needing to work on H.264 (Shumway) have something to use and do a followup for hardware. Does that sound reasonable?

I have some build issues to deal with:

1) Can we work around having to have an Android OS source checkout for each android version we need to support
2) Can we work around requiring binary .so files for each Android OS version we want to support

I'm sure we can resolve these but for the first iteration I've gone for specifying paths to the binary .so files and the Android OS source as configure flags.

Comment 11

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Maire Reavy [:mreavy] from comment #8)
> With Android Flash going away from the Google Play store next month (Aug
> 15), we (Mozilla) need to decide if software decoding using libstagefright
> could be used as an interim solution until the hardware decoding solution is
> ready. (BTW, I was reminded today that Android doesn't run on the train
> system yet, and even if it did, we'd fast-track an interim solution to this
> problem.)  
Not sure where you heard this, but Firefox for Android runs on the trains.

Comment 12

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Chris Double (:doublec) from comment #10)
> I have not looked at Froyo at all. Finding a device running it can be
> problematic. I'd prefer to concentrate on Gingerbread and above for now. If
> someone has a test device I can do builds for them to try until we get one
> ourselves.
We've got those devices. If you post builds, we can get them tested. Also, we'll ship you a phone to test yourself.


> A "Plan B" of "spawn the built in media player when user clicks a video"
> might be wise. We could use this on Froyo and older in any case.

I've got a proof of concept that composites in a SurfaceView that a MediaPlayer can play to. I was kinda hoping not to have to continue down that path though.

FWIW, we don't care about anything older than Froyo. So if we get Froyo software decoding working, there shouldn't be a need for a Plan B

> I have some build issues to deal with:
> 
> 1) Can we work around having to have an Android OS source checkout for each
> android version we need to support
In the past we've imported select hearders or portions of headers into the tree. Most of the AOSP is Apache licensed, so it is compatible with MPL3

> 2) Can we work around requiring binary .so files for each Android OS version
> we want to support

Are you thinking we'll pull them into the tree? That shouldn't be a problem if they are built from AOSP, right?

Comment 13

[cajbir (:cajbir)]

(In reply to Brad Lassey [:blassey] from comment #12)
> We've got those devices. If you post builds, we can get them tested. Also,
> we'll ship you a phone to test yourself.

Great, if you can ship one to the Auckland office that would be excellent.

> FWIW, we don't care about anything older than Froyo. So if we get Froyo
> software decoding working, there shouldn't be a need for a Plan B

Ok, I'll work on getting something going on Froyo.

> In the past we've imported select hearders or portions of headers into the
> tree. Most of the AOSP is Apache licensed, so it is compatible with MPL3

Yep, that's my plan.

> Are you thinking we'll pull them into the tree? That shouldn't be a problem
> if they are built from AOSP, right?

I was hoping to avoid that. I'd previously tried building stub shared libraries just for linking against but had issues with exporting vtables correctly. I'll have another quick try at resolving that. I hadn't thought of building binaries from AOSP source directly and bundling those rather than getting them from the phone. I'll check that out, thanks.

Comment 14

[Maire Reavy [:mreavy]]

(In reply to Brad Lassey [:blassey] from comment #11)
> (In reply to Maire Reavy [:mreavy] from comment #8)
> > With Android Flash going away from the Google Play store next month (Aug
> > 15), we (Mozilla) need to decide if software decoding using libstagefright
> > could be used as an interim solution until the hardware decoding solution is
> > ready. (BTW, I was reminded today that Android doesn't run on the train
> > system yet, and even if it did, we'd fast-track an interim solution to this
> > problem.)  
> Not sure where you heard this, but Firefox for Android runs on the trains.

I don't want to sidetrack the discussions in this bug, but this seems important to clarify (probably outside this bug, but I'm open on where this happens):

Jonath announced in the Mozilla weekly meeting this week (the one at 11am Pacific, Mondays) that Firefox for Android does not run on the train system yet.  He said something like (I'm probably paraphrasing, but this is what I remember him saying), "If someone tells you that Firefox Android is currently running on the trains, they are wrong. We are trying to get there, but we're not there now and won't be for a while. I'll let you know when we get there."

If Jonath's info is wrong/stale, then someone should correct this misunderstanding.  If I misinterpreted what Jonath said or meant, that would be good/important to clarify (because I'm not alone).  Thanks.

Comment 15

[Maire Reavy [:mreavy]]

Hey Chris (Double) - Could you provide an update on where this stands and how much progress we've made on this in the last week?  I know we're having to juggle B2G support and software decoding on Android (since Flash for Android goes away from the Google Play Store in about 2 weeks).

Some specific questions are below (as follow up), but I think any/all news is useful and appreciated. 

If it makes more sense to put the update for this in Bug 759945, that's cool by me.  Please just add a short note to this bug saying that you're putting the update there (759945) since not everyone following this bug is copied on that bug.  (I realize they probably want to be.)  Thanks!

(In reply to Chris Double (:doublec) from comment #13)
> (In reply to Brad Lassey [:blassey] from comment #12)
> > We've got those devices. If you post builds, we can get them tested. Also,
> > we'll ship you a phone to test yourself.
> 
> Great, if you can ship one to the Auckland office that would be excellent.

Did you get a phone?


> > FWIW, we don't care about anything older than Froyo. So if we get Froyo
> > software decoding working, there shouldn't be a need for a Plan B
> 
> Ok, I'll work on getting something going on Froyo.

How is that looking?

Is there anything you're blocked on?  Is there anything you'd like the Android guys to do or to help with?

Comment 16

[cajbir (:cajbir)]

(In reply to Maire Reavy [:mreavy] from comment #15)
> Hey Chris (Double) - Could you provide an update on where this stands and
> how much progress we've made on this in the last week?  

I worked on getting the patch in a state where it doesn't need a path to the Android OS source or to pull binaries from the device to build so that I can use try server for doing Mochitests.

I now include the required OS header source. I built versions of the required shared libraries from the AOSP (Android Open Source Platform) directly and include those in the patch. I'm pretty sure we can redistribute those if needed. This might be a bit easier than getting people to pull binaries from the phone.

For landing what's there at the moment I'll probably stick with the "provide a path to binaries" as a configure option then it could get some wider use. Later we can re-visit the previous approach we tried of building stub libraries that we can link against. I'm not treating that as a priority since we can just use the binaries for now.

I've been working on getting Mochitests passing. This shows a few playback issues - video playback stalling for example. Most of my time is spent in investigating these issues.

Once I've solved the stalled playback issue I'll be looking at getting Gingerbread working.

> > Great, if you can ship one to the Auckland office that would be excellent.
> 
> Did you get a phone?

I have not got a froyo phone yet so nothing has been done for this platform.

> Is there anything you're blocked on?  Is there anything you'd like the
> Android guys to do or to help with?

Currently not blocked on anything. I'll be getting in touch with the Android guys directly to discuss any help they can provide.

Comment 17

[Maire Reavy [:mreavy]]

Re-purposing this bug to act as a meta bug for tracking Platform decoder support on Android.

Comment 18

[Maire Reavy [:mreavy]]

Adding dependency on GB.  Also adding the one Honeycomb bug that I saw in bugzilla (Bug 786248).  We may decide not to support Honeycomb, but I wanted to add this bug so it doesn't get forgotten.

Comment 19

[Brad Lassey [:blassey] (use needinfo?)]

minus because we don't track meta bugs

Comment 20

[Nils Ohlmeier [:drno]]

Mass closing do to inactivity.
Feel free to re-open if still needed.