Support unlisted codecs when decoding via the MediaCodec NDK on Android
Categories
(Core :: Audio/Video: Playback, enhancement)
Tracking
()
People
(Reporter: aosmond, Assigned: aosmond)
References
(Blocks 1 open bug)
Details
On Android, some codecs can reach Gecko and be decoded by the platform MediaCodec even though they have no MediaCodec entry in MediaCodecsSupport.h. Today these only work through the Java AndroidDecoderModule, which falls back to matching the raw MIME type against the decoders the platform reports (restored for the Java path in bug 2067689). The MediaCodec NDK path through the FFmpeg PDM (ffvpx) rejects all of them, because GetCodecId in FFmpegVideoDecoder/FFmpegAudioDecoder has no mapping for them. The Java PDM is not available in isolated processes, so these formats stop playing once we rely on the NDK path.
How they reach Gecko:
- HLS (
media.hls.enabled, on by default on Android):GeckoHlsAudioRenderer/GeckoHlsVideoRenderersupportsFormat()accept any format the device has a MediaCodec decoder for, andHLSDemuxerpasses the Media3sampleMimeTypestraight to Gecko. - Our MP4 demuxer:
video/mp4v-es(MPEG-4 Part 2) for plain<video src>playback without acodecsparameter.
Codecs to consider, roughly by priority:
- E-AC-3, including Atmos/JOC (
audio/eac3,audio/eac3-joc): HLS TS/fMP4. OEM/licensed decoders, common. No upstream FFmpeg MediaCodec decoder. - AC-3 (
audio/ac3): HLS TS/fMP4. OEM/licensed, common. No upstream FFmpeg MediaCodec decoder. - AC-4 (
audio/ac4): HLS fMP4. Some newer devices. No upstream FFmpeg MediaCodec decoder. - Dolby Vision (
video/dolby-vision): HLSdvh1/dvhe. Some devices. No upstream FFmpeg MediaCodec decoder. - MPEG-4 Part 2 (
video/mp4v-es): MP4 withoutcodecs, HLS TS. AOSP software decoder. Upstream FFmpeg hasmpeg4_mediacodec. - MPEG-2 video (
video/mpeg2): HLS TS. AOSP software decoder. Upstream FFmpeg hasmpeg2_mediacodec. - MPEG audio Layer I/II (
audio/mpeg-L1,audio/mpeg-L2): HLS TS. Device dependent. No upstream FFmpeg MediaCodec decoder. - Lower priority: DTS family (
audio/vnd.dts,audio/vnd.dts.hd), H.263 (video/3gpp), AMR-NB/WB (audio/3gpp,audio/amr-wb).
Work needed:
- For each codec we want: a MIME type to
AVCodecIDmapping in the FFmpeg PDM, the matching*_mediacodecdecoder enabled in ffvpx (patchingmediacodecdec.cwhere upstream has none), and a check that the device actually has a decoder. MediaCodecsSupportedis anEnumSetbacked byuint64_twith five entries per codec, and the eleven existing codecs already use 55 bits. Adding more than one codec requires widening it, or not allocating the unused hardware and lack-of-extension entries for audio codecs.- Make the HLS renderers'
supportsFormat()only accept formats Gecko can decode, so ExoPlayer picks a playable rendition (e.g. AAC rather than E-AC-3) instead of one that then fails to decode. This is worth doing independently of 1. - Tests: none of these formats are covered.
dom/media/test/sample.3gpandsample.3g2contain MPEG-4 Part 2 video but are skipped everywhere becausevideo/3gppis not an accepted container type. Add Android-only mochitests for at least MPEG-4 Part 2 in MP4 and an HLS stream with E-AC-3 audio.
Device availability of these decoders varies by OEM; codec lists from a few reference devices would help prioritise.
| Assignee | ||
Comment 1•1 month ago
|
||
It appears Chrome doesn't support any of these codecs. So maybe we can safely deprecate them with Java decoding?
Description
•