Open Bug 2078311 Opened 1 month ago Updated 1 month ago

Support unlisted codecs when decoding via the MediaCodec NDK on Android

Categories

(Core :: Audio/Video: Playback, enhancement)

All
Android
enhancement

Tracking

()

ASSIGNED

People

(Reporter: aosmond, Assigned: aosmond)

References

(Blocks 1 open bug)

Details

On Android, some codecs can reach Gecko and be decoded by the platform MediaCodec even though they have no MediaCodec entry in MediaCodecsSupport.h. Today these only work through the Java AndroidDecoderModule, which falls back to matching the raw MIME type against the decoders the platform reports (restored for the Java path in bug 2067689). The MediaCodec NDK path through the FFmpeg PDM (ffvpx) rejects all of them, because GetCodecId in FFmpegVideoDecoder/FFmpegAudioDecoder has no mapping for them. The Java PDM is not available in isolated processes, so these formats stop playing once we rely on the NDK path.

How they reach Gecko:

  • HLS (media.hls.enabled, on by default on Android): GeckoHlsAudioRenderer/GeckoHlsVideoRenderer supportsFormat() accept any format the device has a MediaCodec decoder for, and HLSDemuxer passes the Media3 sampleMimeType straight to Gecko.
  • Our MP4 demuxer: video/mp4v-es (MPEG-4 Part 2) for plain <video src> playback without a codecs parameter.

Codecs to consider, roughly by priority:

  • E-AC-3, including Atmos/JOC (audio/eac3, audio/eac3-joc): HLS TS/fMP4. OEM/licensed decoders, common. No upstream FFmpeg MediaCodec decoder.
  • AC-3 (audio/ac3): HLS TS/fMP4. OEM/licensed, common. No upstream FFmpeg MediaCodec decoder.
  • AC-4 (audio/ac4): HLS fMP4. Some newer devices. No upstream FFmpeg MediaCodec decoder.
  • Dolby Vision (video/dolby-vision): HLS dvh1/dvhe. Some devices. No upstream FFmpeg MediaCodec decoder.
  • MPEG-4 Part 2 (video/mp4v-es): MP4 without codecs, HLS TS. AOSP software decoder. Upstream FFmpeg has mpeg4_mediacodec.
  • MPEG-2 video (video/mpeg2): HLS TS. AOSP software decoder. Upstream FFmpeg has mpeg2_mediacodec.
  • MPEG audio Layer I/II (audio/mpeg-L1, audio/mpeg-L2): HLS TS. Device dependent. No upstream FFmpeg MediaCodec decoder.
  • Lower priority: DTS family (audio/vnd.dts, audio/vnd.dts.hd), H.263 (video/3gpp), AMR-NB/WB (audio/3gpp, audio/amr-wb).

Work needed:

  1. For each codec we want: a MIME type to AVCodecID mapping in the FFmpeg PDM, the matching *_mediacodec decoder enabled in ffvpx (patching mediacodecdec.c where upstream has none), and a check that the device actually has a decoder.
  2. MediaCodecsSupported is an EnumSet backed by uint64_t with five entries per codec, and the eleven existing codecs already use 55 bits. Adding more than one codec requires widening it, or not allocating the unused hardware and lack-of-extension entries for audio codecs.
  3. Make the HLS renderers' supportsFormat() only accept formats Gecko can decode, so ExoPlayer picks a playable rendition (e.g. AAC rather than E-AC-3) instead of one that then fails to decode. This is worth doing independently of 1.
  4. Tests: none of these formats are covered. dom/media/test/sample.3gp and sample.3g2 contain MPEG-4 Part 2 video but are skipped everywhere because video/3gpp is not an accepted container type. Add Android-only mochitests for at least MPEG-4 Part 2 in MP4 and an HLS stream with E-AC-3 audio.

Device availability of these decoders varies by OEM; codec lists from a few reference devices would help prioritise.

It appears Chrome doesn't support any of these codecs. So maybe we can safely deprecate them with Java decoding?

You need to log in before you can comment on or make changes to this bug.