Open Bug 2083178 Opened 1 month ago Updated 1 month ago

Broken seeking in MSE on Android with H265/VP9 MP4 missing the stss box

Categories

(Core :: Audio/Video: Playback, defect, P3)

defect

Tracking

()

UNCONFIRMED

People

(Reporter: tnibler, Unassigned, NeedInfo)

References

Details

These very specific circumstances pose problems with tests added in bug 2072130. Both on Android and WMF MediaEngine, playing the stream from 0:00 works but seeking to any other point makes decoding stall.
It's plausible that whatever happens with WMF is out of our control, but the Android issue merits investigation.

Note that the MP4 file is not spec compliant, so this is more of a compatibility issue and Firefox is not wrong strictly speaking. The missing stss box in the means that all frames are considered as sync (=keyframes) when in reality they are not.
MP4TrackDemuxer already doesn't rely on information from the container to avoid problems like this. Instead it inspects NAL unit contents with the appropriate decoder. So there's either something subtly wrong with that analysis (somewhat unlikely), or a bug somewhere else.

Thanks for the report.

Analysis

On Android and Windows, MP4TrackDemuxer does not check the NAL units of HEVC samples. The HEVC keyframe check in MP4Demuxer.cpp is inside #ifdef MOZ_APPLEMEDIA. Bug 2049615 added it for macOS only. H.264 samples get the NAL check on all platforms, so only HEVC has this problem.

With no sync table and no sample flags, every sample has the container keyframe flag. TrackBuffersManager keeps that flag. A seek then starts decoding on a non-IRAP picture. The file has only one IDR, so the MediaCodec decoder and the MF decoder probably output no frame after the seek. The WMF MediaEngine path uses the same flag through MFSampleExtension_CleanPoint in MFMediaEngineStream.cpp, so it has the same cause.

endOfStream() only changes how the failure shows. Without it, MediaFormatReader waits for more data after the drain, and the seek stalls. With it, the seek completes at the end of the stream.

Fix direction

Run the existing HEVC check (H265::IsKeyFrame or H265::IsRandomAccessPoint) on all platforms in MP4TrackDemuxer::GetNextSample(). Keep the container flag when the sample does not parse. Use the same policy that bug 2072130 chooses for H.264. The "single-idr-no-stss" test in bug 2072130 can cover this change.

:jolin, could you please confirm that MediaCodec can start decoding after a seek on a CRA picture? This decides if Android can use IRAP pictures as keyframes, or only IDR pictures.


To give feedback on this comment, add a thumbs-up or thumbs-down reaction.

Severity: -- → S3
Flags: needinfo?(jolin)
Priority: -- → P3
See Also: → 2072130, 2049615

Same thing observed with VP9, not consistently though, the test occasionally passes (example). It appears to happen more often without endOfStream, but even with the hand occurs.

Summary: Broken seeking in MSE on Android with H265 MP4 missing the stss box, only if endOfStream not called → Broken seeking in MSE on Android with H265/VP9 MP4 missing the stss box
You need to log in before you can comment on or make changes to this bug.