1530234 - Android data decoder takes over 2s to create audio decoder

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Jean-Yves Avenard [:jya]]

In bug 1222851 we implemented support for audio gapless.

However, when you're playing test file https://jyavenard.github.io/htmltests/tests/mse_mp4/gapless/1222851.html you can see the audio stopping every 1s for close to 2s.

Adding some logs we see:

02-25 01:06:24.210 10381 22893 I RemoteDataDecoderModule:
02-25 01:06:24.236 10381 22843 I AndroidDecoderModule: CreateAudioFormat with mimeType=audio/mp4a-latm, mRate=44100, channels=2
02-25 01:06:26.376 10381 22878 I TrimmerDecoderModule: sample[5976779,5999999] has trimming info ([6000000,6000001]

It takes here 2.14s to create the AAC decoder. This makes for a very audible audio silence.

What's surprising however is this is intermittent. Sometimes it's super quick and the test page behaves properly.

Comment 1

[Jean-Yves Avenard [:jya]]

Not this issue will be seen with normal playback too like if the audio content change. A 2s gap will be heard

Comment 2

[Jean-Yves Avenard [:jya]]

:jhlin

In this particular case I can check if only the AudioInfo has changed between video and avoid recreating the audio decoder, but still this is particularly limiting.

We don't want an audio gap of 2s if you're changing from stereo to 5.1 for example.

Comment 3

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1530234 - Ignore init segment for audio only buffer if audio content didn't change. r!bryce

This prevents re-creating a new audio decoder which on Android can take an awful long time.

Comment 4

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1530234 - P1. Ignore init segment for audio only buffer if audio content didn't change. r=bryce

This prevents re-creating a new audio decoder which on Android can take an awful long time.

Comment 5

[John Lin [:jhlin][:jolin]]

It looks like the remote decoding service/process gets restarted because of audio decoder recreation so it takes much longer to create a new decoder. Although we can just add a delay for the service unbinding action to avoid these kinds of use cases, I think we should try integrating Android remote decoder into the RDD mechanism introduced in bug 1471535.

Comment 6

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1530234 - P2. Don't recalculate audio sample start time. r=bryce!

The WMF audio decoder recalculated the timestamp of each audio sample according to the number of frames decoded so far.
This is incompatible with the trimming mechanism that rely on the timestamps of the audio to be matching what is found in the container.

All the other audio decoders do it that way already.

Depends on D20969

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0bd5d8684436
P1. Ignore init segment for audio only buffer if audio content didn't change. r=bryce
https://hg.mozilla.org/integration/autoland/rev/4e5ac450f2b6
P2. Don't recalculate audio sample start time. r=bryce

Comment 8

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/0bd5d8684436
https://hg.mozilla.org/mozilla-central/rev/4e5ac450f2b6