Implement OpenMax IL AAC audio decoding client.

RESOLVED FIXED in Firefox 45

Status

()

defect
P2
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: ayang, Assigned: ayang)

Tracking

(Depends on 1 bug)

unspecified
mozilla45
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox45 fixed)

Details

Attachments

(2 attachments, 6 obsolete attachments)

No description provided.
Posted patch add_openmax_il_headers (obsolete) — Splinter Review
Posted patch omx_pdm (obsolete) — Splinter Review
Posted patch test_enable_patch (obsolete) — Splinter Review
test patch which enable omx-il PDM.
Blocks: 1224889
Attachment #8687626 - Flags: review?(jyavenard)
Attachment #8687626 - Flags: feedback?(sotaro.ikeda.g)
Attachment #8687627 - Flags: review?(jyavenard)
Attachment #8687627 - Flags: feedback?(sotaro.ikeda.g)
Comment on attachment 8687626 [details] [diff] [review]
add_openmax_il_headers

Review of attachment 8687626 [details] [diff] [review]:
-----------------------------------------------------------------

should remove all the trailing spaces.
Attachment #8687626 - Flags: review?(jyavenard) → review+
Comment on attachment 8687627 [details] [diff] [review]
omx_pdm

Review of attachment 8687627 [details] [diff] [review]:
-----------------------------------------------------------------

Some comments and questions.

It would be nice to allocate hw codec in media server and to allocat sw codec in application process. It could affect to the performace of apps that uses a lot of audio files like games. It could minimize IPC overhead and memory usage and was done on OmxDecoder in Bug 864180. In audio codec case, we did it in OmxDecoder::AllocateMediaResources()
  https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.cpp#299

Dont't we need to check Compontnt Quirks? We need to use mOMX->allocateBufferWithBackup() instead of mOMX->useBuffer() in some cases. Compontnt Quirks are in OMXCodec::getComponentQuirks().
  http://androidxref.com/6.0.0_r1/xref/frameworks/av/media/libstagefright/OMXCodec.cpp#255

The patch seems not set OMX_IndexParamAudio*** parameters like OMX_IndexParamAudioAmr and OMX_IndexParamAudioAmr. It seems better to set them as same level as OMXCodec to prevent unintended regression.

Do we have a way to set OMX IL component to set to OMX_StatePause? On code aurora platform, it is necessary to set to OMX_StatePause reduce power consumption when audio playback is in paused state. It is added to nsMediaOmxReader in Bug 860760.
(In reply to Sotaro Ikeda [:sotaro] from comment #5)
> Do we have a way to set OMX IL component to set to OMX_StatePause? On code
> aurora platform, it is necessary to set to OMX_StatePause reduce power
> consumption when audio playback is in paused state. It is added to
> nsMediaOmxReader in Bug 860760.

NB: aosp OMXNodeInstance does not handle OMX_StatePause state. It cause a crash on aosp. b2g caf fixed it in
Bug 871018.
Comment on attachment 8687626 [details] [diff] [review]
add_openmax_il_headers

Review of attachment 8687626 [details] [diff] [review]:
-----------------------------------------------------------------

We usually put 3rd party libraries under media/ like media/openmax_dl.
(In reply to JW Wang [:jwwang] from comment #7)
> Comment on attachment 8687626 [details] [diff] [review]
> add_openmax_il_headers
> 
> Review of attachment 8687626 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> We usually put 3rd party libraries under media/ like media/openmax_dl.

hmmm that's what he did. He put them in media/openmax_il ; or you mean remove the 3rd level of directory?
Comment on attachment 8687627 [details] [diff] [review]
omx_pdm

Review of attachment 8687627 [details] [diff] [review]:
-----------------------------------------------------------------

Will continue the remaining files tomorrow. in the mean time...

::: dom/media/platforms/moz.build
@@ +26,5 @@
>      'wrappers/FuzzingWrapper.cpp',
>      'wrappers/H264Converter.cpp'
>  ]
>  
> +DIRS += ['agnostic/gmp',

DIRS += [
    'agnostic/gmp
...
]

::: dom/media/platforms/omx/GonkOmxPlatformLayer.cpp
@@ +28,5 @@
> +extern void GetPortIndex(nsTArray<uint32_t>& aPortIndex);
> +
> +class GonkOmxObserver : public BnOMXObserver {
> +public:
> +  void onMessage(const omx_message& aMsg) {

{ on following lines

::: dom/media/platforms/omx/OmxDataDecoder.cpp
@@ +170,5 @@
> +  // According to the definition of Flush() in PDM:
> +  // "the decoder must be ready to accept new input for decoding".
> +  // So it needs to wait for the Omx to complete the flush command.
> +  MonitorAutoLock lock(mMonitor);
> +  MOZ_ASSERT(!mFlushPromiseRequest.Exists());

If you're only aim is to start a task that will perform something and then wait, using a promise isn't the most appropriate object, especially as InvokeAsync is rather heavy handed (it creates another promise)

Better start a task setting a boolean while waiting for that boolean to complete.
Promise is for performing asynchronous task, here you want a synchronous one

@@ +235,5 @@
> +  LOG("DoAsyncShutdown");
> +
> +  {
> +    MonitorAutoLock lock(mMonitor);
> +    mFlushPromiseRequest.DisconnectIfExists();

this can't happen.
Flush() is called (which is synchronous) followed by Shutdown()

@@ +266,5 @@
> +           });
> +}
> +
> +void
> +OmxDataDecoder::CheckIfNeedNotifyInputExhaust()

CheckIfInputExhausted()

Exhaust has an entirely different meaning :)

@@ +268,5 @@
> +
> +void
> +OmxDataDecoder::CheckIfNeedNotifyInputExhaust()
> +{
> +  if (!mCheckInputExhaust) {

you arrive here because the mirror has set it to true.

should assert as it would indicate a logic error (see below why)

@@ +272,5 @@
> +  if (!mCheckInputExhaust) {
> +    return;
> +  }
> +
> +  mCheckInputExhaust = false;

I'm not sure I follow the logic nor why you need a watcher. mCheckInputExhaust is only ever modified or read on the OMX task queue. So why use a watcher? A watcher is designed to synchronise two threads mirroring a variable.

It would also trigger an immediate call to CheckIfNeedNotifyInputExhausted which would exit early.

@@ +346,5 @@
> +  }
> +
> +  // If the latest decoded sample's MediaRawData is also the latest input sample,
> +  // it means there is no input data in queue and component, calling InputExhaust.
> +  if (data->mRawData == mLatestInputRawData) {

to me this shouldn't be done in FillBufferDone.

It should be done in FillBufferDone's caller

@@ +644,5 @@
> +
> +template<class T> void
> +OmxDataDecoder::InitOmxParameter(T* aParam)
> +{
> +  memset(aParam, 0, sizeof(T));

Use PodZero

@@ +650,5 @@
> +  aParam->nVersion.s.nVersionMajor = 1;
> +}
> +
> +bool
> +OmxDataDecoder::IfPortBuffersCanBeRelease(OMX_DIRTYPE aType)

Don't like that name, but in any case: CanBeReleased
I would remove If too

::: dom/media/platforms/omx/OmxDataDecoder.h
@@ +155,5 @@
> +  MozPromiseHolder<InitPromise> mInitPromise;
> +
> +  // It is accessed in Omx/reader TaskQeueu.
> +  // TODO: should we use Mirror here for multithread accessing?
> +  bool mShutdown;

it is accessed on multiple thread, with no appropriate lock. Make this atomic or with an associated monitor.
(In reply to Jean-Yves Avenard [:jya] from comment #8)
> (In reply to JW Wang [:jwwang] from comment #7)
> > Comment on attachment 8687626 [details] [diff] [review]
> > add_openmax_il_headers
> > 
> > Review of attachment 8687626 [details] [diff] [review]:
> > -----------------------------------------------------------------
> > 
> > We usually put 3rd party libraries under media/ like media/openmax_dl.
> 
> hmmm that's what he did. He put them in media/openmax_il ; or you mean
> remove the 3rd level of directory?

Sorry, my bad. I thought it was dom/media/...
(In reply to Sotaro Ikeda [:sotaro] from comment #5)
> Comment on attachment 8687627 [details] [diff] [review]
> omx_pdm
> 
> Review of attachment 8687627 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Some comments and questions.
> 
> It would be nice to allocate hw codec in media server and to allocat sw
> codec in application process. It could affect to the performace of apps that
> uses a lot of audio files like games. It could minimize IPC overhead and
> memory usage and was done on OmxDecoder in Bug 864180. In audio codec case,
> we did it in OmxDecoder::AllocateMediaResources()
>  
> https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.
> cpp#299

hmm... Do you suggest sw codec should have higher priority than hw codec in audio decoding?

> 
> Dont't we need to check Compontnt Quirks? We need to use
> mOMX->allocateBufferWithBackup() instead of mOMX->useBuffer() in some cases.
> Compontnt Quirks are in OMXCodec::getComponentQuirks().
>  
> http://androidxref.com/6.0.0_r1/xref/frameworks/av/media/libstagefright/
> OMXCodec.cpp#255

Will do.

> 
> The patch seems not set OMX_IndexParamAudio*** parameters like
> OMX_IndexParamAudioAmr and OMX_IndexParamAudioAmr. It seems better to set
> them as same level as OMXCodec to prevent unintended regression.

Will do.

> 
> Do we have a way to set OMX IL component to set to OMX_StatePause? On code
> aurora platform, it is necessary to set to OMX_StatePause reduce power
> consumption when audio playback is in paused state. It is added to
> nsMediaOmxReader in Bug 860760.

There is no pause api in MediaDataDecoder. When should we enter the pause mode? Or using a timer to wait a period of time when there is no more input data?
(In reply to Jean-Yves Avenard [:jya] from comment #9)
> Comment on attachment 8687627 [details] [diff] [review]
> omx_pdm
> 
> Review of attachment 8687627 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Will continue the remaining files tomorrow. in the mean time...

Thanks for comments, I'll address them.
> > 
> > It would be nice to allocate hw codec in media server and to allocat sw
> > codec in application process. It could affect to the performace of apps that
> > uses a lot of audio files like games. It could minimize IPC overhead and
> > memory usage and was done on OmxDecoder in Bug 864180. In audio codec case,
> > we did it in OmxDecoder::AllocateMediaResources()
> >  
> > https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.
> > cpp#299
> 
> hmm... Do you suggest sw codec should have higher priority than hw codec in
> audio decoding?

No, hw codec should have higher priority. When we load sw codec, it need to be loaded in application process instead of mediaserver process.

> > Do we have a way to set OMX IL component to set to OMX_StatePause? On code
> > aurora platform, it is necessary to set to OMX_StatePause reduce power
> > consumption when audio playback is in paused state. It is added to
> > nsMediaOmxReader in Bug 860760.
> 
> There is no pause api in MediaDataDecoder. When should we enter the pause
> mode? Or using a timer to wait a period of time when there is no more input
> data?

It seems related to implementation of MediaFormatReader. jya seems to have a better idea than me.
(In reply to Alfredo Yang (:alfredo) from comment #11)

> There is no pause api in MediaDataDecoder. When should we enter the pause
> mode? Or using a timer to wait a period of time when there is no more input
> data?

The client of the MediaDataDecoder doesn't even know itself that it's in paused mode.

Why not go into low power mode whenever the queue is empty or the decoder's buffer are empty?

otherwise a timer is probably the way to go. Someone would need to implement it regardless. Better be in the MediaDataDecoder
(In reply to Jean-Yves Avenard [:jya] from comment #14)
> (In reply to Alfredo Yang (:alfredo) from comment #11)
> 
> > There is no pause api in MediaDataDecoder. When should we enter the pause
> > mode? Or using a timer to wait a period of time when there is no more input
> > data?
> 
> The client of the MediaDataDecoder doesn't even know itself that it's in
> paused mode.
> 
> Why not go into low power mode whenever the queue is empty or the decoder's
> buffer are empty?

It is just codeaurora's implementation. I asked same question to them before.
(In reply to Sotaro Ikeda [:sotaro] from comment #15)
> > The client of the MediaDataDecoder doesn't even know itself that it's in
> > paused mode.
> > 
> > Why not go into low power mode whenever the queue is empty or the decoder's
> > buffer are empty?
> 
> It is just codeaurora's implementation. I asked same question to them before.

It might be better to check if this problem is still exist on latest codeaurora platform.
Priority: -- → P2
(In reply to Jean-Yves Avenard [:jya] from comment #9)
> Comment on attachment 8687627 [details] [diff] [review]
> omx_pdm
> 
> Review of attachment 8687627 [details] [diff] [review]:
> -----------------------------------------------------------------
> @@ +346,5 @@
> > +  }
> > +
> > +  // If the latest decoded sample's MediaRawData is also the latest input sample,
> > +  // it means there is no input data in queue and component, calling InputExhaust.
> > +  if (data->mRawData == mLatestInputRawData) {
> 
> to me this shouldn't be done in FillBufferDone.
> 
> It should be done in FillBufferDone's caller

I fixed others except for this.
This is called by promise resolved in omx callback so there is no caller in this case.
Posted patch omx_pdm (obsolete) — Splinter Review
Updated patch according to previous review/feedback comments.
Attachment #8687627 - Attachment is obsolete: true
Attachment #8687627 - Flags: review?(jyavenard)
Attachment #8687627 - Flags: feedback?(sotaro.ikeda.g)
Attachment #8690631 - Flags: review?(jyavenard)
Attachment #8690631 - Flags: feedback?(sotaro.ikeda.g)
Attachment #8687626 - Flags: feedback?(sotaro.ikeda.g) → feedback+
Comment on attachment 8690631 [details] [diff] [review]
omx_pdm

Review of attachment 8690631 [details] [diff] [review]:
-----------------------------------------------------------------

Looks better! The followings are some comments.

::: dom/media/platforms/omx/GonkOmxPlatformLayer.cpp
@@ +223,5 @@
> +}
> +
> +nsresult
> +GonkOmxPlatformLayer::Shutdown()
> +{

Isn't it better to shutdown OMX il component gracefully like OMX_StateIdle->OMX_StateExecuting ->OMX_StateLoaded.

@@ +270,5 @@
> +    }
> +  }
> +
> +  // fallback to sw codec
> +  err = mOmx->allocateNode(swcomponent, mOmxObserver, &mNode);

This code seems not allocate sw codec in app process.
In OmxDecoder case, OMX instance is created in app process by calling GetOMX() to allocate the sw codec in app process.
 https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.cpp#95
 https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.cpp#312

@@ +302,5 @@
> +OMX_ERRORTYPE
> +GonkOmxPlatformLayer::SendCommand(OMX_COMMANDTYPE aCmd,
> +                                  OMX_U32 aParam1,
> +                                  OMX_PTR aCmdData)
> +{

Don't we need to handle kRequiresFlushCompleteEmulation quirk?

::: dom/media/platforms/omx/OmxDataDecoder.cpp
@@ +567,5 @@
> +OmxDataDecoder::ConfigAudioCodec()
> +{
> +  const AudioInfo* audioInfo = mTrackInfo->GetAsAudioInfo();
> +  OMX_ERRORTYPE err;
> +

It might be better to add a comment why it handle only aac. mp4+amr recording was used also on fxos produces.

@@ +811,5 @@
> +  // There is no 'Drain' API in OpenMax, so it needs to wait for output sample
> +  // with EOS flag. However, MediaRawData doesn't provide EOS information,
> +  // so here it generates an empty BufferData with eos OMX_BUFFERFLAG_EOS in queue.
> +  // This behaviour should be compliant with spec, I think...
> +  RefPtr<BufferData> inbuf = FindAvailableBuffer(OMX_DirInput);

Can we always get valid pointer here?

::: dom/media/platforms/omx/OmxPromiseLayer.h
@@ +41,5 @@
> +  OmxPromiseLayer(TaskQueue* aTaskQueue, OmxDataDecoder* aDataDecoder);
> +
> +  class BufferData;
> +
> +  typedef nsTArray<RefPtr<BufferData>> BUFFERDATA;

'BUFFERDATA' name seems confusing. Can it be changed to a name that shows it is array?
Depends on: 1228183
(In reply to Sotaro Ikeda [:sotaro] from comment #19)
> Comment on attachment 8690631 [details] [diff] [review]
> omx_pdm
> 
> Review of attachment 8690631 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks better! The followings are some comments.
> 
> ::: dom/media/platforms/omx/GonkOmxPlatformLayer.cpp
> @@ +223,5 @@
> > +}
> > +
> > +nsresult
> > +GonkOmxPlatformLayer::Shutdown()
> > +{
> 
> Isn't it better to shutdown OMX il component gracefully like
> OMX_StateIdle->OMX_StateExecuting ->OMX_StateLoaded.

Done.

> 
> @@ +270,5 @@
> > +    }
> > +  }
> > +
> > +  // fallback to sw codec
> > +  err = mOmx->allocateNode(swcomponent, mOmxObserver, &mNode);
> 
> This code seems not allocate sw codec in app process.
> In OmxDecoder case, OMX instance is created in app process by calling
> GetOMX() to allocate the sw codec in app process.
>  https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.
> cpp#95
>  https://dxr.mozilla.org/mozilla-central/source/dom/media/omx/OmxDecoder.
> cpp#312

IOMX uses software codec after KK, but it's worth to do if we want to enable try test on ICS.
I'll do it when enable ICS test later.

> 
> @@ +302,5 @@
> > +OMX_ERRORTYPE
> > +GonkOmxPlatformLayer::SendCommand(OMX_COMMANDTYPE aCmd,
> > +                                  OMX_U32 aParam1,
> > +                                  OMX_PTR aCmdData)
> > +{
> 
> Don't we need to handle kRequiresFlushCompleteEmulation quirk?

It is for some components don't handle flush command well when all buffers are returned to client already.
I will check the buffer status in OmxPromiseLayer when receiving flush command.

> 
> ::: dom/media/platforms/omx/OmxDataDecoder.cpp
> @@ +567,5 @@
> > +OmxDataDecoder::ConfigAudioCodec()
> > +{
> > +  const AudioInfo* audioInfo = mTrackInfo->GetAsAudioInfo();
> > +  OMX_ERRORTYPE err;
> > +
> 
> It might be better to add a comment why it handle only aac. mp4+amr
> recording was used also on fxos produces.

Yes, I plan to enable other format in following bugs. Maybe I should change title of this bug to "Implement an omx client for AAC decoding".

> 
> @@ +811,5 @@
> > +  // There is no 'Drain' API in OpenMax, so it needs to wait for output sample
> > +  // with EOS flag. However, MediaRawData doesn't provide EOS information,
> > +  // so here it generates an empty BufferData with eos OMX_BUFFERFLAG_EOS in queue.
> > +  // This behaviour should be compliant with spec, I think...
> > +  RefPtr<BufferData> inbuf = FindAvailableBuffer(OMX_DirInput);
> 
> Can we always get valid pointer here?

No. This is buggy code. I'll fix it.

Thank you for catching this.

> 
> ::: dom/media/platforms/omx/OmxPromiseLayer.h
> @@ +41,5 @@
> > +  OmxPromiseLayer(TaskQueue* aTaskQueue, OmxDataDecoder* aDataDecoder);
> > +
> > +  class BufferData;
> > +
> > +  typedef nsTArray<RefPtr<BufferData>> BUFFERDATA;
> 
> 'BUFFERDATA' name seems confusing. Can it be changed to a name that shows it
> is array?

Done.
Posted patch omx_pdm (obsolete) — Splinter Review
Update patch according to sotaro's comments.
Attachment #8690631 - Attachment is obsolete: true
Attachment #8690631 - Flags: review?(jyavenard)
Attachment #8690631 - Flags: feedback?(sotaro.ikeda.g)
Attachment #8692804 - Flags: review?(jyavenard)
Attachment #8692804 - Flags: feedback?(sotaro.ikeda.g)
Comment on attachment 8692804 [details] [diff] [review]
omx_pdm

Review of attachment 8692804 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM, I would split however OmxDataDecoder between an audio and video part. The code is very big and difficult to follow otherwise.

::: dom/media/platforms/omx/OmxDataDecoder.cpp
@@ +279,5 @@
> +  if (mMediaRawDatas.Length()) {
> +    return;
> +  }
> +
> +  // When all input buffers are not in omx component, it means all samples have

have been fed

@@ +289,5 @@
> +  }
> +
> +  // When all output buffers are held by component, it means client is waiting for output.
> +  for (auto buf : mOutPortBuffers) {
> +    if (buf->mStatus == BufferData::BufferStatus::OMX_COMPONENT) {

if (buf->mStatus != BufferData::BufferStatus::OMX_COMPONENT) {
  return;
}

avoid the unnecessary continue
Attachment #8692804 - Flags: review?(jyavenard) → review+
Comment on attachment 8692804 [details] [diff] [review]
omx_pdm

Looks good!
Attachment #8692804 - Flags: feedback?(sotaro.ikeda.g) → feedback+
Thanks for review! sotaro and jya.
Attachment #8687626 - Attachment is obsolete: true
Attachment #8687645 - Attachment is obsolete: true
Attachment #8692804 - Attachment is obsolete: true
Attachment #8694193 - Flags: review+
Attachment #8694195 - Attachment is patch: true
Keywords: checkin-needed
Blocks: 1229360
Blocks: 1229361
https://hg.mozilla.org/mozilla-central/rev/0bc40a9a5804
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
Hi sheriff, my both patches are mis-backed out due to bug 1228339 and check in again with an empty one.
Could you please check in my patches again?
Thank you.
Flags: needinfo?(cbook)
Flags: needinfo?(cbook)
Posted patch omx_pdmSplinter Review
fix on emulator-ics, emulator-l.
Attachment #8694195 - Attachment is obsolete: true
Attachment #8695656 - Flags: review+
Both patches didn't check in last time. Re-check in.
Status: RESOLVED → REOPENED
Keywords: checkin-needed
Resolution: FIXED → ---
Bah, wrong bug. Please ignore my comment above.
checkin done by nigel
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/6ea4c940c066
https://hg.mozilla.org/mozilla-central/rev/71ee1fa3d127
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
Summary: Implement OpenMax IL audio decoding client. → Implement OpenMax IL AAC audio decoding client.
Duplicate of this bug: 1207969
You need to log in before you can comment on or make changes to this bug.