945134 - [flatfish] Play Mp4 (H.264 + AAC) again no video frame output after playing to the end-of-file, but audio is no problem.

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Gaia::Video, defect, P1)

Description

[tommy_chiu]

Attachment: demo.mp4 is a normal mp4 (H.264+AAC) file.

1. Lunch Video App.
2. Play a mp4 file.
3. Play to the end-of-file.
4. Press "play" button.
No video frame output, but audio is Okay.

Comment 1

[Vincent Liu[:vliu]]

Nexus-4 (JB 4.2.2) works normally and can't reproduce this problem. Comment it as reference.

Comment 2

[Jeff Chuang]

Attachment: Invoking hidePlayer(true) to stop codec is a workaround for this issue

After checked the flow on Android image on flatfish, the player will stop the codec ever time it finishes the playback. On flatfish, it will encounter errors on back to 1st frame if we do not stop codec after playback. This solution will take as a workaround. Galaxy Note 8 has the similar behavior and it back to thumbnail list view when finishing playback.

Comment 3

[Sotaro Ikeda [:sotaro]]

What is an actual cause of the problem? Hw codec's limitation?

Comment 4

[Sotaro Ikeda [:sotaro]]

If it is a hw codec's limitation, even if we fixed the problem by fixing the gaia side, some third parth apps could fall into this problem.

Comment 5

[tommy_chiu]

I use software codec, and it has the same issue. 

##### software codec patch ####
--- a/media_codecs.xml
+++ b/media_codecs.xml
@@ -91,7 +91,7 @@ Only the three quirks included above are recognized at this point:
 <MediaCodecs>
     <Decoders>
-        <MediaCodec name="OMX.allwinner.video.decoder.avc" type="video/avc" />
+        <MediaCodec name="OMX.google.h264.decoder" type="video/avc" />

--- a/content/media/omx/OmxDecoder.cpp
+++ b/content/media/omx/OmxDecoder.cpp
@@ -484,7 +501,8 @@ bool OmxDecoder::AllocateMediaResources()
     // is currently used, and for formats this code is currently used
     // for (h.264).  So if we don't get a hardware decoder, just give
     // up.
-    int flags = kHardwareCodecsOnly;
+    int flags = kSoftwareCodecsOnly;
 
     char propQemu[PROPERTY_VALUE_MAX];
     property_get("ro.kernel.qemu", propQemu, "");

###### software codec patch ######

Comment 6

[Marco Chen [:mchen]]

Hi tommy,

Q1: If you pause the video in the middle by yourself, could video be continued to play?

Q2: If you pause the video in the middle and seek to another time slot could video be continued to play?

PS: Did your BSP was already modified for OMXCodec::Pause()?

Comment 7

[tommy_chiu]

Q1: If you pause the video in the middle by yourself, could video be continued to play?
==> yes.

Q2: If you pause the video in the middle and seek to another time slot could video be continued to play?
==> yes.

PS: Did your BSP was already modified for OMXCodec::Pause()?
==> yes.

When I play the video, if it doesn't reach to the end of the file, the seek and the pause action are OK. If it reaches to the end of the file, the video could be decoded anymore.

Comment 8

[tommy_chiu]

Sorry, Correct some 
Q1: If you pause the video in the middle by yourself, could video be continued to play?
==> yes.

Q2: If you pause the video in the middle and seek to another time slot could video be continued to play?
==> yes.

PS: Did your BSP was already modified for OMXCodec::Pause()?
==> yes.

When I play the video, if it doesn't reach to the end of the file, the seek and the pause action are OK. If it reaches to the end of the file, the video could NOT be decoded anymore.

Comment 9

[John Hu [:johnhu][:johu][:醬糊小弟]]

(In reply to Jeff Chuang from comment #2)
> Created attachment 8345072 [details] [diff] [review]
> Invoking hidePlayer(true)  to stop codec is a workaround for this issue

Sorry for jumping in. This workaround will causes another regression and will not work correctly in landscape mode of flatfish because we need to keep player on the screen and have the preview loaded. We changed to pause() because of bug 921421.

Comment 10

[Bruce Sun [:brsun]]

Attachment: big_buck_bunny.mp4

This file is downloaded from: http://www.quirksmode.org/html5/videos/big_buck_bunny.mp4

Comment 11

[Bruce Sun [:brsun]]

Attachment: software_patch.tar.gz

Just update my testing result.

While playing demo.mp4 and big_buck_bunny.mp4 by using the hardware decoder (with the modification of OMXCodec::pause() only), the video and the audio both play well at the first time. If I click the play button after end-of-stream to play it again, the audio can playback without problems, but the video always keeps the same as the first video frame. If I try to seek to other position in this case, the audio can play well as usual, but the video changes to the video frame at the seek time and keeps still.

I've try to apply the suggested patches to use the software decoder OMX.google.h264.decoder, and I have some slightly different results. While playing demo.mp4, the audio output is good but the video is always blank, no matter end-of-stream has been encountered before or not. While playing big_buck_bunny.mp4, both the audio and the video play very well without any problems.

The followings are modifications of my source codes:

============================================================
path: frameworks/av
based on rev: a974242677d2f8bfeb5ffcdd0319a847910a2069
------------------------------------------------------------
diff --git a/media/libstagefright/OMXCodec.cpp b/media/libstagefright/OMXCodec.cpp
index 1c6e8b4..6921c56 100755
--- a/media/libstagefright/OMXCodec.cpp
+++ b/media/libstagefright/OMXCodec.cpp
@@ -4523,6 +4523,7 @@ void OMXCodec::initOutputFormat(const sp<MetaData> &inputFormat) {
 
 status_t OMXCodec::pause() {
     Mutex::Autolock autoLock(mLock);
+    return ERROR_UNSUPPORTED;
 
     mPaused = true;
============================================================

============================================================
path: device/allwinners/common
based on rev: d03484f2a4dbcd3b039ed0ed739f6c7ad4a664e9
------------------------------------------------------------
diff --git a/media_codecs.xml b/media_codecs.xml
index e349411..f7dd3f4 100644
--- a/media_codecs.xml
+++ b/media_codecs.xml
@@ -91,7 +91,7 @@ Only the three quirks included above are recognized at this point:
 -->
 <MediaCodecs>
     <Decoders>
-        <MediaCodec name="OMX.allwinner.video.decoder.avc" type="video/avc" />
+        <MediaCodec name="OMX.google.h264.decoder" type="video/avc" />
         <MediaCodec name="OMX.allwinner.video.decoder.mpeg2" type="video/mpeg2" />
                <MediaCodec name="OMX.google.mpeg4.decoder" type="video/mp4v-es" />
                <MediaCodec name="OMX.google.h263.decoder" type="video/3gpp" />
============================================================

============================================================
path: gecko
based on rev: 163883:9bcc52594322 (mozilla-central on hg)
------------------------------------------------------------
diff --git a/content/media/omx/OmxDecoder.cpp b/content/media/omx/OmxDecoder.cpp
--- a/content/media/omx/OmxDecoder.cpp
+++ b/content/media/omx/OmxDecoder.cpp
@@ -519,17 +519,17 @@ bool OmxDecoder::AllocateMediaResources(
     mNativeWindowClient = new GonkNativeWindowClient(mNativeWindow);
 #endif
 
     // Experience with OMX codecs is that only the HW decoders are
     // worth bothering with, at least on the platforms where this code
     // is currently used, and for formats this code is currently used
     // for (h.264).  So if we don't get a hardware decoder, just give
     // up.
-    int flags = kHardwareCodecsOnly;
+    int flags = kSoftwareCodecsOnly;
 
     char propQemu[PROPERTY_VALUE_MAX];
     property_get("ro.kernel.qemu", propQemu, "");
     if (!strncmp(propQemu, "1", 1)) {
       // If we are in emulator, allow to fall back to software.
       flags = 0;
     }
     mVideoSource =
============================================================

The source codes of gaia were the same as rev: fecb15413917f16938a4963157584568d7d19017 without modifications.

Comment 12

[Bruce Sun [:brsun]]

(In reply to tommy_chiu from comment #5)
> I use software codec, and it has the same issue. 
> 

Would you please try to play big_buck_bunny.mp4 on your environment to see if we have the same results?

> ##### software codec patch ####
> --- a/media_codecs.xml
> +++ b/media_codecs.xml
> @@ -91,7 +91,7 @@ Only the three quirks included above are recognized at
> this point:
>  <MediaCodecs>
>      <Decoders>
> -        <MediaCodec name="OMX.allwinner.video.decoder.avc" type="video/avc"
> />
> +        <MediaCodec name="OMX.google.h264.decoder" type="video/avc" />
> 
> --- a/content/media/omx/OmxDecoder.cpp
> +++ b/content/media/omx/OmxDecoder.cpp
> @@ -484,7 +501,8 @@ bool OmxDecoder::AllocateMediaResources()
>      // is currently used, and for formats this code is currently used
>      // for (h.264).  So if we don't get a hardware decoder, just give
>      // up.
> -    int flags = kHardwareCodecsOnly;
> +    int flags = kSoftwareCodecsOnly;
>  
>      char propQemu[PROPERTY_VALUE_MAX];
>      property_get("ro.kernel.qemu", propQemu, "");
> 
> ###### software codec patch ######

Comment 13

[Joe Young]

Use Allwinner H.264 Hardware Codec on flatfish.
It has some problem when playbackended.

END_OF_STREAM(EOS) Event cannot report to Gecko Player(PlaybackEnded) that make Player run on Playing status.

Why EOS is not trigger by OMXCodec !?
Because SignalEOS is set to true then emptyBuffer will push OMX_BUFFERFLAH_ENDOFFRAME(flag) to the InputBuffer.
But it doesn't return OMX_BUFFERFLAG_EOS in OutputBuffer, so on_message::FILL_BUFFER_DONE will not set mNoMoreOutputData to true.

It look like that Allwinner hardware codec has some issue that block last two OutputBuffer.

In this case we should enable hardware EOS in OMXCodec:

OMXCodec.cpp:2458
#if 0
        case OMX_EventBufferFlag:
        {
            CODEC_LOGV("EVENT_BUFFER_FLAG(%ld)", data1);

            if (data1 == kPortIndexOutput) {
                mNoMoreOutputData = true;
            }
            break;
        }
#endif
OMXCodec.cpp:2468

Change "#if 0" to "#if 1".


One more thing:

Allwinner OMX IL Component has its own eos flag.
eos will set to OMX_TRUE when it get OMX_BUFFERFLAG_EOS.

There is only a way to set eos to OMX_FALSE.
Changet Allwinner IL Component StateMachine from Other State to OMX_StateExecuting.

What's wrong with this:
If eos is set to OMX_TRUE when the first playback is ended.

1. 
on Android Player will be killed when the first playback is ended. 
when Player is killed Allwinner IL Component StateMachine is set to OMX_StateIdle.
the second playback will change Allwinner IL Component StateMachine from OMX_StateIdle to OMX_StateExecuting.
eos is set to OMX_FALSE.

2.
on B2G Player will NOT killed when the first playback is ended.
The Allwinner IL Component StateMachine will stay in OMX_StateExecuting.
the second playback will stop immediately because eos is OMX_TRUE.

According to this, we should disable eos usage in Allwinner IL Component.
It doesn't matter because we enable hardware eos in AOSP OMXCodec.

Comment 14

[Joe Young]

There is related bug.
See more detail in bug 104648 comment #14

Comment 15

[Joe Young]

(In reply to Joe Young from comment #14)
> There is related bug.
> See more detail in bug 104648 comment #14

Sorry, type error.

See more detail in bug 1040648 comment #14

Comment 16

[Bruce Sun [:brsun]]

clear the needinfo request

Comment 17

[BMO Automation]

Firefox OS is not being worked on