[blcked issue] sound is cut off while in MT/coming call

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
5 years ago
6 months ago

People

(Reporter: oedipus31, Unassigned)

Tracking

(Depends on: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

5 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36

Steps to reproduce:

STR.
1. play "Cuttherope" on the LEO
2. play any level as you wish.
3. press home button
4. Play any mp3 file by music application
5. while mp3 is playing by the music application, receive any call.

then, music is cut off frequently.


Actual results:

MP3 file is should not be cut off.

By 1.0.1 version of FFOS, I cannot not detect same problem because it uses different mechanism to using H/W codec.
But in 1.1 version of FFOS, the problem have occurred so frequently.



Expected results:

The audio stream should not be cut off.
Component: General → AudioChannel

Comment 1

5 years ago
At the time STR [3], Cuttherope app doesn't release MP3 H/W codec.

So, Music application uses S/W codec to play MP3 file.
When it receives call, two S/W codec decodes data simultaneously.
Then the sound is cut off.

The problem doesn't occur when Music application uses H/W codec.

Updated

5 years ago
blocking-b2g: --- → leo?

Updated

5 years ago
blocking-b2g: leo? → leo+

Comment 2

5 years ago
Sotaro,

I have tested this STR on 1.0.1 but I cannot find same problem on it.
(There's no same application "Cuttherope", so I checked by another one which uses MP3 H/W codec)

When I test it on 1.0.1, all application except Music uses S/W codec.
So, Music application always uses H/W codec, then the problem doesn't occur.

Do you have any idea why it doesn't use H/W codec?

Updated

5 years ago
Flags: needinfo?(sotaro.ikeda.g)
Marco,

Can you look at this?
Flags: needinfo?(mchen)

Comment 4

5 years ago
(In reply to hjlee7111 from comment #2)
> When I test it on 1.0.1, all application except Music uses S/W codec.
> So, Music application always uses H/W codec, then the problem doesn't occur.
> 
> Do you have any idea why it doesn't use H/W codec?

In 1.0.1, Gecko didn't use IOMX to perform H/W codec and all processes do OMX decoding on it's own process.
So only web app with correct permission can access the H/W codec (with root) this is why others application can just uses S/W codec.
Flags: needinfo?(mchen)

Comment 5

5 years ago
(In reply to Seil Park from comment #0)
> Steps to reproduce:
> 
> STR.
>...
> then, music is cut off frequently.
> 

Hi Leo,

May I know the detail of this assumption?
   I think the correct behavior here is "music will be paused then only ringer is firing".
   Is your behavior about music will be played intermittently during the ringer is firing?

Thanks.
As mchen already explained in Comment 4.
- v1.01. all hw codecs are instantiated in app process.
- To instantiate hw codecs in app process, app process needs a special linux level permission.
  Music app has this permission.
- Since v1.1, hw codecs are loaded in android's media server.
  By the change, app process does not need the special linux level permission.
  Any app can use hw codec.
- Hw codec can be get by an first app try to get the hw codec.
Flags: needinfo?(sotaro.ikeda.g)
Music playback needs a hw codec to extend the music playback duration. To realize it, app needs to inform to gecko that the audio is for 'Music playback'.
(In reply to Sotaro Ikeda [:sotaro] from comment #7)
> Music playback needs a hw codec to extend the music playback duration. To
> realize it, app needs to inform to gecko that the audio is for 'Music
> playback'.

mchen, do you have any idea about this? One idea is to add music audio channel type. How do you think about it?
Flags: needinfo?(mchen)

Comment 9

5 years ago
(In reply to Marco Chen [:mchen] (Summit 10/03 ~10/08) from comment #5)
>    I think the correct behavior here is "music will be paused then only
> ringer is firing".
>    Is your behavior about music will be played intermittently during the
> ringer is firing?
Yes, exactly.

In this case, Music and ringer are overlapped for a while (1~3 seconds).
And the music stammers in the overlapping period.
(Because two S/W codec handle its data simultaneously)

The best behavior is "music will be paused then only ringer is firing" as you said.
But if it's hard to fix or if it needs a lot of change, 
at least, the music should not stammers while it's overlapped with ringer.

Comment 10

5 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #8)
> (In reply to Sotaro Ikeda [:sotaro] from comment #7)
Hi Sotaro,

>> Music playback needs a hw codec to extend the music playback duration.
Do you mean we need HW Codec to enhance the battery life of music playback because HW codec's current consumption is small then SW codec? 

> mchen, do you have any idea about this? One idea is to add music audio
> channel type. How do you think about it?
If my assumption above is correct then you need a mechanism like priorities between media elements then media element wit higher priority can preempt low priority one. Is this what you want?

Thanks.
Flags: needinfo?(mchen)

Comment 11

5 years ago
(In reply to hjlee7111 from comment #9)
> In this case, Music and ringer are overlapped for a while (1~3 seconds).
The problem here is discussed on Bug 908525 so we will ignore it in the discussion here.

> But if it's hard to fix or if it needs a lot of change, 
> at least, the music should not stammers while it's overlapped with ringer.
I am not sure why SW codec will cause this issue but HW codec not. Will take a look at it first.

Thanks for your clarification.

Comment 12

5 years ago
Created attachment 817720 [details] [diff] [review]
Patch: For test purpose. Let normal channel not join AudioChannelService


Hi Leo,

Could you help to test this patch on your side with your reproduced steps? Thanks. I tried to flash our own gecko only but device can't boot into home screen.

I am afraid that this is not related to decode by S/W or H/W codec but related to IPC between AudioChannelAgent and AudioChannelService. 
  1. Once there are many audio element with normal channel created by cuttherope in the background.
  2. There is a music player playing in the foreground.
  3. MT call is received and ringer is fired.
  4. All of audio elements are starting to negotiate with AudioChannelService via blocking IPC call.
  5. The music player in the background had low priority so it is effected by many IPC calls.
  6. The ringer in the foreground has more higher priority so it didn't be effected.

In this patch I let audio elements with normal channel (from cuttherope) not to join AudioChannelService. This can ignore the IPC calls and prove my thought.

Thanks.
(In reply to Marco Chen [:mchen] (Summit 10/03 ~10/08) from comment #10)
> >> Music playback needs a hw codec to extend the music playback duration.
> Do you mean we need HW Codec to enhance the battery life of music playback
> because HW codec's current consumption is small then SW codec? 

Yes. hw codec is low power consumption than sw codec. And in android, qcom provides low power audio mode for music playback. In this mode, audio buffer depth is changed to longer to save cpu wake up. It seems not supported on current b2g. But in future, it becomes important.

> > mchen, do you have any idea about this? One idea is to add music audio
> > channel type. How do you think about it?
> If my assumption above is correct then you need a mechanism like priorities
> between media elements then media element wit higher priority can preempt
> low priority one. Is this what you want?

In video codec's case, preemption works. In video codec's case, number of the instance is normally one or so. But in audio codec case, preemption do not work correctly. Multiple audio codecs could exist and basically any instance could play sound any timing even when app is in background.
(In reply to Sotaro Ikeda [:sotaro] from comment #13)
> Yes. hw codec is low power consumption than sw codec. And in android, qcom
> provides low power audio mode for music playback. In this mode, audio buffer
> depth is changed to longer to save cpu wake up. It seems not supported on
> current b2g. But in future, it becomes important.

If there is a music channel type, music channel's audio buffer size can be changed to longer in gecko.

Comment 15

5 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> If there is a music channel type, music channel's audio buffer size can be
> changed to longer in gecko.

In b2g18 branch using libsydney, I added a code for changing the audio buffer size to more large. And that is based on AudioChannelType. I assume that content type is mapped to music_stream so I set it's buffer larger.

http://mxr.mozilla.org/mozilla-b2g18/source/media/libsydneyaudio/src/sydney_audio_gonk.cpp#179

But I didn't look into libcubeb _ openSL backend yet.

Comment 16

5 years ago
Hi Leo,

Could you help the comment on comment 12? Thanks.
Flags: needinfo?(leo.bugzilla.gecko)
(Reporter)

Comment 17

5 years ago
(In reply to Marco Chen [:mchen] (Summit 10/03 ~10/08) from comment #16)
> Hi Leo,
> 
> Could you help the comment on comment 12? Thanks.
I tested your patch and found the issue is reproduced. It can`t be helpful by this patch. 
From the precondition point of view, 
1. Music used SW decoder, not HW decoder.
2. Music and ringer was overlapped in the foreground state.

Comment 18

5 years ago
(In reply to Seil Park from comment #17)
> (In reply to Marco Chen [:mchen] (Summit 10/03 ~10/08) from comment #16)
> I tested your patch and found the issue is reproduced. It can`t be helpful
> by this patch. 
> From the precondition point of view, 

Hi,

Thanks for your help to verify with this patch.
And sorry for I didn't give the correct point to you for what is expected to prove by this patch.

The purpose of this patch is that try to eliminate the issue about 
    "at least, the music should not stammers while it's overlapped with ringer."

Is this patch workable for the issue above? Thanks.

> 1. Music used SW decoder, not HW decoder.
Currently this is a limitation.
> 2. Music and ringer was overlapped in the foreground state.
Bug 908525
Flags: needinfo?(leo.bugzilla.gecko) → needinfo?(oedipus31)

Comment 19

5 years ago
(In reply to Marco Chen [:mchen] from comment #18)
> The purpose of this patch is that try to eliminate the issue about 
>     "at least, the music should not stammers while it's overlapped with
> ringer."
> 
> Is this patch workable for the issue above? Thanks.

FYI, there's not any progress after your patch.
It still has same problem.
 
> 1. Music used SW decoder, not HW decoder.
> Currently this is a limitation.
Yes, right.
But is it possible that later app uses HW decoder by force?
It might be helpful.

Thanks

Comment 20

5 years ago
Thanks for your verification.

> But is it possible that later app uses HW decoder by force?
> It might be helpful.

Or the paused media element in background should always release the HW Codec. Will investigate it.
(Reporter)

Updated

5 years ago
Flags: needinfo?(oedipus31)
Created attachment 820325 [details] [diff] [review]
temporary patch - Always instantiate omx codec in android mediaserver

This is a temporary patch to instantiate omx codec in mediaserver.

In the early day's of b2g v1.1, all omx codecs are instantiated in android's mediaserver. After that, only audio sw codecs are moved to a content process by Bug 864180. It did not fixed the ture cause of the problem. The problem had happened by incorrectly low thread priority. It is fixed by Bug 882552. Therefore, the current gecko should not have a problem even when audio codecs are instantiated in android mediaserver.
Seil, can you try attachment 820325 [details] [diff] [review] if it works for the problem?
Flags: needinfo?(oedipus31)

Comment 23

5 years ago
Hi Sotaro,

This is a good point for this issue.

By the way, I had considered to raise up media threads's priority and this can generally solve the background audio fell into underrun issue. The threads may include decoding thread (ex: ogg) and audioloop thread. In your patch even we move the decoding thread to media server for higher priority, the AudioLoop thread is still on low priority which can still cause the underrun. And if the background music is an ogg type this issue could be reproduced again.

Refer to Android design, all media play threads are running on the media server so their priorities are the same. May I know your suggestion for this?
(Reporter)

Comment 24

5 years ago
(In reply to Sotaro Ikeda [:sotaro] from comment #22)
> Seil, can you try attachment 820325 [details] [diff] [review] if it works
> for the problem?

Hi Sotaro,
I tested attachment 820325 [details] [diff] [review] , This issue was improved. but if there is some other task in background it sometimes could be reproduced. 
Before this patch, If use sw decoder 100% issue is reproduced.
After this patch, the issue is reproduced when It has high cpu usage.

Marco`s opinion(audioloop, ogg type) is right. but we are like to exclude the music use ogg file type because of the low usability.
We should find the row risk and the easy case.
Flags: needinfo?(oedipus31)
(Reporter)

Comment 25

5 years ago
Hi Marco
I don`t know whether this proposal is approved by the quality management department. but I think this is the most safe way. 
And the more safe way is "Cut the rope"game release the resource for using HW codec if suspend state. Even though "Cut the rope" game is 3rd party , but this is builtin app, not downloaded app, I think the game should follow this logic.
(In reply to Marco Chen [:mchen] (Business Trip 10/21 ~ 10/23) from comment #20)
> > But is it possible that later app uses HW decoder by force?
> > It might be helpful.
> 
> Or the paused media element in background should always release the HW
> Codec. Will investigate it.

I do not think it is a good idea. I just make the usage of hw audio codec unpredictable and do not solve the specific use case 100%.
(In reply to Seil Park from comment #24)
> (In reply to Sotaro Ikeda [:sotaro] from comment #22)
> Marco`s opinion(audioloop, ogg type) is right. but we are like to exclude
> the music use ogg file type because of the low usability.
> We should find the row risk and the easy case.

For a longer term, it is necessary. But it is a non trivial task to change it and evaluate it.

Updated

5 years ago
Depends on: 932701

Comment 28

5 years ago
This bug doesn't have any update for a while. Do we really still mark this bug with leo+? Thank you.
Flags: needinfo?(wchang)
(In reply to Sotaro Ikeda [:sotaro] from comment #27)
> (In reply to Seil Park from comment #24)
> > (In reply to Sotaro Ikeda [:sotaro] from comment #22)
> > Marco`s opinion(audioloop, ogg type) is right. but we are like to exclude
> > the music use ogg file type because of the low usability.
> > We should find the row risk and the easy case.
> 
> For a longer term, it is necessary. But it is a non trivial task to change
> it and evaluate it.


Sotaro,

Is this something that we will still experience and therefore need to consider a fix for (maybe 1.5)?
blocking-b2g: leo+ → ---
Flags: needinfo?(wchang) → needinfo?(sotaro.ikeda.g)
Wayne, the problem is still present. Bug 932701 seems to fix the problem. I am not sure about the timeline.
Flags: needinfo?(sotaro.ikeda.g)

Comment 31

3 years ago
[Blocking Requested - why for this release]:

[Tracking Requested - why for this release]:

[Blocking Requested - why for this release]:
blocking-b2g: --- → spark?
tracking-b2g: --- → backlog

Updated

3 years ago
blocking-b2g: spark? → ---
tracking-b2g: backlog → ---

Comment 32

6 months ago
Firefox OS is not being worked on
Status: UNCONFIRMED → RESOLVED
Last Resolved: 6 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.