829409 - [Web API - AudioChannelManager] Give a Way for Joining AudioChannelService by Application Context not MediaElement

Status: RESOLVED INVALID

(Firefox OS Graveyard :: General, defect)

Description

[Marco Chen [:mchen]]

Currently we have a mechanism on Gecko to handle audio competing issue between different media element. But according to different applications will have different requirement and there is a limitation on current mechanism (media element based). We need to provide a WEB API for application joining AudioChannelService by itself and design the behavior based on it's own requirement.

Ex: The music app would want to occupy the content channel all the time when it plays a list of songs. But the current media element based design will loss release the content channel during song by song. If music app can joining the service by application context then this can be resolved.

Comment 1

[Marco Chen [:mchen]]

Hi Jonas,

We had discussed about this feature. From your point of view, do you think this should be done in next milestone?

Thanks.

Comment 2

[Marco Chen [:mchen]]

Hi Jonas,

May I know any comments from you?
Thanks.

Comment 3

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Yeah, I think this is something we should add. Adding a

occupyChannel(DOMString channel);

function on AudioChannelManager might work? A better name would be nice though :)

Comment 4

[Marco Chen [:mchen]]

Attachment: Part 1: WebAPI-WIPv1

Dear Steven,

Could you help to give a feedback for this new change on audiochannelmanager since you are the author of it?

Thanks.

Comment 5

[Marco Chen [:mchen]]

Attachment: Part 1: WebAPI-WIPv1

Steven Lee is on vacation so look for feedback from Janos.
Thanks.

Comment 6

[Marco Chen [:mchen]]

Hi Jonas (:sicking),

Could you help to give feedback on this bug if you are available? Thanks.

Comment 7

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Comment on attachment 707396 [details] [diff] [review]
Part 1: WebAPI-WIPv1

Review of attachment 707396 [details] [diff] [review]:
-----------------------------------------------------------------

I updated https://wiki.mozilla.org/WebAPI/AudioChannels to what I think we should use. Please coordinate between you, Randy and Andrea who implements what. And please provide feedback if you think the API still needs work.

Comment 8

[Marco Chen [:mchen]]

(In reply to Jonas Sicking (:sicking) from comment #7)
> Comment on attachment 707396 [details] [diff] [review]
> Part 1: WebAPI-WIPv1
> And please provide feedback if you think the API still needs work.

Hi Jonas,

The setOccupyChannel() on Wiki returns void only and it means that app has no way to know the status of this channel (ex: allow to play or pause). 

I would like to propose that setOccupyChannel() return a nsIXXXCallback function to app. Thus app can know whether this channel is allowed to play or not. Then the Gecko can focus on the work of muting channel and make sure our Audio Competing Policy. To remain other additional works to app is applied by this new Web API. This can make AudioChannel more flexible.

Advantage:
  1. This API can be a general way to let app know whether it's audio is muted or not by AudioChannel. Then we can remove events in each audio related Web Interfaces (ex: HTMLMediaElement and maybe FMRadio).

  2. For any requirement that is not only mute the audio but also others, we can leave this work to app by using this new API. Ex: FM app would like to disable FM in some case but only mute it.

Thanks.

Comment 9

[Andrea Marchesini [:baku]]

Couple of questions:

1. What's the difference between occipiedChannels and activeChannels or soundingChannels?

2. 
> between different media element. But according to different applications
> will have different requirement and there is a limitation on current
> mechanism (media element based).

Can you describe better these requirements? And this limitations?

> Ex: The music app would want to occupy the content channel all the time when
> it plays a list of songs. But the current media element based design will
> loss release the content channel during song by song. If music app can
> joining the service by application context then this can be resolved.

I think this is a serious problem: What about if the music app is not playing audio?
Is the content channel in use when there is not media element playing?
If yes I see this:

1. app X starts and occupies the channel C.
2. app X stops playing but the channel is still occupied.
3. app Y starts and wants to use the channel C but it's occupied.

I really would like to see good reasons before adding new methods or properties.

Comment 10

[Andrea Marchesini [:baku]]

> I updated https://wiki.mozilla.org/WebAPI/AudioChannels to what I think we
> should use. Please coordinate between you, Randy and Andrea who implements
> what. And please provide feedback if you think the API still needs work.

I would like to go step by step. We have an API still to be fully implemented.

The new API (part of it has been already reviewed) adds new properties/methods to nsIWindowUtils. I think we should have 1 single object for any audio API. I propose 2 approaches:

1. get rid of AudioChannelManager
currently it manages only headphones, that is not related to audio channels and keep all the audio methods as part of the window or the windowUtils.

2. rename AudioChannelManager to AudioManager and move all that methods there.

Comment 11

[Marco Chen [:mchen]]

(In reply to Andrea Marchesini (:baku) from comment #9)
> Couple of questions:
> 
> 1. What's the difference between occipiedChannels and activeChannels or
> soundingChannels?

The active/soundingChannels represents all audio channels in Gecko.
The occupiedChannels represents audio channel occupied by app only.

> 2. 
> Can you describe better these requirements? And this limitations?
> 

Consider to FMRadio app, currently we just do muting with AudioChannel. In order to save power, the pause will be better. But FMRadio is not just used to output sound but also seeking and setFrequency. If we do pause FMRadio in Gecko, there will be a trouble when app try to do seeking.

In order to solve it and just like HTMLMediaElement, we can add events into FMRadio WebAPI for notifying app about the mute (by gecko) then app can decide whether FMRadio could be paused. There are two issues here

  1. Add additional events into FMRadio WebAPI is just like HTMLMediaElement. This means that we need to add additional events to each WebAPI which is related to AudioChannel. If there are 100 then we need to add these events to 100 WebAPI.

  2. Even we used active/soundingChannels to know whether content channel is muted then stop the FMRadio from app. But AudioChannelAgent with FMRadio is also removed. Then how does App know what time it can resume to play.

> I think this is a serious problem: What about if the music app is not
> playing audio?
> Is the content channel in use when there is not media element playing?
> If yes I see this:
> 
> 1. app X starts and occupies the channel C.
> 2. app X stops playing but the channel is still occupied.
> 3. app Y starts and wants to use the channel C but it's occupied.
> 

I think this is not a problem on API itself but the mistake from app. We can allow that only privileged app can use this kind of API and maybe app can occupy content channel only. This means that we will review that app to avoid this kind of mistake.

Comment 12

[Marco Chen [:mchen]]

> Ex: The music app would want to occupy the content channel all the time when
> it plays a list of songs. But the current media element based design will
> loss release the content channel during song by song. If music app can
> joining the service by application context then this can be resolved.

Also, I think the new web api on Bug 853101 can not solve this issue too.

Comment 13

[Andrea Marchesini [:baku]]

What about comment 9 ?

Comment 14

[Andrea Marchesini [:baku]]

ah wait... ignore my last comment :)

Comment 15

[Andrea Marchesini [:baku]]

> > 1. What's the difference between occipiedChannels and activeChannels or
> > soundingChannels?
> 
> The active/soundingChannels represents all audio channels in Gecko.
> The occupiedChannels represents audio channel occupied by app only.

No. it's per window/iframe. So it's per app.

I don't understand this problem with the FMRadio. Why do we mute when we seek?

> I think this is not a problem on API itself but the mistake from app. We can
> allow that only privileged app can use this kind of API and maybe app can
> occupy content channel only. This means that we will review that app to
> avoid this kind of mistake.

I agree, but this logic must be implemented in any app... if we move it in gecko, this will be transparent.

Can we talk about removing the wakelock from apps?
It seems this the main problem for the music app.

What I don't like is that:

The current Audio API is completely hidden for the apps. They have just to set an attribute in the media elements and the system does the rest. What you are proposing is to move the logic into the apps and to me this seems wrong, or at least something that we have to decide in a meeting or something else.

Comment 16

[Marco Chen [:mchen]]

(In reply to Andrea Marchesini (:baku) from comment #15)
> No. it's per window/iframe. So it's per app.

I mean occupiedChannels just record the one created by setOccupiedChannel() from Gaia JS code.

Ex: An app has 3 audio tag in playing and call setOccupiedChannel() once.
  The active/soundAudioChannels are related to these 3 audio tag and one by setOccuiedChannel().
  The occupiedChannels is related to one created by setOccuiedChannel() only.
 
> I don't understand this problem with the FMRadio. Why do we mute when we
> seek?

Consider the flow as below.

  1. FMRadio app starts to play then to into background.
  2. Music app starts to play in foreground then FMRadio is asked to mute by AudioChannel.
  3. In order to save power, we assume that we pause FMRadio in Gecko.
  4. FMRadio app doesn't know it is paused then try to seek for all available station (maybe triggered by timer for periodical scan). In this case, the seek will be failed but app don't know why.

> 
> Can we talk about removing the wakelock from apps?
> It seems this the main problem for the music app.

We can discuss it on Bug 859824 and leave API discussion here.

> I agree, but this logic must be implemented in any app... if we move it in
> gecko, this will be transparent.
> 
> What I don't like is that:
> 
> The current Audio API is completely hidden for the apps. They have just to
> set an attribute in the media elements and the system does the rest. What
> you are proposing is to move the logic into the apps and to me this seems
> wrong, 

Please refer to link as below and in the last paragraph it mentioned about to make more flexibility by putting some logic into Gaia but all in gecko.
https://bugzilla.mozilla.org/show_bug.cgi?id=855208#c10

>or at least something that we have to decide in a meeting or
> something else.

Right, maybe discuss it on workweek.

Comment 17

[Marco Chen [:mchen]]

This Web API is bacame invalid after a discussion with Andrea, Jonas and Justin so close it as invalid.