Closed Bug 1084464 Opened 10 years ago Closed 6 years ago

Web content needs a way to play audio in the background

Categories

(Firefox OS Graveyard :: AudioChannel, defect, P1)

Product:
Firefox OS Graveyard
Component:
AudioChannel
Platform:
x86
macOS
Type:
defect

Tracking

(tracking-b2g:backlog)

Status:
RESOLVED WONTFIX
Project Flags:
tracking-b2g backlog

People

(Reporter: kgrandon, Unassigned)

References

Details

(Whiteboard: [dependency: marketplace-partners]) 

Spawned from: https://groups.google.com/forum/#!topic/mozilla.dev.gaia/7V9OEBhJOb4

We need some way to allow web content to play music in the background. Currently on other phone OS's they allow user-initiated meda to play in the background. If we want to be able to compete we need to fix problems like this to make the phone usable.
Web content can already do this by sticking mozaudiochannel=content in the markup (I might remember the attribute name wrong). Setting this attribute also causes the music/video apps to automatically mute when the foreground app starts playing music.

Obviously essentially no content on the web does this. Which means it doesn't help users a whole lot. The question i what to do. We could

A) Evangelize some of the big websites to get them to add this attribute.
B) Try to create a web standard for this and hope that that drives more adoption.
C) Change default behavior so that all audio on the web automatically keeps playing even when the user switches away from the website. And maybe also automatically mute the music app.
D) Enable the user to say "this website should continue to play even when I switch away from it".
E) develop some heuristic for what audio should keep playing in the background and what shouldn't.
F) Something else.

B is likely to take quite a while. And also is unlikely to go anywhere until other browser's start muting audio of background tabs. Which to my knowledge none currently do. And sadly we can't do it on Firefox desktop since we can't mute flash which is a bit source of audio from web on desktop.

Note that playing in the background without having the "mute music app" behavior could be quite annoying. This mostly affects option C


Generally speaking, which option to use here is a UX question. Unless we can create a good enough version of E that thing Just Works.
Stephany can you offer some guidance here (or NI someone else).
Flags: needinfo?(swilkes)
(In reply to Jonas Sicking (:sicking) from comment #1)
> C) Change default behavior so that all audio on the web automatically keeps
> playing even when the user switches away from the website. And maybe also
> automatically mute the music app.

This is my favourite option. Why is the Music app an exception to the rule? Wouldn't the most recent 'user-inititated' Media playback trump the last. So a simple example:

1. Play track in Music app -> Music track plays
2. Switch to soundcloud.com and play a track -> Music track pauses, SoundCloud track plays
3 Switch to Music app and play (unpause) track -> SoundCloud track pauses, Music track plays
FWIW soundcloud.com does play in the background on Firefox Android.
(In reply to Wilson Page [:wilsonpage] from comment #3)
> (In reply to Jonas Sicking (:sicking) from comment #1)
> > C) Change default behavior so that all audio on the web automatically keeps
> > playing even when the user switches away from the website. And maybe also
> > automatically mute the music app.
> 
> This is my favourite option. Why is the Music app an exception to the rule?
> Wouldn't the most recent 'user-inititated' Media playback trump the last.

+1. IMO this is the best UX and requires the least amount of evangelism to get our phone usable.
(In reply to Kevin Grandon :kgrandon from comment #5)
> (In reply to Wilson Page [:wilsonpage] from comment #3)
> > (In reply to Jonas Sicking (:sicking) from comment #1)
> > > C) Change default behavior so that all audio on the web automatically keeps
> > > playing even when the user switches away from the website. And maybe also
> > > automatically mute the music app.
> > 
> > This is my favourite option. Why is the Music app an exception to the rule?
> > Wouldn't the most recent 'user-inititated' Media playback trump the last.
> 
> +1. IMO this is the best UX and requires the least amount of evangelism to
> get our phone usable.

+1 here as well. I would argue that both users *and* web devs would expect this to be the default behavior. We shouldn't be requiring web devs to use a '-moz'-prefixed API to get what should be considered standard behavior.
I was on PTO until today. Flagging Katie who is UX on Media.
Flags: needinfo?(swilkes) → needinfo?(kcaldwell)
(In reply to Wilson Page [:wilsonpage] from comment #3)
> (In reply to Jonas Sicking (:sicking) from comment #1)
> > C) Change default behavior so that all audio on the web automatically keeps
> > playing even when the user switches away from the website. And maybe also
> > automatically mute the music app.
> 
> This is my favourite option. Why is the Music app an exception to the rule?
> Wouldn't the most recent 'user-inititated' Media playback trump the last. So
> a simple example:
> 
> 1. Play track in Music app -> Music track plays
> 2. Switch to soundcloud.com and play a track -> Music track pauses,
> SoundCloud track plays
> 3 Switch to Music app and play (unpause) track -> SoundCloud track pauses,
> Music track plays

+1. IMO, this reflects an expected behaviour - play audio from the most current, user-initiated "play/unpause" activity. 

NI'ing Jacqueline from Media/Music - patterns should be consistent with Music app and general FFOS audio/media playing behaviours.
Flags: needinfo?(kcaldwell) → needinfo?(jsavory)
Keep in mind that option C means that if you go play a game which plays background music, when you switch away from that game the background music will keep playing. Or even if it simply plays sound effects as different in-game characters shoots or does whatever, those sound effects will keep playing while the game is in the background.

This not only means that the audio keeps playing. It also means that the game process keeps running. So if you press the power button, the CPU doesn't go to sleep and instead keeps using significant amounts of battery.

The only way for the user to stop the music from playing is to longpress the home button and kill the game in question.

This stuff also gets more complicated when we switch to the full sheet model. In the sheet model each clicked link is like "opening a new tab" rather than remove the current page and load a new page in its place. Does that mean that clicking a link on a page that plays music doesn't stop the music? What if it's not music but an ad?

Yes, audio keeps playing today in background tabs. But quite often that's annoying rather than a good thing. How many times have you had some background tab start playing an ad or other audio and you have to hunt through all your open tabs to figure out where it's coming from?

This is also much worse on mobile where the smaller screen and the lack of keyboard makes it much harder to look through your "open tabs" to try to kill some background audio.

And while the web does generally allow background pages to play audio like normal, this is not how mobile app platforms generally behave. Most apps on Android and iOS are silenced when the user switch away from them.
Also note that the Music App, or any other Gaia app, is no exception here. It too follows the same policies as webpages, and it too needs to explicitly opt in to having its audio keep playing in the background.
(In reply to Wilson Page [:wilsonpage] from comment #3)
> 1. Play track in Music app -> Music track plays
> 2. Switch to soundcloud.com and play a track -> Music track pauses,
> SoundCloud track plays
> 3 Switch to Music app and play (unpause) track -> SoundCloud track pauses,
> Music track plays

Does this mean that if you instead of going to soundcloud.com, you go to some game which plays some "blip blop" sound effects, that those sound effects should cause the music track to pause? Currently the two sound sources mix.
(In reply to Jonas Sicking (:sicking) from comment #9)
> Keep in mind that option C means that if you go play a game which plays
> background music, when you switch away from that game the background music
> will keep playing. Or even if it simply plays sound effects as different
> in-game characters shoots or does whatever, those sound effects will keep
> playing while the game is in the background.
> 
> This not only means that the audio keeps playing. It also means that the
> game process keeps running. So if you press the power button, the CPU
> doesn't go to sleep and instead keeps using significant amounts of battery.
> 

This seems to me to be somewhat invalid. More than likely, the game dev would want to perform some sort of action when the game enters the background (`visibilitychange`) such as "pausing" the game.

I think we need to be trying to accommodate the most reasonable default behavior with regards to background audio. If a developer does nothing more than play audio, it should be understood that it will continue playing if the app goes to the background. This also applies to games. However, as I already stated, the developer would likely want to pause their game if they enter the background and therefore, they'd already be listening for the `visibilitychange` event to begin with. Part of their handling of the `visibilitychange` event could be to stop the audio and stop the game loop.

Also note that `visibilitychange` should have *NOTHING* to do with background audio. I've seen some strange bugs lately where apps in Gaia were not properly receiving `visibilitychange` events and it had to do with the absence of a seemingly irrelevant `audio-channel-content` permission. The developer should expect `visibilitychange` to *ALWAYS* fire when their app enters the background or re-enters the foreground. If they so happen to choose to pause/resume audio to coincide with that event, then so be it. This seems like a much more intuitive way to allow/prohibit an app from playing audio in the background.
(In reply to Jonas Sicking (:sicking) from comment #10)
> Also note that the Music App, or any other Gaia app, is no exception here.
> It too follows the same policies as webpages, and it too needs to explicitly
> opt in to having its audio keep playing in the background.

I also don't think that this is a valid argument. We happen to have what seems to be a "block first" approach to background audio, so of course the built-in Gaia "Music" app is going to implement the necessary permissions/API calls to be capable of playing music in the background.

However, from a developer evangelism perspective, it is ridiculous for us to expect hosted web pages/apps to implement our -moz-specific API calls in order for audio to play in the background. We should be putting our users first, and as part of that, we should be making is as painless as possible for the web devs out there who are creating web content that our users want to consume.
(In reply to Justin D'Arcangelo [:justindarc] from comment #12)
> This seems to me to be somewhat invalid. More than likely, the game dev
> would want to perform some sort of action when the game enters the
> background (`visibilitychange`) such as "pausing" the game.

If that would be the case that'd be great. But if you actually check with most websites that play audio, very few of them do this.

(In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> (In reply to Jonas Sicking (:sicking) from comment #10)
> > Also note that the Music App, or any other Gaia app, is no exception here.
> > It too follows the same policies as webpages, and it too needs to explicitly
> > opt in to having its audio keep playing in the background.
> 
> I also don't think that this is a valid argument. We happen to have what
> seems to be a "block first" approach to background audio, so of course the
> built-in Gaia "Music" app is going to implement the necessary
> permissions/API calls to be capable of playing music in the background.
> 
> However, from a developer evangelism perspective, it is ridiculous for us to
> expect hosted web pages/apps to implement our -moz-specific API calls in
> order for audio to play in the background. We should be putting our users
> first, and as part of that, we should be making is as painless as possible
> for the web devs out there who are creating web content that our users want
> to consume.

I agree that it's really bad that the API is moz-specific. However I actually think that blocking is the much more sensible default. Out of the websites that I visit that play audio, many more of them play audio that I would like to be muted when I switch the website to the background than are music players that I want to play background audio.

I.e. I visit many more ads/games/sound-effect/video-sharing content than music player content.
(In reply to Jonas Sicking (:sicking) from comment #14)
> (In reply to Justin D'Arcangelo [:justindarc] from comment #12)
> > This seems to me to be somewhat invalid. More than likely, the game dev
> > would want to perform some sort of action when the game enters the
> > background (`visibilitychange`) such as "pausing" the game.
> 
> If that would be the case that'd be great. But if you actually check with
> most websites that play audio, very few of them do this.
> 

I understand that `visibilitychange` is probably underutilized, but compared to our -moz-specific APIs, I'd be willing to bet that you'd find more instances of `visibilitychange` being used in the wild.

> (In reply to Justin D'Arcangelo [:justindarc] from comment #13)
> > (In reply to Jonas Sicking (:sicking) from comment #10)
> > > Also note that the Music App, or any other Gaia app, is no exception here.
> > > It too follows the same policies as webpages, and it too needs to explicitly
> > > opt in to having its audio keep playing in the background.
> > 
> > I also don't think that this is a valid argument. We happen to have what
> > seems to be a "block first" approach to background audio, so of course the
> > built-in Gaia "Music" app is going to implement the necessary
> > permissions/API calls to be capable of playing music in the background.
> > 
> > However, from a developer evangelism perspective, it is ridiculous for us to
> > expect hosted web pages/apps to implement our -moz-specific API calls in
> > order for audio to play in the background. We should be putting our users
> > first, and as part of that, we should be making is as painless as possible
> > for the web devs out there who are creating web content that our users want
> > to consume.
> 
> I agree that it's really bad that the API is moz-specific. However I
> actually think that blocking is the much more sensible default. Out of the
> websites that I visit that play audio, many more of them play audio that I
> would like to be muted when I switch the website to the background than are
> music players that I want to play background audio.
> 
> I.e. I visit many more ads/games/sound-effect/video-sharing content than
> music player content.

I agree that ads with sound could potentially be problematic with an "allow-first" behavior, but that's the only case I can think of that I *wouldn't* want the audio to continue playing in the background.

Perhaps this is a very subjective behavior that would differ from user to user. Would it make sense to have a system dialog appear in certain cases to prompt the user to determine what the appropriate course of action should be? (e.g.: Continue playing audio in background for `soundcloud.com`?: Yes/No/Always/Never)

If we had some sort of dialog prompt, we'd also need a "Behaviors"/"Permissions" pane under "Sounds" in the Settings app that listed all of the apps/domains that have played audio so the user could override a previously-specified selection (e.g.: if the user selected "Never" for `soundcloud.com`, they'd be able to reset it to "Prompt" or change it to "Always"). This would likely increase complexity, but I feel it would provide a much better user experience and it would also not require developers to opt-in to any -moz-specific APIs.

Currently, if a user wants to play audio in the background from a hosted app/site that has not opted-in to our audio competing policies, they have absolutely NO WAY of doing this. So, if we aren't going to abandon our "block first" behavior, we at least need to provide the freedom for our users to override our defaults.
UX has discussed about the audio competing rules and Jenny should have some ideas about this, also flagging her to give some inputs.
Flags: needinfo?(jelee)
I also agree with the UX mentioned in comment 3. It will make the most sense to the user that pressing the play button on a song from Soundcloud will override their current song playing on the music app. 

As well, I think that if the audio is playing in the background we should include the ability to play/pause the audio on the lockscreen and utility tray as is currently done in music with the playback controls. 

I think we can include more complex settings for the audio controls if needed but I think we should first try to provide the best default experience as Justin mentioned. Ideally we would restrict sound from things like background music in games and automatically playing advertisements, but allow audio to play from music applications when the user initiates playing the content.
Flags: needinfo?(jsavory)
(In reply to jsavory from comment #17)
> I think we can include more complex settings for the audio controls if
> needed but I think we should first try to provide the best default
> experience as Justin mentioned. Ideally we would restrict sound from things
> like background music in games and automatically playing advertisements, but
> allow audio to play from music applications when the user initiates playing
> the content.

Please understand that this is not possible. Or rather, we already are doing this, but web content needs to first indicate to us if they are "music player" or "background music in games". The problem is that no content does this.
I'd also like to have a UX draft which details how this is going to work once we switch to a full sheets model. I don't want to get stuck not being able to go to full sheets because of a decision made in this bug.
(In reply to Jonas Sicking (:sicking) from comment #18)
> (In reply to jsavory from comment #17)
> > I think we can include more complex settings for the audio controls if
> > needed but I think we should first try to provide the best default
> > experience as Justin mentioned. Ideally we would restrict sound from things
> > like background music in games and automatically playing advertisements, but
> > allow audio to play from music applications when the user initiates playing
> > the content.
> 
> Please understand that this is not possible. Or rather, we already are doing
> this, but web content needs to first indicate to us if they are "music
> player" or "background music in games". The problem is that no content does
> this.

Yes, this wouldn't be possible without the content specifying what audio channel it belongs on, and requiring content to do that brings us back to the problem we currently have.

I think maybe we're blocking ourselves from moving on from our current, flawed audio competing model by setting certain expectations impossibly high. For instance, having our audio policy be aware of whether the content is "music" or a "game" is impossible without requiring explicit information from the app itself.

In any case, we would never want to "mix" music from a music player app with in-game music. I would expect that if the user was playing a game that featured in-game music/sound effects that it would result in any background music playing apps to be paused. Ideally, we'd have an easy way for the user to "mute" an entire app/tab (maybe via Task Manager?). If it were possible to completely "mute" any app/tab, then the user would have full-control of their experience and we would be able to provide a more reasonable default audio competing policy.
> In any case, we would never want to "mix" music from a music player app with
> in-game music. I would expect that if the user was playing a game that
> featured in-game music/sound effects that it would result in any background
> music playing apps to be paused.

I agree with this. Though it's a bit tricky to tell "in game music" from "in game sound effects". But we certainly could pause background music if the front-most app is playing audio for more than X seconds continuously.
Let's go back to the key point: 'user-initiated media should continue the play in the background'.

USER INITIATED MEDIA (TRUSTED)

Audio and Video content that began playing as a result of a 'trusted' user interaction (eg. 'click') should be considered a `mozAudioChannel` type of 'content'. This type of media should continue playing in the background, and pause any currently playing 'content' media (eg. Music app).

AUTONOMOUS MEDIA (UNTRUSTED)

Anything that began playing automatically (eg. ads, background-music, sound-effects) should be considered a `mozAudioChannel` type of 'normal'. This type of media should cease playing when the app is put in the background, and should not interrupt any currently playing 'content' audio-channels.
Jonas, is the above suggestion at all plausible?
Flags: needinfo?(jonas)
dkuo (resident music expert): Were you able to identify *why* soundcloud's hosted app doesn't play music in the background despite jumping through the required -moz- hoops?
Flags: needinfo?(dkuo)
(In reply to Wilson Page [:wilsonpage] from comment #24)
> dkuo (resident music expert): Were you able to identify *why* soundcloud's
> hosted app doesn't play music in the background despite jumping through the
> required -moz- hoops?

I tried to write a simple hosted app to perform the audio behaviour like the soundcloud app(with audio-channel-content permission and set mozAudioChannelType), but the test app was able play music in the background, our audio channel api works as expected, but I was unable to identify why the soundcloud app couldn't, with the source code you captured in the email thread.

The source code explained they knew about the moz audio channel api though it didn't work as expected, but I guess the soundcloud service should have tested(play in background) before they submit to the market? and used to work before?
Flags: needinfo?(dkuo)
If would be good to have someone from Gecko side of things to debug why the the channel is not behaving as expected. Do you know of anyone?
Flags: needinfo?(dkuo)
(In reply to Dominic Kuo [:dkuo] from comment #25)
> The source code explained they knew about the moz audio channel api though
> it didn't work as expected, but I guess the soundcloud service should have
> tested(play in background) before they submit to the market? and used to
> work before?

First off, the SoundCloud app appears to be a hosted "app" that is nothing more than a bookmark to soundcloud.com. If you open soundcloud.com in the Rocketbar vs. installing the app from the Marketplace, you get the *exact* same content, including a big button in the top-right corner that says "Download App" that appears to do nothing. Its clear that this is simply SoundCloud's mobile web player that is intended to be used across platforms. Therefore, I don't think its even relevant to assume that SoundCloud should have tested their web content on FxOS to see if it could play audio in the background. Did anyone from SoundCloud actually have to submit this "app" to our Marketplace for it to get there? Or did someone on our end create the entry?

Secondly, if this used to work before, doesn't that indicate that there is a regression somewhere on our end?
Here's what I propose:

We work on a "media keys" API. Google put up a proposal at [1] and we've been working on something similar. I think we can get agreement pretty quickly.

Make any page which is declaring that it can handle media keys continue playing in the background.

Use the media keys API to implement policies similar to the ones we have right now where if you start playing a video in the video app, we automatically pause music in the music app.

Hook up the play/pause UI on the lock screen to the media keys API.

This means that we'd have a standardized API for enabling background audio. It should be quite easy to evangelize a few top music sites to support such an API.

It also means that no app will be able to play background audio without at the same time giving users an easy way to control that background audio. I.e. no risk of having to hunt down which app is causing audio to play.

[1] http://smus.com/remote-controls-web-media/
Flags: needinfo?(jonas)
A lot of smaller third party sites that I use (especially ones not in English) behave quite poorly on FxOS because of our audio implementation. I highly doubt that we would ever evangelize these sites, and these make dogfooding the OS unbearable for me. I think we need a solution here which doesn't involve evangelism for now. Once FxOS actually has market share, and we can influence the market, then that's the time to consider evangelism opportunities. Comment 1, option C, D, or E still work best for me.
I really don't want to do C. I'm definitely fine with doing D. In order to usefully debate E I think we need to have a more concrete proposal for heuristic.
I think D is better than what we have now. Could there be some way to remember the setting per-origin, so we can only ask once? I still think it's not nearly as good as other mobile OSes though, but it's a good first step.
(In reply to Wilson Page [:wilsonpage] from comment #26)
> If would be good to have someone from Gecko side of things to debug why the
> the channel is not behaving as expected. Do you know of anyone?

Randy Lin(the previous comment author :P) might be someone you are looking for, though he is probably busy on refactoring the audio channel things in gecko for 2.2.
Flags: needinfo?(dkuo)
Let's keep this bug focused on the ability for apps to play background audio without using mozaudiochannel=content.

If we have mozaudiochannel=content in some cases doesn't work as expected, please file a separate bug for that and let's move that discussion there.
(In reply to Justin D'Arcangelo [:justindarc] from comment #27)
> (In reply to Dominic Kuo [:dkuo] from comment #25)
> > The source code explained they knew about the moz audio channel api though
> > it didn't work as expected, but I guess the soundcloud service should have
> > tested(play in background) before they submit to the market? and used to
> > work before?
> 
> First off, the SoundCloud app appears to be a hosted "app" that is nothing
> more than a bookmark to soundcloud.com. If you open soundcloud.com in the
> Rocketbar vs. installing the app from the Marketplace, you get the *exact*
> same content, including a big button in the top-right corner that says
> "Download App" that appears to do nothing. Its clear that this is simply
> SoundCloud's mobile web player that is intended to be used across platforms.
> Therefore, I don't think its even relevant to assume that SoundCloud should
> have tested their web content on FxOS to see if it could play audio in the
> background. Did anyone from SoundCloud actually have to submit this "app" to
> our Marketplace for it to get there?

They are a business partner and submitted the app. I am working with them as technical support and they added the attribute on my recommendation.

Jonas, I am fully for the standard as it also gets my partners on the lockscreen (another feature I am pushing for since a long time). It also gets us away from non-standard content channel attributes.

If needed I can get more music partners involved to gather input and more use cases. But reading the proposal it looks like everything I was looking for is covered. As you mentioned that we are working on something similar, where can I find the WIP?
Flags: needinfo?(jonas)
Whiteboard: [dependency: marketplace-partners]
Right now the general guideline for an app that is inactive is: 
"Each app can decide to continue playback or to pause when inactive/screen is off. By default, an app will NOT continue playback when it becomes inactive or when screen is off."

So basically the app/browser that's playing web sound can continue playback in background if specified. I'd suggest that if we're doing this, a set of media control should be provided so that user can quickly pause/stop web sound from ex. utility tray or task manager like what we have for Music. Thanks!
Flags: needinfo?(jelee)
blocking-b2g: --- → backlog
Priority: -- → P1
> But reading the proposal it looks like everything I was looking
> for is covered. As you mentioned that we are working on something similar,
> where can I find the WIP?

There's no WIP patches. Just an API discussion here:
https://groups.google.com/d/msg/mozilla.dev.webapi/hZB_rWeGBJM/ER6_wzMKBwoJ
Flags: needinfo?(jonas)
blocking-b2g: backlog → ---
tracking-b2g: --- → backlog
See Also: → 1190434
Please consider something effective and tactical here.  For many, many sites (including Radio
Paradise--the app has bitrotted out of operability) the way to play them is via the browser.
But you can't use your phone for anything else while playing.  And you have to leave the screen
on.  Demonstrably, letting the tabs keep playing is OK--since that is the behavior on every
other web computing platform I can lay hands on.  So please just set the default to keep playing.
Or give us a button to set the default to this behavior.  This bug has already weathered all
of 2015, please just do fix the behavior, even if you have deeper more subtle plans for the
future.  Thanks.
Hi, Andrew,
Actually we have already enabled this behavior in b2g after landing the bug1190434.
But it's just like a workaround, we would implement better solution with the discussion of the MediaSession API.
Firefox OS is not being worked on
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.