753600 - Add support for per stream volume in cubeb backends

Status: RESOLVED FIXED

(Core :: Audio/Video, defect)

Description

[Michael Wu [:mwu]]

We should allow the use of the OS's volume control where available.

Comment 1

[Matthew Gregan [:kinetik]]

Of the currently supported platforms, the only one that could provide this functionality is the OS X backends, so I decided it wasn't worth the added complexity.

Comment 2

[Michael Wu [:mwu]]

Ah. Pulseaudio supports this too so I figured we might as well use it if we can.

Comment 3

[Matthew Gregan [:kinetik]]

I couldn't find a way to adjust the volume of an individual pa_stream.  As far as I can tell, PA offers a really nice volume control API that does something completely different to what I want; basically some magic adjustment of both the per-application and master volume.  The code I used to adjust what I thought would affect the per-stream volume (but altered both the application and master volumes) was here: https://github.com/kinetiknz/cubeb/commit/bcbeeabd5c77c8b6140dcce6743b925aae263e3c#diff-4.  Maybe someone who knows PA better can help out with this.

For the other platforms, the waveOut volume control changes the system-wide "PCM" volume.  The DirectSound backend we're not using (and is currently incomplete) does offer per-buffer volume control.  ALSA only seems to offer global volume control, but the API is fairly large and complex, so I could have missed something.

Comment 4

[Arun Raghavan]

(In reply to Matthew Gregan [:kinetik] from comment #3)
> I couldn't find a way to adjust the volume of an individual pa_stream.  As
> far as I can tell, PA offers a really nice volume control API that does
> something completely different to what I want; basically some magic
> adjustment of both the per-application and master volume.  The code I used
> to adjust what I thought would affect the per-stream volume (but altered
> both the application and master volumes) was here:
> https://github.com/kinetiknz/cubeb/commit/
> bcbeeabd5c77c8b6140dcce6743b925aae263e3c#diff-4.  Maybe someone who knows PA
> better can help out with this.

The code there is absolutely correct. :) I really think that per-stream volumes is what applications should be doing. This also makes sure if that if I have multiple streams running where one is of low volume and the other is higher, I can increase the volume of one and decrease the other independently. Also, it ensures consistency with all other applications on the platform.

The long explanation of why your code altered both mixers in the link below, but the short explanation is this -- by default, PA runs in flat volume mode, where the device volume is set to the highest volume of all streams attached to it and all other streams are attenuated in software. As I understand it, this is based somewhat on how Windows Vista does things, and presents a more user-friendly experience. (p.s.: Ubuntu disables this behaviour)

http://0pointer.de/blog/projects/oh-nine-fifteen.html

Taking this a step further, with per-stream volumes in place, we can do even things by setting the application name (PA stream property) to the website that owns the audio/video element. I don't know if your infrastructure allows something like that, but it would allow PA to be intelligent on this basis, like automatically storing/restoring per-website volumes, remembering which device a user moved the stream for a particular website's streams to, and other such (IMO) nifty things.

> For the other platforms, the waveOut volume control changes the system-wide
> "PCM" volume.  The DirectSound backend we're not using (and is currently
> incomplete) does offer per-buffer volume control.  ALSA only seems to offer
> global volume control, but the API is fairly large and complex, so I could
> have missed something.

AFAIK, ALSA does not support per-stream volumes.

Comment 5

[Matthew Gregan [:kinetik]]

Thanks, that link reminded me of the problem I ran into with the code I linked to.  The default volume for a <video> element is 1.0.  Setting that volume via the code I linked above causes the system volume to be set to maximum, which is the expected behaviour according to oh-nine-fifteen.html, but not at all what you want to allow web content to do.  I need a way to change the per-stream volume without allowing it to increase the final output volume.

I tried a variation of the linked code where I calculated a per-stream volume relative to the system volume, but obviously that results in a per-stream volume that does not remain relative to the system volume as the system volume changes.  Perhaps I could've worked around that by setting up an observer for the system volume, but it didn't seem worth it at the time.

Is there an easier way to get the behaviour I want from PA?

Doing per-tab volume control is something I'd like to do in the future.  The next step that I'm working on for Firefox audio is introducing a lighter weight stream abstraction that performs software mixing and uses a single cubeb_stream for output; it'd be fairly easy to use one cubeb_stream per tab (with the appropriate metadata passed through to PA), I think.

Comment 6

[Arun Raghavan]

(In reply to Matthew Gregan [:kinetik] from comment #5)
> Thanks, that link reminded me of the problem I ran into with the code I
> linked to.  The default volume for a <video> element is 1.0.  Setting that
> volume via the code I linked above causes the system volume to be set to
> maximum, which is the expected behaviour according to oh-nine-fifteen.html,
> but not at all what you want to allow web content to do.  I need a way to
> change the per-stream volume without allowing it to increase the final
> output volume.
> 
> I tried a variation of the linked code where I calculated a per-stream
> volume relative to the system volume, but obviously that results in a
> per-stream volume that does not remain relative to the system volume as the
> system volume changes.  Perhaps I could've worked around that by setting up
> an observer for the system volume, but it didn't seem worth it at the time.
> 
> Is there an easier way to get the behaviour I want from PA?

I think worrying about the volume at this granularity is a losing battle. Instead of fighting system behaviour, it makes more sense to create a stream with default parameters, query the stream volume after setup is complete, and use report that as the stream volime. This is what the WebKit are doing as well, and seems to be acceptable as per the spec: http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-element.html#dom-media-volume

> Doing per-tab volume control is something I'd like to do in the future.  The
> next step that I'm working on for Firefox audio is introducing a lighter
> weight stream abstraction that performs software mixing and uses a single
> cubeb_stream for output; it'd be fairly easy to use one cubeb_stream per tab
> (with the appropriate metadata passed through to PA), I think.

What problem are you trying to solve with this approach? This opens something of a can of worms. For example, if you have a single tab that has the ability to play music (high latencies and associated power savings acceptable) as well as run voip streams (low latency required, more power intensive). Your mixer will either need to just pick some sane default latency and force all streams to use that, or start worrying about dynamic adjustments, and before you know it, you're solving a lot of problems that your system stack has hopefully already solved for you.

Comment 7

[Matthew Gregan [:kinetik]]

(In reply to Arun Raghavan from comment #6)
> I think worrying about the volume at this granularity is a losing battle.
> Instead of fighting system behaviour, it makes more sense to create a stream
> with default parameters, query the stream volume after setup is complete,
> and use report that as the stream volime. This is what the WebKit are doing
> as well, and seems to be acceptable as per the spec:
> http://www.whatwg.org/specs/web-apps/current-work/multipage/the-video-
> element.html#dom-media-volume

I don't agree with that approach, and as I said before, I don't think web content should be able to directly control the system volume.

(In reply to Arun Raghavan from comment #6)
> What problem are you trying to solve with this approach? 

The system stacks aren't designed to have hundreds of live streams per process running.  For example, the ALSA Pulse plugin uses a PA context per ALSA PCM, resulting in a 64MB shared memory mapping and at least one additional thread being created per PCM created.  On a 32-bit system, that leaves us with no address space left after opening ~42 streams or so.  See bug 617852.  Also, from memory, PA has something like a 32 stream limit per context.  And this is just on one platform, there are others we have to worry about.  But Linux is certainly the worst.

There are also a bunch of use cases where it simply doesn't make sense to pay the high cost of creating a system-level stream to play a sound, like playing back very short sound effects in games.

Comment 8

[Arun Raghavan]

(In reply to Matthew Gregan [:kinetik] from comment #7)
[...]
> I don't agree with that approach, and as I said before, I don't think web
> content should be able to directly control the system volume.

By default, PulseAudio does not expect applications to worry about the system volume. By definition, the correlation between the two is left to PA as policy. Apps should just be working with stream volumes and not care about anything else.

In this brave new world, each website is effectively an app unto itself. :) I don't see that users would find them behaving like all their other applications objectionable.

> (In reply to Arun Raghavan from comment #6)
[...]
> process running.  For example, the ALSA Pulse plugin uses a PA context per
> ALSA PCM, resulting in a 64MB shared memory mapping and at least one
> additional thread being created per PCM created.  On a 32-bit system, that
> leaves us with no address space left after opening ~42 streams or so.  See

This will become a non-issue on PulseAudio systems once there is a native backend.

> bug 617852.  Also, from memory, PA has something like a 32 stream limit per
> context.  And this is just on one platform, there are others we have to
> worry about.  But Linux is certainly the worst.

The stream limit is up for modification if there's a good case for it. That said, how do you intend to deal with this limit on other platforms?

> There are also a bunch of use cases where it simply doesn't make sense to
> pay the high cost of creating a system-level stream to play a sound, like
> playing back very short sound effects in games.

There are multiple solutions to this. 1. Ignore it -- stream creation isn't that expensive. 2. Write apps correctly so they don't create a stream for each sound (don't know if the web APIs allow this). 3. Maintain a stream pool in the backend and reuse them for new streams. The pool can be pruned during periods of inactivity.

I understand that Firefox needs to work across platforms, but reimplementing a mixer just means that you give up the ability to use system features that can actually make Firefox a better-behaved app (latency management, routing policy, ...).

Comment 9

[Colin Guthrie]

Yeah I very much agree with Arun here. Ideally each website (domain?) should have it's own stream (they can all share a context, but that may be tricky depending on how the sandboxing works).

Having seen some of the really funky things recently regarding the potential of web-based VoIP applications, I'd really like to see the system audio policy come into effect here for such applications. i.e. I'd like to know that the stream playing from last.fm or soundcloud is a "music" one. When a voip call comes in, I'd like to know it's a "voip" stream. Doing this allows system policy in PulseAudio to take appropriate actions: e.g. automatically muting (or if implemented fully, actually pausing) the music streams while the call is in progress (with automatic un-mute/un-pause after the call is finished). Other tricks we could do at the policy level is to ensure that when the voip app plays it's "ringing sound" that we temporarily lower the volume of the playing music to make the ring noise more obvious. Or automatically using the Bluetooth or USB headset for voip calls, or the Airtunes Airplay device for music playback etc. Basically all the things we're trying to do with desktop apps, but applying equally for web apps as, IMO, it should be.

And on the note of volume controls, I firmly believe applications (whether a web site or a desktop app) should *only* control their own stream volumes. What we decide to do in PulseAudio with that stream volume is our business. If we do something you don't like (e.g. perhaps our policy of setting the h/w volume controls to the loudest stream volume is something you don't like, even if it saves doing attenuation in software and thus could save power and extend battery life), then users should have an option in PA to change that behaviour generally and it should not be "worked around" at a higher level as that leads to inconsistent behaviour between different applications which is something PA has been very good at stamping out and I would consider it a specific backwards step (this is the same reason I was totally against changes in VLC to compensate for "Base Volumes" - it's totally the wrong place to do such stuff).

We are always open upstream to suggestions to make your life easier, but please don't implement kludges - speak to us and we'll work out the best way for everyone going forward!

I really look forward to seeing first class audio support in Firefox, so please do KUTGW :)

Thanks for a very happy Firefox user and PA developer.

Comment 10

[Michael Wu [:mwu]]

BTW, http://msdn.microsoft.com/en-us/library/windows/desktop/dd370977%28v=vs.85%29.aspx seems to indicate that windows also supports per stream volumes, though it may be restricted to newer versions of windows.

Comment 11

[Paul Adenot (:padenot)]

Matthew, we did that a while back, did we not?

Comment 12

[Matthew Gregan [:kinetik]]

(In reply to Paul Adenot (:padenot) from comment #11)
> Matthew, we did that a while back, did we not?

Yeah (thanks to you!), I think this bug has basically been implemented.  Where we can't use the OS volume control we've got soft-volume within libcubeb,

Comment 13

[Paul Adenot (:padenot)]

I'm closing this, then.