1771179 - AudioContext power optimizations

Status: NEW

(Core :: Web Audio, defect, P2)

Description

[Paul Adenot (:padenot)]

This is mostly a power efficiency optimization. The idea is to pause (not stop) the cubeb stream running a graph, when no audio has been output for some time.

Pausing (vs. destroying) a cubeb stream allows restarting it quite quickly (especially on Windows, about 200x faster) compared to initializing a new one. Because this optimization is something that should be transparent to developers, it's critical.

I believe it should look something like the following:

• Add a way of knowing how long an AudioCalbackDriver has been outputing silence (we have this information on a particular AudioContext, but not globally). I've measured Chromium specifically stops
• Add a way to pause an AudioCallbackDriver, storing it somewhere, paused, and switch to a SystemClockDriver in the meantime. The graph should still be rendered, because lots of state might have to be updated. We can revisit this at a later stage
• Sprinkle calls to switch back to the AudioCallbackDriver a bit everywhere, or maybe in a very specific location (when receiving any control message maybe is a good catch-all and is simple?)

Some notes, after having tested a few things:

• Chrome does a variant of this, but this optimization never kicks in when an AudioWorkletNode is present in the graph. This might stem from the fact that AudioWorkletProcessors can output audio without any API call (say, just a store in a SharedArrayBuffer), and they're beind conservative
• Chrome always pauses the underlying system-level audio stream after exactly 35 seconds of silence

Comment 1

[Karl Tomlinson (:karlt)]

The proposal here differs from what previously existed (but was removed) in that

1. Cubeb streams are merely paused instead of destroyed.
2. The streams can be paused even when the AudioDestinationNode is not the only node in the AudioContext.

In general AudioNodeTrack::mIsActive is a good indicator (on graph thread) that the nodes will not produce output before another control message. This is unset even for AudioWorkletNode when [[active source]] is false and it has no active inputs. This could be used to determine whether a cubeb stream can be paused even when an AudioWorkletNode is present.

AudioDestinationNode is the exception, remaining active just to track time and provide main thread updates of time.

Suspension of tracks is the more general MediaTrackGraph concept, but EXTERNAL_OUTPUT AudioNodeTracks are not suspended when inactive.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:padenot, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Paul Adenot (:padenot)]

We also want to address the problem where MediaStream cause graph processing to always be active, e.g. in bug 1836994, but other sites that use MediaStream-related AudioNodes, or just regular MediaStream created normally.

Comment 5

[Florian Quèze [:florian]]

The Performance Impact Calculator has determined this bug's performance impact to be high. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.

Platforms: [x] Windows [x] macOS [x] Linux [x] Android
[x] Causes severe resource usage
[x] Able to reproduce locally
[x] Bug affects multiple sites
[x] Multiple reporters

Severe resource usage: some measurements padenot ran when filing this bug showed that needlessly keeping an AudioContext active can waste multiple watts.
Bug affects multiple sites. Websites affected: common. We have seen multiple reports on different sites. In bug 1836994 we saw that Zimbra creates an audio context at page load for feature detection purpose and never stops it.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S3. However, the Performance Impact field flags this bug as having a high impact on the performance.
:padenot, could you consider increasing the severity of this performance-impacting bug? Alternatively, if you think the performance impact is lower than previously assessed, could you request a re-triage from the performance team by setting the Performance Impact flag to ??

For more information, please visit BugBot documentation.

Comment 7

[Frank Doty [:fdoty]]

:karlt can you review this bug and determine if the severity matches the performance impact? A High impact would translate to a S2, which I reverted back 10 months ago since I am not the owner. If you feel this is an S3, can you comment on the mismatch between our calculations and yours and we can reassess the impact?

Comment 8

[Karl Tomlinson (:karlt)]

The severity of this bug has been changed several times, because it has been considered a candidate for S2.

I'll speculate on the reasoning for the choice of S3, and point out that more recent evidence would obsolete the suspected reasoning.

S2 (Serious) Major functionality/product severely impaired or a high impact issue and a satisfactory workaround does not exist

S3 (Normal) Blocks non-critical functionality or a work around exists

Functionality and product are not severely impaired (even though CPU usage is higher than sane).

I suspect the choice of S3 might have been on the assumption that this problem was largely due to sites using AudioContext recklessly without suspend() (bug 1863193), in which case the workaround was for sites to use suspend(). Such sites also affect Chrome to some extent, but not all such sites have the same effect in Chrome.

The subclasses AudioContext and OfflineAudioContext should be considered expensive objects. Creating these objects may involve creating a high-priority thread, or using a low-latency system audio stream, both having an impact on energy consumption..

suspend and close allow authors to release system resources, including threads, processes and audio streams..

Bug 1836994 subsequently identified a scenario where suspend() would not be sufficient to address the resource usage. Chrome does not use an audio device for that testcase. (Chrome uses so much CPU that I couldn't tell whether the testcase contributes to its CPU usage.) Bug 1836994 could plausibly be fixed without fixing bug 1863193 and without using the approach described here. If that happens, then we can re-assess the severity of this bug. Bug 1836994 is currently duped to this bug, however, so I'll mark this S2 based on "a satisfactory workaround does not exist".