1089193 - Expose ChromeOnly property on AudioNode for memory usage

Status: ASSIGNED

(Core :: Web Audio, defect, P4)

Description

[Jordan Santell [:jsantell] [@jsantell]]

Similar to exposing `cpuUsage` on AudioNodes (bug 1089192), exposing `memoryUsage` would be helpful as well (although less useful than cpuUsage, it sounds like). 

This is not currently in the spec at all, so should definitely be a ChromeOnly API for now.

If there's any way to aggregate this information as well for the entire AudioContext, great.

Talking with padenot, it sounds like there is work for this already in place, it's just a matter of exposing it -- feel free to CC around :)

Comment 1

[Paul Adenot (:padenot)]

Attachment: Expose ChromeOnly property on AudioNode for memory usage. r=

Jordan, can you try this? It adds a (chrome-only) method on AudioContext,
getMemoryReport(), that returns a promise that resolves to a dict that has node
id as keys and memory usage in bytes as values.

Going with a memoryUsage member was not too practical because this really needs
to be async, since we're doing cross-thread communication.

This is only a first cut, because we need something for the AudioBuffers and the
context itself as well.

Comment 2

[Jordan Santell [:jsantell] [@jsantell]]

Cool, roughly hooking into this, dumping the node type, id, and byte:

console.log: AudioDestinationNode 11 20944
console.log: OscillatorNode 12 2368
console.log: OscillatorNode 13 2368
console.log: GainNode 14 1904
console.log: BiquadFilterNode 15 3584
console.log: OscillatorNode 16 1344
console.log: GainNode 17 672
console.log: AnalyserNode 18 33808
console.log: ScriptProcessorNode 19 2160
console.log: OscillatorNode 20 1344
console.log: GainNode 21 672	

How heavy is this to call? Is this something that can be called once a second maybe without issue? Should this be an option to permanently set (call once a second, for example), or only when interacting with a certain node? I guess thinking about the performance of this.

Also, the only node that seemed to change was the AudioDestinationNode -- all the other nodes had consistent memory usage -- is that normal/to be expected?

Comment 3

[Jordan Santell [:jsantell] [@jsantell]]

Second, due to lacking IDs, how are the AudioBuffers going to be identified? Are they going to be calculated within the "context" itself, or from the AudioBufferSourceNode that uses them? Is the context memory usage overhead significant? Still trying to figure out how best to show this information.

FWIW, if the AudioBuffer/context memory information is not trivial, the nodes alone would still be helpful, I think. Thanks, Paul!

Comment 4

[Paul Adenot (:padenot)]

(In reply to Jordan Santell [:jsantell] [@jsantell] from comment #2)
> Cool, roughly hooking into this, dumping the node type, id, and byte:
> 
> console.log: AudioDestinationNode 11 20944
> console.log: OscillatorNode 12 2368
> console.log: OscillatorNode 13 2368
> console.log: GainNode 14 1904
> console.log: BiquadFilterNode 15 3584
> console.log: OscillatorNode 16 1344
> console.log: GainNode 17 672
> console.log: AnalyserNode 18 33808
> console.log: ScriptProcessorNode 19 2160
> console.log: OscillatorNode 20 1344
> console.log: GainNode 21 672	
> 
> How heavy is this to call? Is this something that can be called once a
> second maybe without issue? Should this be an option to permanently set
> (call once a second, for example), or only when interacting with a certain
> node? I guess thinking about the performance of this.
Yes, once a second should be fine. This runs on the audio thread, and is not super expensive to call, I'd say (I haven't measured). Of course if the audio thread is already short on CPU, it's not going to be great, heh.

> Also, the only node that seemed to change was the AudioDestinationNode --
> all the other nodes had consistent memory usage -- is that normal/to be
> expected?

I guess if you changed the length of a delay, you would see a change in some cases as well, probably some other nodes as well (I'm thinking OscillatorNode is you set a PeriodicWave, maybe?).

Comment 5

[Paul Adenot (:padenot)]

(In reply to Jordan Santell [:jsantell] [@jsantell] from comment #3)
> Second, due to lacking IDs, how are the AudioBuffers going to be identified?
> Are they going to be calculated within the "context" itself, or from the
> AudioBufferSourceNode that uses them? Is the context memory usage overhead
> significant? Still trying to figure out how best to show this information.

Maybe we could toy with the idea of showing AudioBuffers "linked" to AudioBufferSourceNodes? The issue is that AudioBuffer can be used in more than one AudioBufferSourceNode. We would give them an id as well or something.

The context is not big, but the underlying thing (the MediaStreamGraph, that runs all the contexts) is a (somewhat) big one-time cost (there is roughly one MSG per process).

> 
> FWIW, if the AudioBuffer/context memory information is not trivial, the
> nodes alone would still be helpful, I think. Thanks, Paul!

I'm sure we can do something once we figure out what to do, heh.

Comment 6

[Jordan Santell [:jsantell] [@jsantell]]

The oscillator nodes in that example were just square/sine waves, no fancy PeriodicWaves.

The interval that the memory gets pinged could be modified a bit so it wasn't consistent like, 1 second +/- 0.1s at random or something, if you think some sort of cycle could occur.

For non-node memory usage, maybe a little HUD could show aggregate and non-node values like:

* context overhead: 500kb
* audiobuffers: 3000kb
* audionodes: 4000kb
* total: 7500kb

at this point, tagging audio buffers and saying how much memory they have individually isn't that much use, because we can't infer where this buffer came from or what's in it, etc... but i think displaying the aggregate could be useful. All this can be done on the devtools side, if we have context/buffer information returned from the getMemoryUsage map

Comment 7

[Jordan Santell [:jsantell] [@jsantell]]

Trying out this patch more, some things I'm seeing:

* The context's getMemoryReport is displaying memory information from previously document/contexts. This is probably because they're not yet collected, just wanted to make sure if this is intended (not an issue, since we can tell they're no longer relevent)

* A bigger issue is `getMemoryReport` doesn't always return memory information for nodes. If I create nodes and call `getMemoryReport`, those new nodes aren't always there. If I wait a second or two, they're available. What is this dependent on?

Comment 8

[Paul Adenot (:padenot)]

(In reply to Jordan Santell [:jsantell] [@jsantell] from comment #7)
> Trying out this patch more, some things I'm seeing:
> 
> * The context's getMemoryReport is displaying memory information from
> previously document/contexts. This is probably because they're not yet
> collected, just wanted to make sure if this is intended (not an issue, since
> we can tell they're no longer relevent)
> 
> * A bigger issue is `getMemoryReport` doesn't always return memory
> information for nodes. If I create nodes and call `getMemoryReport`, those
> new nodes aren't always there. If I wait a second or two, they're available.
> What is this dependent on?

I'd say the underlying MediaStream has to be created, this takes some time (because it has to go across a thread, and then wait for the audio callback to kick in, etc.), and there is no way to know then the node has been created on the audio thread, from the js point of view.

Although it should not be "a second a two", it should be much less.

Comment 9

[Jordan Santell [:jsantell] [@jsantell]]

Ah okay, that makes sense, wanted to make sure that was intended. This won't be an issue for usage, but just in tests, and can get around that via polling. Thanks! What's left to add on this one?

Comment 10

[Paul Adenot (:padenot)]

Not sure, this is kind of hacky, heh.

Comment 11

[Bulk Bug Changes for mreavy's org]

Mass change P3->P4 to align with new Mozilla triage process.