1632750 - Implement memory-limiting IPCs

Status: RESOLVED FIXED

(Core :: Gecko Profiler, task, P2)

Description

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

After bug 1630872, the profiler now uses a chunk-based storage.
Chunks are created, released (after they're full of data), and destroyed by "chunk managers". The main profile storage uses ProfileBufferChunkManagerWithLocalLimit, which has a memory limit in its process (just like with the previous storage strategies).

Now we want to limit the total memory used by all processes in the Firefox application.
This can be done by instrumenting the chunk manager to inform a central controller about chunks (created, released, locally destroyed), and this controller can then decide to destroy the oldest chunk (from all processes!) when the memory limit is reached.

A beneficial side effect is that at the end, all processes should have a similar time range of data.

One potentially surprising side effect is that the profiler will now really use the given limit for the whole Firefox app, instead of for each process, meaning that for the same limit we will record less data.
We may need to change some defaults to try and keep the experience the same for most users?

Comment 1

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfileBufferControlledChunkManager - r?canaltinova!,gregtatum

Interface class for a chunk manager that can be controlled: It will provide updates about chunks, and release chunks on command.

Comment 2

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - Make ProfileBufferChunkManagerWithLocalLimit a ProfileBufferControlledChunkManager - r?canaltinova!,gregtatum

Depends on D72362

Comment 3

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - profiler_get_controlled_chunk_manager - r?canaltinova!,gregtatum

The Gecko Profiler can provide its current controlled chunk manager.
It is the responsibility of the caller to keep track of the state of the profiler, to avoid using the chunk manager after it's discarded.

Depends on D72363

Comment 4

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfilerParent::ProfilerWillStopIfStarted - r?canaltinova!,gregtatum

The Gecko Profiler can notify the ProfilerParent when it's about to stop (if it was started, but the notification will happen even when it's not started).

Depends on D72364

Comment 5

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - New profiler IPDL functions to manage chunks across processes - r?canaltinova!,gregtatum

Depends on D72365

Comment 6

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfileBufferGlobalController - r?canaltinova!,gregtatum

When there is at least one ProfilerParent (i.e., we are interacting with at least one child process) AND the parent profiler is running, the ProfilerParentTracker sets up the ProfileBufferGlobalController that will manage all chunks.
As a first step, it connects with the local chunk manager (to receive chunk updates), and sends update requests to all children.

(The actual controller logic is not implemented in this patch, nor is the ProfilerChild side, see following patches.)

Depends on D72366

Comment 7

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfileBufferGlobalController::HandleChunkManagerUpdate logic - r?canaltinova!,gregtatum

The logic part of the controller receives all updates, and makes decisions to destroy old chunks when the memory limit is reached.

Depends on D72367

Comment 8

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfilerChild handling update and destruction requests - r?canaltinova!,gregtatum

This implement the child side:
When the first request for update arrives, it connects to the local chunk manager, to receive its updates.
If multiple updates are received, they are folded into one.
If there are both an update and a pending request, the request is fulfilled with the update and local data is reset.
And ProfilerChild handles "destroy" instructions to destroy local chunks.

At this point, the whole machinery is in place, and all combined profile buffers used in all processes should use around the maximum amount allowed.
A bit more memory may still be used, e.g., due to IPC delays, and because of recycling which keeps some unused chunks alive for later reuse. But overall that should be a small amount compared to the usual user-requested limit.

Depends on D72368

Comment 9

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - Recycle chunks that are destroyed through DestroyChunksAtOrBefore or the local limit - r?canaltinova!,gregtatum

Whenever chunks are about to be destroyed, we try to keep one or two alive, to hopefully fulfill the next request, thereby avoiding a deallocation+allocation pair.

Depends on D72369

Comment 10

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Class diagram with new classes&functions

Class diagram showing the important existing classes, and new classes and function in bold.

Comment 11

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - ProfileBufferChunkManagerWithLocalLimit doesn't need to track chunk metadata sizes - r?canaltinova!,gregtatum

The chunk metadata size is tiny (less than 100 bytes) compared to the buffer size (1MB by default), so it's fine to ignore it while dealing with cross-Firefox limits.

Depends on D72362

Comment 12

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - Tweak MOZ_PROFILER_STARTUP_ENTRIES default values - r?canaltinova!

These defaults were per process and there are usually around 8 processes.
Now these sizes apply to all processes, so they can be 8 times as big (but less on Android where memory may be more limited.)

Not changing Base Profiler defaults, as its buffer is not cross-process controlled.

Depends on D72370

Comment 13

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

Attachment: Bug 1632750 - Tweak about:profiling default buffer sizes - r?gregtatum,canaltinova

Presets used 16Mi entries (128MiB) by default, but these numbers were a limit per process.
Now that the limit is for the whole Firefox app, we would record a lot less information, because there are typically around 8 or more processes.
So the default are now 8 times larger to roughly record a similar amount of profiling data.

Also the custom buffer maximum size has been increased from 1GiB to 2GiB.

Depends on D73215

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ecfdb624443a
ProfileBufferControlledChunkManager - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/4f71e3dfbef9
ProfileBufferChunkManagerWithLocalLimit doesn't need to track chunk metadata sizes - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/1c5a0c331c78
Make ProfileBufferChunkManagerWithLocalLimit a ProfileBufferControlledChunkManager - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/6c08ebdecab1
profiler_get_controlled_chunk_manager - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/f735a32f6683
ProfilerParent::ProfilerWillStopIfStarted - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/913272e74602
New profiler IPDL functions to manage chunks across processes - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/a44300bb4c83
ProfileBufferGlobalController - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/ae9b81e42327
ProfileBufferGlobalController::HandleChunkManagerUpdate logic - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/8debffa7a857
ProfilerChild handling update and destruction requests - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/986db0e8266e
Recycle chunks that are destroyed through DestroyChunksAtOrBefore or the local limit - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/b0a6620396aa
Tweak MOZ_PROFILER_STARTUP_ENTRIES default values - r=canaltinova
https://hg.mozilla.org/integration/autoland/rev/037361148517
Tweak about:profiling default buffer sizes - r=gregtatum

Comment 15

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/ecfdb624443a
https://hg.mozilla.org/mozilla-central/rev/4f71e3dfbef9
https://hg.mozilla.org/mozilla-central/rev/1c5a0c331c78
https://hg.mozilla.org/mozilla-central/rev/6c08ebdecab1
https://hg.mozilla.org/mozilla-central/rev/f735a32f6683
https://hg.mozilla.org/mozilla-central/rev/913272e74602
https://hg.mozilla.org/mozilla-central/rev/a44300bb4c83
https://hg.mozilla.org/mozilla-central/rev/ae9b81e42327
https://hg.mozilla.org/mozilla-central/rev/8debffa7a857
https://hg.mozilla.org/mozilla-central/rev/986db0e8266e
https://hg.mozilla.org/mozilla-central/rev/b0a6620396aa
https://hg.mozilla.org/mozilla-central/rev/037361148517

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

I am testing windows10 cppunit tests on hardware (instead of aws cloud) and I get some failures:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=LOoWN3ReTrajPBMZ5gkTyw.0&revision=36d5d16e8501531d164cf5faef467da1280b35a7&searchStr=cppunit

specifically I hit this assertion (added in this bug):

•   MOZ_ASSERT(
  

•       !chunk->GetNext() || (chunk->ChunkHeader().mDoneTimeStamp <
  

•                             chunk->GetNext()->ChunkHeader().mDoneTimeStamp),
  

•       "Released chunk groups must have increasing timestamps");
  

Is there a way to get more information about this?

Comment 17

[Gerald Squelart (he/him) (not at Mozilla since 2022-09-15)]

I've filed bug 1651863 to look into it...