need benchmarks for measuring cycle collection / garbage collection pauses

RESOLVED WONTFIX

Status

Testing
Talos
RESOLVED WONTFIX
7 years ago
18 days ago

People

(Reporter: dbaron, Unassigned)

Tracking

(Depends on: 1 bug)

Trunk
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
I was just discussing with Andreas the need for benchmarks to measure cycle collection pause times.  He thinks he can find somebody to work on this.


I think there are two basic things we want to measure:
 * how long it takes to do a cycle collection with a large heap but little garbage
 * how long it takes to collect a lot of garbage
but I think we also want to be careful to measure the realistic situations that are affected in these situations.

So I'd propose that we measure the following two situations:

Situation 1 (to test collection on a large heap with little garbage):
 1. open the 200 (?) pages in the Tp pageset in tabs spread across something like 20-50 browser windows.
 2. force a cycle collection that we don't measure (since we don't want the random variation based on how much garbage there was since the last cycle collection, since the time since the last collection may vary randomly)
 3. then, immediately afterwards, force a series of cycle collections (10?) and measure how long the bunch of them take

Situation 2 (to test collection of a lot of garbage):
 1. repeat step (1) from above (or, optionally, all of 1-3, so that we can run all tests at once)
 2. start the measurement timer
 3. close all but one of the browser windows
 4. force a cycle collection
 5. stop the measurement timer
This situation measures more than just cycle collection:  it includes all of window closing/destruction, but I think it's a more realistic measure, and it does measure the performance effects of our decisions about whether or not to do intermediate cycle collections during the process of closing a lot of windows.

We can use nsIDOMWindowUtils.garbageCollect to force collection, but we probably want to switch it to doing only one collection.

I came up with these ideas pretty quickly; there are probably some easy improvements that could be made.

Comment 1

7 years ago
Jesse, could you explain briefly how you trigger a cycle collection from within chrome code?

Also, we noticed that :CC is called twice in that hook. We should probably change that to 1 invocation, and you call it twice from the outside if you really want to.

Comment 2

7 years ago
You can change nsIDOMWindowUtils.garbageCollect() to do what you want, right?

Comment 3

7 years ago
Yeah, I want to take out the 2nd CC call inside of it. If you want 2 cycle collections to run, call it twice.
(Reporter)

Comment 4

7 years ago
See lines 140-142 in:
http://hg.mozilla.org/mozilla-central/annotate/98901ab12e04/dom/tests/mochitest/bugs/test_bug346659.html#l140
To create a reliable test here we'd need:

- a means to disable garbage collection (maybe a pref or something) - if we can't fully control when garbage collection occurs our test may attempt to do a measurement direction after a run

- this may already exist, but some means of knowing when all pages/tabs have loaded; ie, an onLoad event that would be browser wide instead of just per-page.  This would very much simplify determining when to start/stop timers, etc.
Having the requirements listed in comment #5 worked on/explored would still be good here.

I am starting to explore hacking the pageloader extension to be able to do this type of test, but without the hooks to disable garbage collection I don't know if we'll get much good data out of test.

Updated

7 years ago
Depends on: 573175

Comment 7

7 years ago
bug 573175 adds a way to explicitly trigger a GC only, and a CC only using DOMUtils

Updated

7 years ago
Depends on: 580049
I've got a pretty good skeleton bundle in place now that:

- read a manifest file that specifies windows & tabs to load in each window
- opens all the windows + tabs and monitors for when they have all completed loading
- can then close all the windows/re-test

I can start playing with the gc/cc triggering, but I'm still waiting on being able to disable automatic memory collection so as to get useful results.
gal:
can you give me the code chunks to force a cycle collection so that I can
integrate it into my code (I also want to be sure that the test is designed to
do what you want it to do...)

Comment 10

7 years ago
#9: we are working on the API for this, bent is currently moving the CC to a separate thread, as part of that work the api will be implemented.

Also, as for the amount of memory, lets just go with the 2GB we have in the boxes and see if we run into problems.

Comment 11

5 years ago
I'm not sure if talos is the best platform for this.  CCing :wlach and :mcote in case they have any idea.  I think that the in-built profiler is better for this needs;  not sure what frameworks use this or what would be the appropriate one for this bug.
There's a newish interface for getting GC/CC pause times that can be used.  Terrence wrote some documentation for it. It has also been hooked into the profiler, I believe.
(In reply to Jeff Hammel [:jhammel] from comment #11)
> I'm not sure if talos is the best platform for this.  CCing :wlach and
> :mcote in case they have any idea.  I think that the in-built profiler is
> better for this needs;  not sure what frameworks use this or what would be
> the appropriate one for this bug.

We talked about measuring this type of thing out-of-band (either via the profiler or some other method) in peptest and a still-hypothetical eideticker-for-desktop-firefox at last week's snappy meeting. Whether we do that, or add something to talos I guess really depends on what we want to measure and how we want to report it. Talos, peptest, and eideticker each have their respective pros and cons.

I don't see any harm in keeping this bug open in case measuring this type of thing during talos runs turns out to be a useful thing.

Comment 14

5 years ago
I also think this is fair game, at least in theory, for talos. But it would require a yet-to-be-filed blocking bug of having talos read from the builtin profiler, which isn't a small task especially in light of how much architectural improvement talos really needs.  Let's keep this open for now, but keep this in mind.
Blocks: 1040081

Updated

18 days ago
Status: NEW → RESOLVED
Last Resolved: 18 days ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.