Closed Bug 572542 Opened 10 years ago Closed 2 years ago

need benchmarks for measuring cycle collection / garbage collection pauses

Categories

(Testing :: Talos, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: dbaron, Unassigned)

References

(Depends on 1 open bug)

Details

I was just discussing with Andreas the need for benchmarks to measure cycle collection pause times.  He thinks he can find somebody to work on this.


I think there are two basic things we want to measure:
 * how long it takes to do a cycle collection with a large heap but little garbage
 * how long it takes to collect a lot of garbage
but I think we also want to be careful to measure the realistic situations that are affected in these situations.

So I'd propose that we measure the following two situations:

Situation 1 (to test collection on a large heap with little garbage):
 1. open the 200 (?) pages in the Tp pageset in tabs spread across something like 20-50 browser windows.
 2. force a cycle collection that we don't measure (since we don't want the random variation based on how much garbage there was since the last cycle collection, since the time since the last collection may vary randomly)
 3. then, immediately afterwards, force a series of cycle collections (10?) and measure how long the bunch of them take

Situation 2 (to test collection of a lot of garbage):
 1. repeat step (1) from above (or, optionally, all of 1-3, so that we can run all tests at once)
 2. start the measurement timer
 3. close all but one of the browser windows
 4. force a cycle collection
 5. stop the measurement timer
This situation measures more than just cycle collection:  it includes all of window closing/destruction, but I think it's a more realistic measure, and it does measure the performance effects of our decisions about whether or not to do intermediate cycle collections during the process of closing a lot of windows.

We can use nsIDOMWindowUtils.garbageCollect to force collection, but we probably want to switch it to doing only one collection.

I came up with these ideas pretty quickly; there are probably some easy improvements that could be made.
Jesse, could you explain briefly how you trigger a cycle collection from within chrome code?

Also, we noticed that :CC is called twice in that hook. We should probably change that to 1 invocation, and you call it twice from the outside if you really want to.
You can change nsIDOMWindowUtils.garbageCollect() to do what you want, right?
Yeah, I want to take out the 2nd CC call inside of it. If you want 2 cycle collections to run, call it twice.
To create a reliable test here we'd need:

- a means to disable garbage collection (maybe a pref or something) - if we can't fully control when garbage collection occurs our test may attempt to do a measurement direction after a run

- this may already exist, but some means of knowing when all pages/tabs have loaded; ie, an onLoad event that would be browser wide instead of just per-page.  This would very much simplify determining when to start/stop timers, etc.
Having the requirements listed in comment #5 worked on/explored would still be good here.

I am starting to explore hacking the pageloader extension to be able to do this type of test, but without the hooks to disable garbage collection I don't know if we'll get much good data out of test.
Depends on: 573175
bug 573175 adds a way to explicitly trigger a GC only, and a CC only using DOMUtils
Depends on: 580049
I've got a pretty good skeleton bundle in place now that:

- read a manifest file that specifies windows & tabs to load in each window
- opens all the windows + tabs and monitors for when they have all completed loading
- can then close all the windows/re-test

I can start playing with the gc/cc triggering, but I'm still waiting on being able to disable automatic memory collection so as to get useful results.
gal:
can you give me the code chunks to force a cycle collection so that I can
integrate it into my code (I also want to be sure that the test is designed to
do what you want it to do...)
#9: we are working on the API for this, bent is currently moving the CC to a separate thread, as part of that work the api will be implemented.

Also, as for the amount of memory, lets just go with the 2GB we have in the boxes and see if we run into problems.
I'm not sure if talos is the best platform for this.  CCing :wlach and :mcote in case they have any idea.  I think that the in-built profiler is better for this needs;  not sure what frameworks use this or what would be the appropriate one for this bug.
There's a newish interface for getting GC/CC pause times that can be used.  Terrence wrote some documentation for it. It has also been hooked into the profiler, I believe.
(In reply to Jeff Hammel [:jhammel] from comment #11)
> I'm not sure if talos is the best platform for this.  CCing :wlach and
> :mcote in case they have any idea.  I think that the in-built profiler is
> better for this needs;  not sure what frameworks use this or what would be
> the appropriate one for this bug.

We talked about measuring this type of thing out-of-band (either via the profiler or some other method) in peptest and a still-hypothetical eideticker-for-desktop-firefox at last week's snappy meeting. Whether we do that, or add something to talos I guess really depends on what we want to measure and how we want to report it. Talos, peptest, and eideticker each have their respective pros and cons.

I don't see any harm in keeping this bug open in case measuring this type of thing during talos runs turns out to be a useful thing.
I also think this is fair game, at least in theory, for talos. But it would require a yet-to-be-filed blocking bug of having talos read from the builtin profiler, which isn't a small task especially in light of how much architectural improvement talos really needs.  Let's keep this open for now, but keep this in mind.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.