654501 - weak dictionary leaks over time

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Library, defect)

Description

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: benchmark

If you repeatedly add entries to a weak dictionary but keep the total number of live objects bounded, you would expect that the garbage collector will continually reclaim the entries from the dictionary and that the total amount memory required would remain bounded.

This does not seem to be the case (even for simple cases where you aren't getting into the corner cases like values that point to their keys and such).  Some entries get reclaimed, but the memory usage is not staying beneath any clear bound.

The more flags and tweaks I add to the benchmark to explore its behavior, the more bizarre the behavior appears to me.  In particular:

* The number of entries in the weak dictionary as observed in a for/in loop does necessarily not serve as a predictor for the memory usage.

* Forcing the keys to be non-RC'ed (by turning them into trivial cycles) makes the number of entries in the weak dictionary tend to be much greater.  (I was not surprised that there was growth, but rather with how severe it was.)

Comment 1

[Felix S. Klock II [:pnkfelix, :fklock]]

Another weirdness:
* Emulating a periodic explicit repack of the dictionary by explicitly copying its contents to a fresh weak dictionary and then removing the sole reference to the old one does not reduce the memory usage as much as I expected.  This might mean that I'm misunderstanding where the memory bloat is coming from.  Probably time to switch to instrumenting the internals of the shell in C++ rather than continuing to work in AS3 alone.

Here are the some runs of the benchmark that yielded the observations above.  (Data was gathered on fklockii-iMac, a 2.66 GHz Core i5 with 4 GB RAM, in case that detail matters to me later.)

param:                  b_cyclicInnerCells = false
param:        b_explicitlyRepackDictionary = false
param:                  b_linkedInnerCells = false
param:             b_printOnEveryIteration = false
param:                   i_innerListLength = 100
param:               j_listsBetweenRepacks = 100
param:                       k_repackCount = 100
param:               l_outermostIterations = 200
iter:   1 curr:    49152 max:    49152 len:      330 mem:  2146304
iter:   2 curr:   126976 max:   126976 len:      323 mem:  2224128
iter:   3 curr:   163840 max:   163840 len:      403 mem:  2260992
iter:   4 curr:   176128 max:   176128 len:      391 mem:  2273280
iter:   5 curr:   430080 max:   430080 len:      471 mem:  2527232
iter:   8 curr:   438272 max:   438272 len:      405 mem:  2535424
iter:  13 curr:   446464 max:   446464 len:      229 mem:  2543616
iter:  18 curr:   548864 max:   548864 len:      109 mem:  2646016
iter:  19 curr:   704512 max:   704512 len:      189 mem:  2801664
iter:  20 curr:   946176 max:   946176 len:      177 mem:  3043328
iter:  27 curr:  1056768 max:  1056768 len:      129 mem:  3153920
iter:  33 curr:  1060864 max:  1060864 len:      492 mem:  3158016
iter:  41 curr:  1073152 max:  1073152 len:        1 mem:  3170304
iter:  42 curr:  1150976 max:  1150976 len:       78 mem:  3248128
iter:  78 curr:  1159168 max:  1159168 len:      349 mem:  3256320
iter:  96 curr:  1216512 max:  1216512 len:      388 mem:  3313664
iter: 200 curr:   634880 max:  1216512 len:      403 mem:  2732032



param:                  b_cyclicInnerCells = false
param:        b_explicitlyRepackDictionary = true
param:                  b_linkedInnerCells = false
param:             b_printOnEveryIteration = false
param:                   i_innerListLength = 100
param:               j_listsBetweenRepacks = 100
param:                       k_repackCount = 100
param:               l_outermostIterations = 200
iter:   1 curr:    57344 max:    57344 len:      330 mem:  2154496
iter:   2 curr:   106496 max:   106496 len:      323 mem:  2203648
iter:   3 curr:   167936 max:   167936 len:      403 mem:  2265088
iter:   7 curr:   221184 max:   221184 len:      310 mem:  2318336
iter:  11 curr:   262144 max:   262144 len:       70 mem:  2359296
iter:  16 curr:   450560 max:   450560 len:      406 mem:  2547712
iter:  29 curr:   462848 max:   462848 len:       70 mem:  2560000
iter:  32 curr:   643072 max:   643072 len:        5 mem:  2740224
iter:  34 curr:   905216 max:   905216 len:      478 mem:  3002368
iter:  45 curr:  1142784 max:  1142784 len:      174 mem:  3239936
iter: 200 curr:   651264 max:  1142784 len:       91 mem:  2748416



param:                  b_cyclicInnerCells = true
param:        b_explicitlyRepackDictionary = false
param:                  b_linkedInnerCells = false
param:             b_printOnEveryIteration = false
param:                   i_innerListLength = 100
param:               j_listsBetweenRepacks = 100
param:                       k_repackCount = 100
param:               l_outermostIterations = 200
iter:   1 curr:  9162752 max:  9162752 len:    10212 mem: 11259904
iter:   2 curr: 10604544 max: 10604544 len:    34011 mem: 12701696
iter:   3 curr: 10878976 max: 10878976 len:    46246 mem: 12976128
iter:   4 curr: 11481088 max: 11481088 len:    51924 mem: 13578240
iter:   5 curr: 11657216 max: 11657216 len:    53421 mem: 13754368
iter:  10 curr: 15568896 max: 15568896 len:    89582 mem: 17666048
iter:  13 curr: 16191488 max: 16191488 len:    94989 mem: 18288640
iter:  15 curr: 23928832 max: 23928832 len:   109334 mem: 26025984
iter:  21 curr: 25034752 max: 25034752 len:   127644 mem: 27131904
iter:  29 curr: 26673152 max: 26673152 len:   139563 mem: 28770304
iter:  41 curr: 27439104 max: 27439104 len:   146026 mem: 29536256
iter:  42 curr: 29179904 max: 29179904 len:   162134 mem: 31277056
iter:  58 curr: 29941760 max: 29941760 len:   168380 mem: 32038912
iter:  75 curr: 30060544 max: 30060544 len:   168801 mem: 32157696
iter: 123 curr: 30273536 max: 30273536 len:   170647 mem: 32370688
iter: 128 curr: 32530432 max: 32530432 len:   190228 mem: 34627584
iter: 140 curr: 32735232 max: 32735232 len:   194003 mem: 34832384
iter: 200 curr: 24363008 max: 32735232 len:   116654 mem: 26460160



param:                  b_cyclicInnerCells = true
param:        b_explicitlyRepackDictionary = true
param:                  b_linkedInnerCells = false
param:             b_printOnEveryIteration = false
param:                   i_innerListLength = 100
param:               j_listsBetweenRepacks = 100
param:                       k_repackCount = 100
param:               l_outermostIterations = 200
iter:   1 curr: 17031168 max: 17031168 len:    71838 mem: 19214336
iter:   3 curr: 25788416 max: 25788416 len:   135492 mem: 27885568
iter:   6 curr: 26574848 max: 26574848 len:   143238 mem: 28672000
iter:   9 curr: 27287552 max: 27287552 len:   149665 mem: 29384704
iter:  12 curr: 27844608 max: 27844608 len:   154635 mem: 29941760
iter:  46 curr: 29327360 max: 29327360 len:   164672 mem: 31424512
iter:  52 curr: 29769728 max: 29769728 len:   168484 mem: 31866880
iter:  78 curr: 30154752 max: 30154752 len:   170460 mem: 32251904
iter:  93 curr: 30478336 max: 30478336 len:   173295 mem: 32575488
iter: 114 curr: 30801920 max: 30801920 len:   174858 mem: 32899072
iter: 127 curr: 32030720 max: 32030720 len:   186330 mem: 34127872
iter: 173 curr: 38014976 max: 38014976 len:   201080 mem: 40112128
iter: 191 curr: 38068224 max: 38068224 len:   200728 mem: 40165376
iter: 200 curr: 20013056 max: 38068224 len:    48476 mem: 22110208

Comment 2

[Felix S. Klock II [:pnkfelix, :fklock]]

The issue seems like it might only arise on 64-bit versions of the shell.  (This strengthens my gut feeling that this is a bug, rather than allowable behavior.)

Comment 3

[Felix S. Klock II [:pnkfelix, :fklock]]

(and since I've only done my experiments on OS X, I'll narrow the description's scope on that axis too.)

Comment 4

[Felix S. Klock II [:pnkfelix, :fklock]]

(noticed weak pointers mentioned on Bug 548158, which describes fragmentation yielding sub-par memory utilization.)