657954 - Track down memory usage of buildbot

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

After plugging a few leaks, bm04-tests1 is still using over 4G of memory. It shows interesting growth patterns too:

http://people.mozilla.org/~catlee/sattap/8d6c1691.png

Currently len(gc.garbage) is 0, and len(gc.get_objects()) is 6,802,692

dev-master01 is no longer showing this memory behaviour.

Comment 1

[Chris AtLee [:catlee]]

bm06-tests1 is currently using 3G of memory, has 0 garbage, and has 4,273,053 objects.

Comment 2

[Chris AtLee [:catlee]]

at 18:51 today:

bm04-tests1: 7,643,677 objects; 4,016,020 rss

bm06-tests1: 5,708,081 objects; 2,513,912 rss

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Can you monitor the count by type of gc.get_objects()?  It'd be interesting to see if there's a correlation between count of a certain object type and RSS.

Comment 4

[Chris AtLee [:catlee]]

Sure, here's my little script snippet:

import gc
from types import InstanceType
objs_by_type = {}
for o in gc.get_objects():
    t = type(o)
    if t is InstanceType:
        t = t.__class__
    objs_by_type[t] = objs_by_type.get(t, 0) + 1

objs_by_type = objs_by_type.items()
objs_by_type.sort(key = lambda x:x[1])
for t, count in objs_by_type:
    print t, count

Comment 5

[Chris AtLee [:catlee]]

bm04:

<type 'exceptions.RuntimeError'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class '_ast.Is'> 1
<class '_ast.Param'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'jinja2.loaders.FileSystemLoader'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class 'buildbot.status.web.changes.ChangesResource'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class '_ast.Gt'> 1
<class 'buildbot.status.web.about.AboutBuildbot'> 1
<class '_ast.Mult'> 1
<class '_ast.RShift'> 1
<class '_ast.And'> 1
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.NotIn'> 1
<class 'unittest.TestLoader'> 1
<class 'anyjson._JsonImplementation'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'string._TemplateMetaclass'> 1
<class 'jinja2.utils.MissingType'> 1
<class 'buildbot.status.web.auth.AuthFailResource'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class '_ast.Del'> 1
<class 'twisted.internet._signals._Handler'> 1
<class '_ast.Eq'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class '_ast.GtE'> 1
<class '_ast.Invert'> 1
<class 'calendar.TextCalendar'> 1
<class '_ast.BitOr'> 1
<class 'twisted.python.versions._inf'> 1
<class '_ast.AugLoad'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 1
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 1
<class '_ast.Add'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class '_ast.Div'> 1
<class '_ast.Not'> 1
<class '_ast.Pow'> 1
<class 'ctypes.PyDLL'> 1
<class '_ast.Or'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class 'jinja2.loaders.ChoiceLoader'> 1
<class '_ast.IsNot'> 1
<class 'buildbot.status.web.authz.Authz'> 1
<class '_ast.In'> 1
<class 'threading._MainThread'> 1
<class 'ctypes.CDLL'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'threading._RLock'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class '_ast.NotEq'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class '_ast.Mod'> 1
<class '_ast.BitXor'> 1
<class 'subprocess.Popen'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class '_ast.AugStore'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class 'amqplib.client_0_8.channel.Channel'> 1
<class 'amqplib.client_0_8.serialization.AMQPWriter'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<type 'exceptions.MemoryError'> 1
<class 'site._Helper'> 1
<class 'jinja2.ext.InternationalizationExtension'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'buildbot.status.web.console.ConsoleStatusResource'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'buildbot.status.web.builder.BuildersResource'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class '_ast.Lt'> 1
<class '_ast.UAdd'> 1
<class '_ast.BitAnd'> 1
<class '_ast.Load'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'decimal._Log10Memoize'> 1
<type 'iterator'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 1
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class 'jinja2.loaders.PackageLoader'> 1
<class 'buildbot.status.web.grid.GridStatusResource'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class '_ast.USub'> 1
<class 'zope.interface.adapter.AdapterRegistry'> 1
<class 'pytz.UTC'> 1
<class '_ast.FloorDiv'> 1
<class 'zope.interface.Provides'> 1
<class '_ast.Store'> 1
<class 'mozillapulse.messages.build.BuildMessage'> 1
<class 'jinja2.environment.Environment'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class '_ast.LShift'> 1
<class 'buildbot.util.collections.defaultdict'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'site.Quitter'> 2
<class 'jinja2.runtime.Context'> 2
<class 'random.Random'> 2
<class 'jinja2.nodes.EvalContext'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<type 'tupleiterator'> 2
<type '_ctypes.PointerType'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'ctypes.LibraryLoader'> 2
<type 'generator'> 2
<class 'ctypes._FuncPtr'> 2
<class 'json.scanner.Scanner'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'amqplib.client_0_8.transport.TCPTransport'> 3
<class 'jinja2.environment.TemplateModule'> 3
<class 'carrot.messaging.Publisher'> 3
<class 'jinja2.utils.LRUCache'> 3
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'amqplib.client_0_8.method_framing.MethodReader'> 3
<class 'carrot.backends.pyamqplib.Connection'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'buildbot.status.persistent_queue.IndexedQueue'> 3
<class 'twisted.internet.error.TimeoutError'> 3
<class 'carrot.backends.pyamqplib.Backend'> 3
<type 'collections.defaultdict'> 3
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 3
<class 'twisted.internet.defer.DeferredLock'> 3
<class 'site._Printer'> 3
<class 'carrot.connection.BrokerConnection'> 3
<class 'decimal.Context'> 3
<class 'mozillapulse.publishers.GenericPublisher'> 3
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'MySQLdb.connections.Connection'> 4
<type 'weakproxy'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<class 'twisted.internet.tcp.Client'> 4
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 4
<type 'method-wrapper'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.environment.Template'> 6
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'decimal.Decimal'> 6
<class 'codecs.CodecInfo'> 6
<class 'MySQLdb.DBAPISet'> 7
<class 'twisted.python.threadable.XLock'> 7
<type '_ctypes.CFuncPtrType'> 8
<class 'threading.Thread'> 13
<class 'twisted.python.usage.CoerceParameter'> 13
<type 'listiterator'> 13
<type 'collections.deque'> 14
<class 'twisted.python.versions.Version'> 14
<class 'threading._Event'> 14
<type 'OpenSSL.SSL.Context'> 17
<class 'jinja2.runtime.Macro'> 17
<class 'twisted.internet.tcp.TLSConnection'> 17
<type 'OpenSSL.SSL.Connection'> 17
<class 'pkg_resources.Distribution'> 19
<class 'email.LazyImporter'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<class 'twisted.conch.insults.insults._const'> 26
<type '_ctypes.SimpleType'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'threading._Condition'> 49
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 59
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'socket._socketobject'> 105
<type 'frame'> 137
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'twisted.internet.tcp.Server'> 157
<class 'zope.interface.declarations.ClassProvides'> 190
<class 'zope.interface.declarations.Implements'> 197
<type 'member_descriptor'> 284
<type 'module'> 562
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 843
<type 'classobj'> 926
<type 'wrapper_descriptor'> 1159
<type 'builtin_function_or_method'> 1620
<class 'twisted.internet.task.LoopingCall'> 5154
<type 'cell'> 9045
<class 'weakref.KeyedRef'> 11325
<type 'set'> 22039
<type 'weakref'> 29580
<type 'instancemethod'> 30975
<type 'function'> 78169
<type 'tuple'> 614147
<type 'type'> 1328673
<type 'dict'> 3021115
<type 'list'> 3111428

Comment 6

[Chris AtLee [:catlee]]

bm06:

<type 'exceptions.RuntimeError'> 1
<class '_ast.Del'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class 'ctypes.CFunctionType'> 1
<class 'jinja2.environment.Environment'> 1
<class 'jinja2.loaders.ChoiceLoader'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.grid.GridStatusResource'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class 'anyjson._JsonImplementation'> 1
<class 'threading._RLock'> 1
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 1
<class 'threading._MainThread'> 1
<class 'jinja2.loaders.FileSystemLoader'> 1
<class '_ast.Eq'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'string._TemplateMetaclass'> 1
<class '_ast.Div'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'jinja2.utils.MissingType'> 1
<class 'buildbot.status.web.authz.Authz'> 1
<class 'twisted.internet._signals._Handler'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'jinja2.ext.InternationalizationExtension'> 1
<class '_ast.Mod'> 1
<class '_ast.Invert'> 1
<class '_ast.Pow'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.auth.AuthFailResource'> 1
<class '_ast.Not'> 1
<class '_ast.And'> 1
<class '_ast.Is'> 1
<class 'json.decoder.JSONDecoder'> 1
<class '_ast.IsNot'> 1
<class '_ast.NotIn'> 1
<class 'site._Helper'> 1
<class '_ast.Store'> 1
<class '_ast.NotEq'> 1
<class 'buildbot.status.web.about.AboutBuildbot'> 1
<class '_ast.AugLoad'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class '_ast.BitXor'> 1
<class 'mozillapulse.messages.build.BuildMessage'> 1
<class '_ast.AugStore'> 1
<class '_ast.GtE'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class '_ast.Sub'> 1
<class 'amqplib.client_0_8.channel.Channel'> 1
<class 'zope.interface.Provides'> 1
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class 'jinja2.loaders.PackageLoader'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<type 'iterator'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 1
<class '_ast.Lt'> 1
<class '_ast.Add'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class '_ast.UAdd'> 1
<class '_ast.BitAnd'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class '_ast.Load'> 1
<class '_ast.Param'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class 'ctypes.CFunctionType'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class 'twisted.python.versions._inf'> 1
<class 'ctypes.PyDLL'> 1
<class 'calendar.TextCalendar'> 1
<type 'exceptions.MemoryError'> 1
<class 'decimal._Log10Memoize'> 1
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class '_ast.LtE'> 1
<class '_ast.In'> 1
<class 'pytz.UTC'> 1
<class '_ast.USub'> 1
<class '_ast.LShift'> 1
<class '_ast.FloorDiv'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class 'ctypes.CFunctionType'> 1
<class 'amqplib.client_0_8.serialization.AMQPWriter'> 1
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'unittest.TestLoader'> 1
<class 'buildbot.status.web.console.ConsoleStatusResource'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class 'zope.interface.adapter.AdapterRegistry'> 1
<class 'ctypes.CDLL'> 1
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 1
<class '_ast.Gt'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class '_ast.Mult'> 1
<class 'buildbot.status.web.changes.ChangesResource'> 1
<class '_ast.RShift'> 1
<class 'buildbot.status.web.builder.BuildersResource'> 1
<class 'carrot.connection.BrokerConnection'> 2
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 2
<class 'site.Quitter'> 2
<class 'jinja2.nodes.EvalContext'> 2
<class 'ctypes._FuncPtr'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<class 'subprocess.Popen'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<type 'tupleiterator'> 2
<class 'twisted.internet.defer.DeferredLock'> 2
<class 'random.Random'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<class 'jinja2.lexer.Failure'> 2
<type 'collections.defaultdict'> 2
<class 'amqplib.client_0_8.method_framing.MethodReader'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'jinja2.runtime.Context'> 2
<class 'amqplib.client_0_8.transport.TCPTransport'> 2
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'json.scanner.Scanner'> 2
<class 'carrot.backends.pyamqplib.Connection'> 2
<class 'buildbot.status.persistent_queue.IndexedQueue'> 2
<type 'generator'> 2
<class 'mozillapulse.publishers.GenericPublisher'> 2
<class 'ctypes.LibraryLoader'> 2
<type '_ctypes.PointerType'> 2
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'jinja2.utils.LRUCache'> 3
<class 'jinja2.environment.TemplateModule'> 3
<class 'decimal.Context'> 3
<class 'site._Printer'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'carrot.messaging.Publisher'> 3
<class 'carrot.backends.pyamqplib.Backend'> 3
<class 'abc.abstractproperty'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<class 'uuid.UUID'> 4
<class 'MySQLdb.connections.Connection'> 4
<class 'twisted.internet.error.TimeoutError'> 4
<type 'weakproxy'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<type 'method-wrapper'> 5
<class 'jinja2.environment.Template'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'decimal.Decimal'> 6
<class 'codecs.CodecInfo'> 6
<class 'twisted.internet.tcp.Client'> 6
<class 'MySQLdb.DBAPISet'> 7
<class 'twisted.python.threadable.XLock'> 7
<type '_ctypes.CFuncPtrType'> 8
<type 'OpenSSL.SSL.Context'> 9
<class 'twisted.internet.tcp.TLSConnection'> 9
<type 'OpenSSL.SSL.Connection'> 9
<type 'listiterator'> 10
<type 'collections.deque'> 12
<class 'threading.Thread'> 13
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'threading._Event'> 14
<class 'twisted.python.versions.Version'> 14
<class 'jinja2.runtime.Macro'> 17
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 19
<class 'email.LazyImporter'> 19
<class 'pkg_resources.Distribution'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<type '_ctypes.SimpleType'> 26
<class 'twisted.conch.insults.insults._const'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'threading._Condition'> 46
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 49
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'socket._socketobject'> 103
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<type 'frame'> 143
<class 'twisted.internet.tcp.Server'> 163
<class 'zope.interface.declarations.ClassProvides'> 187
<class 'zope.interface.declarations.Implements'> 194
<type 'member_descriptor'> 284
<type 'module'> 562
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 843
<type 'classobj'> 923
<type 'wrapper_descriptor'> 1159
<type 'builtin_function_or_method'> 1607
<class 'twisted.internet.task.LoopingCall'> 5153
<type 'cell'> 6039
<class 'weakref.KeyedRef'> 8088
<type 'set'> 16910
<type 'instancemethod'> 22571
<type 'weakref'> 24437
<type 'function'> 70542
<type 'tuple'> 483861
<type 'type'> 1057945
<type 'list'> 2342078
<type 'dict'> 2342591

Comment 7

[Chris AtLee [:catlee]]

bm04 rss 4559192
<type 'exceptions.RuntimeError'> 1
<class '_ast.Not'> 1
<class '_ast.Pow'> 1
<class '_ast.UAdd'> 1
<class '_ast.RShift'> 1
<class '_ast.IsNot'> 1
<class '_ast.In'> 1
<class '_ast.NotIn'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class '_ast.Lt'> 1
<class '_ast.Add'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class '_ast.BitAnd'> 1
<class 'threading._RLock'> 1
<class '_ast.AugLoad'> 1
<class '_ast.Load'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'twisted.python.versions._inf'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'twisted.internet._signals._Handler'> 1
<class 'site._Helper'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'ctypes.CFunctionType'> 1
<class 'string._TemplateMetaclass'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class '_ast.USub'> 1
<class '_ast.LShift'> 1
<class '_ast.FloorDiv'> 1
<class '_ast.Store'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class 'zope.interface.Provides'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class 'ctypes.CDLL'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class '_ast.Gt'> 1
<class 'anyjson._JsonImplementation'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'twisted.internet.tcp.TLSConnection'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'calendar.TextCalendar'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class '_ast.Mult'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.And'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class '_ast.Del'> 1
<class 'jinja2.utils.MissingType'> 1
<class '_ast.BitXor'> 1
<type 'iterator'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'ctypes.CFunctionType'> 1
<class 'ctypes.CFunctionType'> 1
<class 'zope.interface.adapter.AdapterRegistry'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'unittest.TestLoader'> 1
<type 'exceptions.MemoryError'> 1
<class 'ctypes.PyDLL'> 1
<class '_ast.Eq'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class '_ast.GtE'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'pytz.UTC'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'subprocess.Popen'> 1
<class 'threading._MainThread'> 1
<class '_ast.Param'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class '_ast.Mod'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class '_ast.AugStore'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'twisted.internet.error.TimeoutError'> 1
<class 'twisted.internet.error.ConnectionLost'> 2
<class 'random.Random'> 2
<class 'json.scanner.Scanner'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<type 'tupleiterator'> 2
<class 'jinja2.environment.Template'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'ctypes._FuncPtr'> 2
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 2
<class 'mozillapulse.messages.build.BuildMessage'> 2
<class 'amqplib.client_0_8.serialization.AMQPWriter'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'jinja2.lexer.Failure'> 2
<type 'listiterator'> 2
<class 'amqplib.client_0_8.channel.Channel'> 2
<type '_ctypes.PointerType'> 2
<class 'site.Quitter'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'jinja2.loaders.FileSystemLoader'> 3
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 3
<class 'buildbot.status.web.console.ConsoleStatusResource'> 3
<class 'amqplib.client_0_8.transport.TCPTransport'> 3
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 3
<class 'jinja2.loaders.ChoiceLoader'> 3
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 3
<class 'jinja2.ext.InternationalizationExtension'> 3
<class 'jinja2.loaders.PackageLoader'> 3
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'buildbot.status.web.about.AboutBuildbot'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'decimal.Context'> 3
<class 'site._Printer'> 3
<type 'collections.defaultdict'> 3
<class 'buildbot.status.web.grid.GridStatusResource'> 3
<class 'jinja2.environment.Environment'> 3
<class 'buildbot.status.web.auth.AuthFailResource'> 3
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 3
<class 'buildbot.status.web.changes.ChangesResource'> 3
<class 'buildbot.status.web.builder.BuildersResource'> 3
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 3
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 3
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 3
<class 'amqplib.client_0_8.method_framing.MethodReader'> 3
<class 'buildbot.status.web.authz.Authz'> 3
<class 'carrot.backends.pyamqplib.Connection'> 4
<class 'MySQLdb.connections.Connection'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<type 'weakproxy'> 4
<class 'carrot.connection.BrokerConnection'> 4
<class 'twisted.internet.defer.DeferredLock'> 5
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 5
<class 'buildbot.status.persistent_queue.IndexedQueue'> 5
<class 'carrot.backends.pyamqplib.Backend'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'carrot.messaging.Publisher'> 5
<class 'jinja2.utils.LRUCache'> 5
<type 'method-wrapper'> 5
<class 'mozillapulse.publishers.GenericPublisher'> 5
<class 'socket.gaierror'> 6
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'codecs.CodecInfo'> 6
<class 'decimal.Decimal'> 6
<class 'twisted.python.threadable.XLock'> 7
<class 'MySQLdb.DBAPISet'> 7
<type '_ctypes.CFuncPtrType'> 8
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 10
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'threading.Thread'> 13
<class 'threading._Event'> 14
<class 'twisted.python.versions.Version'> 14
<class 'urlparse.SplitResult'> 16
<type 'traceback'> 18
<class 'email.LazyImporter'> 19
<class 'pkg_resources.Distribution'> 19
<type 'collections.deque'> 20
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<class 'twisted.conch.insults.insults._const'> 26
<type '_ctypes.SimpleType'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'threading._Condition'> 55
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'twisted.internet.tcp.Server'> 70
<class 'socket._socketobject'> 75
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'zope.interface.declarations.ClassProvides'> 194
<class 'zope.interface.declarations.Implements'> 201
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 943
<type 'generator'> 1004
<type 'wrapper_descriptor'> 1159
<type 'frame'> 1164
<type 'builtin_function_or_method'> 1465
<class 'twisted.internet.task.LoopingCall'> 5072
<type 'cell'> 11890
<class 'weakref.KeyedRef'> 13307
<type 'instancemethod'> 22933
<type 'set'> 25631
<type 'weakref'> 33129
<type 'function'> 76983
<type 'tuple'> 1335293
<type 'type'> 1464732
<type 'dict'> 3518250
<type 'list'> 3624438

Comment 8

[Chris AtLee [:catlee]]

bm06 rss 2672624
<class 'zope.interface.adapter.AdapterRegistry'> 1
<type 'exceptions.RuntimeError'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'jinja2.loaders.PackageLoader'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.authz.Authz'> 1
<type 'iterator'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'string._TemplateMetaclass'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class 'site._Helper'> 1
<class 'jinja2.lexer.Lexer'> 1
<type 'exceptions.MemoryError'> 1
<class '_ast.Lt'> 1
<class '_ast.Not'> 1
<class '_ast.Add'> 1
<class '_ast.UAdd'> 1
<class '_ast.GtE'> 1
<class '_ast.Pow'> 1
<class '_ast.LShift'> 1
<class '_ast.BitAnd'> 1
<class 'zope.interface.Provides'> 1
<class '_ast.IsNot'> 1
<class '_ast.Load'> 1
<class 'twisted.internet._signals._Handler'> 1
<class '_ast.In'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class '_ast.NotIn'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class '_ast.Param'> 1
<class '_ast.Del'> 1
<class 'carrot.backends.pyamqplib.Connection'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'mozillapulse.publishers.GenericPublisher'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'buildbot.status.web.builder.BuildersResource'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class '_ast.USub'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class '_ast.FloorDiv'> 1
<class 'buildbot.status.persistent_queue.IndexedQueue'> 1
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 1
<class 'threading._RLock'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'threading._MainThread'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.console.ConsoleStatusResource'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'ctypes.PyDLL'> 1
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class '_ast.Gt'> 1
<class '_ast.Mult'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class '_ast.RShift'> 1
<class '_ast.And'> 1
<class 'buildbot.status.web.about.AboutBuildbot'> 1
<class 'jinja2.loaders.ChoiceLoader'> 1
<class 'jinja2.environment.Environment'> 1
<class 'buildbot.status.web.grid.GridStatusResource'> 1
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class 'unittest.TestLoader'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class '_ast.Store'> 1
<class 'ctypes.CDLL'> 1
<class 'jinja2.ext.InternationalizationExtension'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'buildbot.status.web.changes.ChangesResource'> 1
<class 'anyjson._JsonImplementation'> 1
<class '_ast.Eq'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<class '_ast.AugLoad'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'ctypes.CFunctionType'> 1
<class 'jinja2.utils.MissingType'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.versions._inf'> 1
<type 'listiterator'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class 'twisted.internet.error.TimeoutError'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class 'buildbot.status.web.auth.AuthFailResource'> 1
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 1
<class 'jinja2.loaders.FileSystemLoader'> 1
<class 'calendar.TextCalendar'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class 'subprocess.Popen'> 1
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 1
<class 'pytz.UTC'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class '_ast.Mod'> 1
<class 'twisted.internet.defer.DeferredLock'> 1
<class '_ast.BitXor'> 1
<class '_ast.AugStore'> 1
<class 'ctypes._FuncPtr'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 2
<type 'tupleiterator'> 2
<type '_ctypes.PointerType'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'json.scanner.Scanner'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<type 'generator'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'site.Quitter'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<class 'random.Random'> 2
<class 'jinja2.runtime.Context'> 3
<class 'decimal.Context'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'jinja2.utils.LRUCache'> 3
<class 'site._Printer'> 3
<class 'jinja2.nodes.EvalContext'> 3
<class 'jinja2.environment.TemplateModule'> 3
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<type 'weakproxy'> 4
<class 'MySQLdb.connections.Connection'> 4
<type 'method-wrapper'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'decimal.Decimal'> 6
<class 'codecs.CodecInfo'> 6
<type 'collections.deque'> 7
<class 'MySQLdb.DBAPISet'> 7
<class 'twisted.python.threadable.XLock'> 7
<type '_ctypes.CFuncPtrType'> 8
<class 'threading.Thread'> 9
<class 'jinja2.environment.Template'> 9
<class 'threading._Event'> 10
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'twisted.python.versions.Version'> 14
<class 'jinja2.runtime.Macro'> 16
<class 'email.LazyImporter'> 19
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 19
<class 'pkg_resources.Distribution'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<type '_ctypes.SimpleType'> 26
<class 'twisted.conch.insults.insults._const'> 26
<class 'threading._Condition'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'twisted.internet.tcp.Server'> 70
<class 'socket._socketobject'> 73
<type 'frame'> 102
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'zope.interface.declarations.ClassProvides'> 182
<class 'zope.interface.declarations.Implements'> 189
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 934
<type 'wrapper_descriptor'> 1159
<type 'builtin_function_or_method'> 1391
<class 'weakref.KeyedRef'> 2353
<type 'cell'> 3932
<class 'twisted.internet.task.LoopingCall'> 5068
<type 'set'> 9395
<type 'weakref'> 16818
<type 'instancemethod'> 20958
<type 'function'> 64010
<type 'tuple'> 512759
<type 'type'> 537225
<type 'list'> 936245
<type 'dict'> 1087015

Comment 9

[Chris AtLee [:catlee]]

bm04 rss 4562152
<type 'exceptions.RuntimeError'> 1
<class '_ast.Gt'> 1
<class '_ast.Pow'> 1
<class '_ast.UAdd'> 1
<class '_ast.RShift'> 1
<class '_ast.IsNot'> 1
<class '_ast.In'> 1
<class '_ast.NotIn'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class '_ast.Lt'> 1
<class '_ast.Add'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.BitAnd'> 1
<class 'threading._RLock'> 1
<class '_ast.AugLoad'> 1
<class '_ast.Load'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'twisted.internet._signals._Handler'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'zope.interface.Provides'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class 'string._TemplateMetaclass'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class '_ast.USub'> 1
<class '_ast.LShift'> 1
<class '_ast.FloorDiv'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'ctypes.CFunctionType'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class 'site._Helper'> 1
<class 'ctypes.CDLL'> 1
<class 'anyjson._JsonImplementation'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'calendar.TextCalendar'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class '_ast.Mult'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.And'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class '_ast.Del'> 1
<class 'jinja2.utils.MissingType'> 1
<class '_ast.BitXor'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'ctypes.CFunctionType'> 1
<class 'zope.interface.adapter.AdapterRegistry'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'unittest.TestLoader'> 1
<type 'exceptions.MemoryError'> 1
<class 'ctypes.PyDLL'> 1
<class '_ast.Eq'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class '_ast.GtE'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<type 'iterator'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'twisted.python.versions._inf'> 1
<class 'pytz.UTC'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<type 'listiterator'> 1
<class '_ast.Param'> 1
<class '_ast.Store'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class '_ast.Not'> 1
<class '_ast.Mod'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class '_ast.AugStore'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'twisted.internet.tcp.Client'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'threading._MainThread'> 1
<class 'random.Random'> 2
<class 'subprocess.Popen'> 2
<class 'json.scanner.Scanner'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<type 'tupleiterator'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'jinja2.nodes.EvalContext'> 2
<class 'ctypes._FuncPtr'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'jinja2.runtime.Context'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'twisted.internet.error.TimeoutError'> 2
<type '_ctypes.PointerType'> 2
<class 'site.Quitter'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'jinja2.loaders.FileSystemLoader'> 3
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 3
<class 'buildbot.status.web.console.ConsoleStatusResource'> 3
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 3
<class 'jinja2.loaders.ChoiceLoader'> 3
<class 'amqplib.client_0_8.serialization.AMQPWriter'> 3
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 3
<class 'mozillapulse.messages.build.BuildMessage'> 3
<class 'jinja2.ext.InternationalizationExtension'> 3
<class 'jinja2.loaders.PackageLoader'> 3
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'buildbot.status.web.about.AboutBuildbot'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'jinja2.environment.Environment'> 3
<class 'decimal.Context'> 3
<class 'site._Printer'> 3
<class 'buildbot.status.web.grid.GridStatusResource'> 3
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 3
<class 'amqplib.client_0_8.channel.Channel'> 3
<class 'buildbot.status.web.auth.AuthFailResource'> 3
<class 'buildbot.status.web.changes.ChangesResource'> 3
<class 'buildbot.status.web.builder.BuildersResource'> 3
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 3
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 3
<class 'buildbot.status.web.authz.Authz'> 3
<class 'jinja2.environment.TemplateModule'> 3
<class 'MySQLdb.connections.Connection'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<type 'weakproxy'> 4
<class 'amqplib.client_0_8.transport.TCPTransport'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.utils.LRUCache'> 5
<type 'method-wrapper'> 5
<type 'collections.defaultdict'> 5
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 5
<class 'amqplib.client_0_8.method_framing.MethodReader'> 5
<class 'carrot.backends.pyamqplib.Connection'> 5
<class 'twisted.internet.defer.DeferredLock'> 6
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 6
<class 'socket.gaierror'> 6
<class 'buildbot.status.persistent_queue.IndexedQueue'> 6
<class 'jinja2.environment.Template'> 6
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'codecs.CodecInfo'> 6
<class 'mozillapulse.publishers.GenericPublisher'> 6
<class 'decimal.Decimal'> 6
<class 'carrot.connection.BrokerConnection'> 6
<class 'carrot.messaging.Publisher'> 7
<class 'twisted.python.threadable.XLock'> 7
<class 'MySQLdb.DBAPISet'> 7
<class 'carrot.backends.pyamqplib.Backend'> 7
<type '_ctypes.CFuncPtrType'> 8
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 11
<class 'threading.Thread'> 13
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'threading._Event'> 14
<class 'twisted.python.versions.Version'> 14
<class 'urlparse.SplitResult'> 16
<class 'jinja2.runtime.Macro'> 17
<type 'traceback'> 18
<class 'email.LazyImporter'> 19
<class 'pkg_resources.Distribution'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<type 'collections.deque'> 25
<class 'twisted.conch.insults.insults._const'> 26
<type '_ctypes.SimpleType'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 35
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'threading._Condition'> 67
<class 'twisted.internet.tcp.Server'> 68
<class 'socket._socketobject'> 76
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'zope.interface.declarations.ClassProvides'> 199
<class 'zope.interface.declarations.Implements'> 206
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 948
<type 'generator'> 1004
<type 'wrapper_descriptor'> 1159
<type 'frame'> 1180
<type 'builtin_function_or_method'> 1487
<class 'twisted.internet.task.LoopingCall'> 5142
<type 'cell'> 12172
<class 'weakref.KeyedRef'> 14571
<type 'instancemethod'> 23716
<type 'set'> 28082
<type 'weakref'> 35670
<type 'function'> 78150
<type 'tuple'> 1128916
<type 'type'> 1572192
<type 'dict'> 3731981
<type 'list'> 3893095

Comment 10

[Chris AtLee [:catlee]]

bm06 rss 2713328
<class 'zope.interface.adapter.AdapterRegistry'> 1
<type 'exceptions.RuntimeError'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'buildbot.status.web.authz.Authz'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class 'jinja2.lexer.Lexer'> 1
<type 'exceptions.MemoryError'> 1
<class '_ast.Lt'> 1
<class '_ast.Not'> 1
<class '_ast.Add'> 1
<class '_ast.UAdd'> 1
<class '_ast.GtE'> 1
<class '_ast.Pow'> 1
<class '_ast.LShift'> 1
<class '_ast.BitAnd'> 1
<class '_ast.USub'> 1
<class 'zope.interface.Provides'> 1
<class '_ast.IsNot'> 1
<class '_ast.Load'> 1
<class 'twisted.internet._signals._Handler'> 1
<class '_ast.In'> 1
<class '_ast.Param'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class '_ast.NotIn'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.Del'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'buildbot.status.web.builder.BuildersResource'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class '_ast.FloorDiv'> 1
<class '_ast.Store'> 1
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 1
<class 'threading._RLock'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'threading._MainThread'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.console.ConsoleStatusResource'> 1
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'ctypes.PyDLL'> 1
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class '_ast.Gt'> 1
<class '_ast.Mult'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class '_ast.RShift'> 1
<class '_ast.And'> 1
<class 'buildbot.status.web.about.AboutBuildbot'> 1
<class 'jinja2.loaders.ChoiceLoader'> 1
<class 'jinja2.environment.Environment'> 1
<class 'buildbot.status.web.grid.GridStatusResource'> 1
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 1
<class 'string._TemplateMetaclass'> 1
<class 'unittest.TestLoader'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class 'ctypes.CFunctionType'> 1
<class 'ctypes.CDLL'> 1
<class 'jinja2.ext.InternationalizationExtension'> 1
<class 'buildbot.status.web.changes.ChangesResource'> 1
<class 'anyjson._JsonImplementation'> 1
<class '_ast.Eq'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<type 'iterator'> 1
<class 'site._Helper'> 1
<class '_ast.AugLoad'> 1
<class 'ctypes.CFunctionType'> 1
<class 'jinja2.utils.MissingType'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.versions._inf'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class 'twisted.internet.error.TimeoutError'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class 'buildbot.status.web.auth.AuthFailResource'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class 'jinja2.loaders.FileSystemLoader'> 1
<class 'calendar.TextCalendar'> 1
<class 'jinja2.loaders.PackageLoader'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class 'subprocess.Popen'> 1
<class 'pytz.UTC'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class '_ast.Mod'> 1
<class '_ast.BitXor'> 1
<class '_ast.AugStore'> 1
<class 'amqplib.client_0_8.transport.TCPTransport'> 2
<class 'ctypes._FuncPtr'> 2
<class 'carrot.backends.pyamqplib.Backend'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'carrot.backends.pyamqplib.Connection'> 2
<class 'jinja2.runtime.Context'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<type 'tupleiterator'> 2
<type '_ctypes.PointerType'> 2
<class 'amqplib.client_0_8.method_framing.MethodReader'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<type 'collections.defaultdict'> 2
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 2
<class 'carrot.connection.BrokerConnection'> 2
<class 'json.scanner.Scanner'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<class 'carrot.messaging.Publisher'> 2
<class 'jinja2.nodes.EvalContext'> 2
<type 'generator'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'site.Quitter'> 2
<class 'random.Random'> 2
<class 'buildbot.status.persistent_queue.IndexedQueue'> 3
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 3
<class 'twisted.internet.tcp.TLSConnection'> 3
<class 'mozillapulse.publishers.GenericPublisher'> 3
<class 'decimal.Context'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'site._Printer'> 3
<class 'jinja2.utils.LRUCache'> 3
<type 'listiterator'> 3
<class 'jinja2.environment.TemplateModule'> 3
<class 'twisted.internet.defer.DeferredLock'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'MySQLdb.connections.Connection'> 4
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<type 'method-wrapper'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<type 'weakproxy'> 5
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'decimal.Decimal'> 6
<class 'codecs.CodecInfo'> 6
<class 'jinja2.environment.Template'> 6
<class 'MySQLdb.DBAPISet'> 7
<class 'twisted.python.threadable.XLock'> 7
<type '_ctypes.CFuncPtrType'> 8
<class 'threading.Thread'> 11
<type 'collections.deque'> 11
<class 'threading._Event'> 12
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'twisted.python.versions.Version'> 14
<class 'jinja2.runtime.Macro'> 17
<class 'pkg_resources.Distribution'> 19
<class 'email.LazyImporter'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<type '_ctypes.SimpleType'> 26
<class 'twisted.conch.insults.insults._const'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<class 'threading._Condition'> 36
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 46
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'twisted.internet.tcp.Server'> 70
<class 'socket._socketobject'> 74
<type 'frame'> 114
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'zope.interface.declarations.ClassProvides'> 190
<class 'zope.interface.declarations.Implements'> 197
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 942
<type 'wrapper_descriptor'> 1159
<type 'builtin_function_or_method'> 1413
<type 'cell'> 4071
<class 'twisted.internet.task.LoopingCall'> 5139
<class 'weakref.KeyedRef'> 7330
<type 'set'> 16261
<type 'instancemethod'> 21720
<type 'weakref'> 23800
<type 'function'> 64801
<type 'tuple'> 643189
<type 'type'> 961849
<type 'list'> 2144853
<type 'dict'> 2162979

Comment 11

[Chris AtLee [:catlee]]

bm04 rss 4566292

<type 'exceptions.RuntimeError'> 1
<class '_ast.Gt'> 1
<class '_ast.Pow'> 1
<class '_ast.UAdd'> 1
<class '_ast.RShift'> 1
<class '_ast.IsNot'> 1
<class '_ast.In'> 1
<class '_ast.NotIn'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class '_ast.Lt'> 1
<class '_ast.Add'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class '_ast.BitAnd'> 1
<class 'threading._RLock'> 1
<class '_ast.AugLoad'> 1
<class '_ast.Load'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'twisted.internet._signals._Handler'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'ctypes.CFunctionType'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'zope.interface.Provides'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class 'string._TemplateMetaclass'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class '_ast.USub'> 1
<class '_ast.LShift'> 1
<class '_ast.FloorDiv'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'ctypes.CFunctionType'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class 'site._Helper'> 1
<class 'ctypes.CDLL'> 1
<class 'anyjson._JsonImplementation'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'calendar.TextCalendar'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class '_ast.Mult'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.And'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class '_ast.Del'> 1
<class 'jinja2.utils.MissingType'> 1
<class '_ast.BitXor'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'ctypes.CFunctionType'> 1
<class 'zope.interface.adapter.AdapterRegistry'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'unittest.TestLoader'> 1
<type 'exceptions.MemoryError'> 1
<class 'ctypes.PyDLL'> 1
<class '_ast.Eq'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class '_ast.GtE'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<type 'iterator'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'twisted.python.versions._inf'> 1
<class 'pytz.UTC'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'subprocess.Popen'> 1
<class '_ast.Param'> 1
<class '_ast.Store'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class '_ast.Not'> 1
<class '_ast.Mod'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class '_ast.AugStore'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'twisted.internet.tcp.Client'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'threading._MainThread'> 1
<class 'random.Random'> 2
<type 'tupleiterator'> 2
<class 'json.scanner.Scanner'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'jinja2.nodes.EvalContext'> 2
<class 'ctypes._FuncPtr'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'jinja2.runtime.Context'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'twisted.internet.error.TimeoutError'> 2
<type '_ctypes.PointerType'> 2
<class 'site.Quitter'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'jinja2.loaders.FileSystemLoader'> 3
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 3
<class 'buildbot.status.web.console.ConsoleStatusResource'> 3
<class 'socket.gaierror'> 3
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 3
<class 'jinja2.loaders.ChoiceLoader'> 3
<class 'amqplib.client_0_8.serialization.AMQPWriter'> 3
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 3
<class 'mozillapulse.messages.build.BuildMessage'> 3
<class 'jinja2.ext.InternationalizationExtension'> 3
<class 'jinja2.loaders.PackageLoader'> 3
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'buildbot.status.web.about.AboutBuildbot'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'jinja2.environment.Environment'> 3
<class 'decimal.Context'> 3
<class 'site._Printer'> 3
<class 'buildbot.status.web.grid.GridStatusResource'> 3
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 3
<class 'amqplib.client_0_8.channel.Channel'> 3
<class 'buildbot.status.web.auth.AuthFailResource'> 3
<class 'buildbot.status.web.changes.ChangesResource'> 3
<class 'buildbot.status.web.builder.BuildersResource'> 3
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 3
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 3
<class 'buildbot.status.web.authz.Authz'> 3
<class 'MySQLdb.connections.Connection'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<type 'weakproxy'> 4
<class 'jinja2.environment.TemplateModule'> 4
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.utils.LRUCache'> 5
<type 'method-wrapper'> 5
<class 'amqplib.client_0_8.transport.TCPTransport'> 6
<type 'exceptions.KeyError'> 6
<type 'collections.defaultdict'> 6
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'codecs.CodecInfo'> 6
<class 'decimal.Decimal'> 6
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 6
<class 'amqplib.client_0_8.method_framing.MethodReader'> 6
<class 'carrot.backends.pyamqplib.Connection'> 6
<class 'twisted.internet.defer.DeferredLock'> 7
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 7
<class 'buildbot.status.persistent_queue.IndexedQueue'> 7
<class 'twisted.python.threadable.XLock'> 7
<class 'MySQLdb.DBAPISet'> 7
<class 'mozillapulse.publishers.GenericPublisher'> 7
<class 'carrot.connection.BrokerConnection'> 7
<type 'OpenSSL.SSL.Connection'> 8
<type '_ctypes.CFuncPtrType'> 8
<class 'twisted.internet.tcp.TLSConnection'> 8
<type 'listiterator'> 8
<type 'OpenSSL.SSL.Context'> 8
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 9
<class 'jinja2.environment.Template'> 9
<class 'carrot.messaging.Publisher'> 9
<class 'carrot.backends.pyamqplib.Backend'> 9
<class 'threading.Thread'> 13
<class 'twisted.python.usage.CoerceParameter'> 13
<class 'threading._Event'> 14
<class 'twisted.python.versions.Version'> 14
<class 'urlparse.SplitResult'> 16
<class 'email.LazyImporter'> 19
<class 'pkg_resources.Distribution'> 19
<type 'classmethod_descriptor'> 20
<class 'jinja2.runtime.Macro'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<class 'twisted.conch.insults.insults._const'> 26
<type '_ctypes.SimpleType'> 26
<type 'staticmethod'> 27
<type 'collections.deque'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<type 'operator.itemgetter'> 43
<type 'traceback'> 63
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<class 'twisted.internet.tcp.Server'> 68
<class 'threading._Condition'> 70
<class 'socket._socketobject'> 85
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<class 'zope.interface.declarations.ClassProvides'> 200
<class 'zope.interface.declarations.Implements'> 207
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 949
<type 'generator'> 1004
<type 'wrapper_descriptor'> 1159
<type 'frame'> 1218
<type 'builtin_function_or_method'> 1542
<class 'twisted.internet.task.LoopingCall'> 5143
<type 'cell'> 13525
<class 'weakref.KeyedRef'> 15220
<type 'set'> 29133
<type 'instancemethod'> 30859
<type 'weakref'> 36725
<type 'function'> 80178
<type 'tuple'> 1173849
<type 'type'> 1630288
<type 'dict'> 3882331
<type 'list'> 4058157

Comment 12

[Chris AtLee [:catlee]]

bm06 rss 2731780

<class 'zope.interface.adapter.AdapterRegistry'> 1
<type 'exceptions.RuntimeError'> 1
<class 'json.encoder.JSONEncoder'> 1
<class 'pkg_resources.WorkingSet'> 1
<class 'twisted.conch.manhole.ColoredManhole'> 1
<class 'buildbot.status.web.authz.Authz'> 1
<class 'twisted.python.win32._ErrorFormatter'> 1
<class 'buildbot.status.web.root.RootPage'> 1
<class 'string._TemplateMetaclass'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.waterfall.WaterfallHelp'> 1
<class 'Crypto.PublicKey.RSA.RSAImplementation'> 1
<class 'jinja2.lexer.Lexer'> 1
<class 'buildbotcustom.steps.unittest.MozillaPackagedReftests'> 1
<type 'exceptions.MemoryError'> 1
<class '_ast.Lt'> 1
<class '_ast.Not'> 1
<class '_ast.Add'> 1
<class '_ast.UAdd'> 1
<class '_ast.GtE'> 1
<class '_ast.Pow'> 1
<class '_ast.LShift'> 1
<class '_ast.BitAnd'> 1
<class '_ast.USub'> 1
<class 'zope.interface.Provides'> 1
<class '_ast.IsNot'> 1
<class '_ast.Load'> 1
<class 'twisted.internet._signals._Handler'> 1
<class '_ast.In'> 1
<class '_ast.Param'> 1
<class 'twisted.internet.task.Cooperator'> 1
<class '_ast.NotIn'> 1
<class 'twisted.internet.tcp.TLSConnection'> 1
<class 'twisted.application.app.AppProfiler'> 1
<class 'ctypes.CFunctionType'> 1
<class '_ast.Del'> 1
<class 'twisted.python.modules.PythonPath'> 1
<class 'json.decoder.JSONDecoder'> 1
<class 'buildbot.util.eventual._SimpleCallQueue'> 1
<class 'buildbot.status.web.slaves.BuildSlavesResource'> 1
<class 'zope.interface.adapter.AdapterLookup'> 1
<class 'twisted.internet.base.ThreadedResolver'> 1
<class 'twisted.scripts._twistd_unix.UnixApplicationRunner'> 1
<class 'buildbot.status.web.builder.BuildersResource'> 1
<class '_ast.LtE'> 1
<class '_ast.Sub'> 1
<class 'twisted.scripts._twistd_unix.UnixAppLogger'> 1
<class '_ast.FloorDiv'> 1
<class '_ast.Store'> 1
<class 'buildbot.status.web.buildstatus.BuildStatusStatusResource'> 1
<class 'threading._RLock'> 1
<class 'buildbot.scripts.runner.Options'> 1
<class 'threading._MainThread'> 1
<class 'ctypes.CFunctionType'> 1
<class 'buildbot.status.web.console.ConsoleStatusResource'> 1
<class 'buildbot.status.web.grid.TransposedGridStatusResource'> 1
<class 'Crypto.PublicKey.DSA.DSAImplementation'> 1
<class 'ctypes.PyDLL'> 1
<class 'buildbot.status.web.waterfall.WaterfallStatusResource'> 1
<class 'carrot.serialization.SerializerRegistry'> 1
<class '_ast.Gt'> 1
<class '_ast.Mult'> 1
<class 'twisted.python.randbytes.RandomFactory'> 1
<class '_ast.RShift'> 1
<class '_ast.And'> 1
<class 'buildbot.status.web.about.AboutBuildbot'> 1
<class 'jinja2.loaders.ChoiceLoader'> 1
<class 'jinja2.environment.Environment'> 1
<class 'buildbot.status.web.grid.GridStatusResource'> 1
<class 'buildbot.status.web.olpb.OneLinePerBuild'> 1
<class 'twisted.scripts._twistd_unix.ServerOptions'> 1
<class 'unittest.TestLoader'> 1
<class 'twisted.internet.posixbase._UnixWaker'> 1
<class 'buildbot.status.web.console.IntegerRevisionComparator'> 1
<class 'twisted.internet.selectreactor.SelectReactor'> 1
<class 'twisted.python.modules._DefaultMapImpl'> 1
<class 'ctypes.CFunctionType'> 1
<class 'ctypes.CDLL'> 1
<class 'jinja2.ext.InternationalizationExtension'> 1
<class 'buildbot.status.web.changes.ChangesResource'> 1
<class 'anyjson._JsonImplementation'> 1
<class '_ast.Eq'> 1
<class 'zope.interface.declarations.Declaration'> 1
<class '_ast.Invert'> 1
<class '_ast.Div'> 1
<class '_ast.BitOr'> 1
<class '_ast.Or'> 1
<type 'iterator'> 1
<class 'site._Helper'> 1
<class '_ast.AugLoad'> 1
<class 'ctypes._endian._swapped_meta'> 1
<class 'ctypes.CFunctionType'> 1
<class 'jinja2.utils.MissingType'> 1
<class 'twisted.internet.posixbase._SIGCHLDWaker'> 1
<class 'decimal._Log10Memoize'> 1
<class 'twisted.python.versions._inf'> 1
<class 'xml.dom.minicompat.EmptyNodeList'> 1
<class 'twisted.internet.error.TimeoutError'> 1
<class 'twisted.application.app.HotshotRunner'> 1
<class 'buildbot.scripts.runner.StartOptions'> 1
<class 'twisted.python.reflect.AccessorType'> 1
<class 'buildbot.status.web.auth.AuthFailResource'> 1
<class 'buildbot.db.dbspec.DBSpec'> 1
<class 'jinja2.loaders.FileSystemLoader'> 1
<class 'calendar.TextCalendar'> 1
<class 'jinja2.loaders.PackageLoader'> 1
<class 'xml.dom.minidom.TypeInfo'> 1
<class 'subprocess.Popen'> 1
<class 'pytz.UTC'> 1
<class '_ast.NotEq'> 1
<class '_ast.Is'> 1
<class '_ast.Mod'> 1
<class '_ast.BitXor'> 1
<class '_ast.AugStore'> 1
<class 'ctypes._FuncPtr'> 2
<class 'twisted.conch.ssh.keys.Key'> 2
<class 'ctypes.LibraryLoader'> 2
<class 'twisted.internet.base._ThreePhaseEvent'> 2
<class 'buildbot.util.collections.defaultdict'> 2
<type 'tupleiterator'> 2
<type '_ctypes.PointerType'> 2
<class 'twisted.internet.error.ConnectionFdescWentAway'> 2
<class 'json.scanner.Scanner'> 2
<class 'jinja2.lexer.Failure'> 2
<class 'twisted.internet.error.ConnectionLost'> 2
<type 'generator'> 2
<type 'listiterator'> 2
<class 'Crypto.PublicKey._slowmath._RSAKey'> 2
<class 'site.Quitter'> 2
<class 'random.Random'> 2
<class 'amqplib.client_0_8.transport.TCPTransport'> 3
<class 'carrot.backends.pyamqplib.Connection'> 3
<class 'jinja2.runtime.Context'> 3
<class 'carrot.backends.pyamqplib.Backend'> 3
<class 'decimal.Context'> 3
<class 'twisted.internet.tcp.Port'> 3
<class 'twisted.internet.error.ConnectionDone'> 3
<class 'amqplib.client_0_8.method_framing.MethodReader'> 3
<type 'collections.defaultdict'> 3
<class 'site._Printer'> 3
<class 'amqplib.client_0_8.method_framing.MethodWriter'> 3
<class 'carrot.connection.BrokerConnection'> 3
<class 'jinja2.utils.LRUCache'> 3
<class 'carrot.messaging.Publisher'> 3
<class 'jinja2.nodes.EvalContext'> 3
<class 'buildbot.status.persistent_queue.IndexedQueue'> 4
<class 'mozillapulse.publishers.GenericPublisher'> 4
<class 'twisted.python.deprecate._ModuleProxy'> 4
<class 'uuid.UUID'> 4
<class 'abc.abstractproperty'> 4
<type 'weakproxy'> 4
<class 'MySQLdb.connections.Connection'> 4
<class 'buildbot.status.persistent_queue.DequeMemoryQueue'> 4
<class 'twisted.internet.defer.DeferredLock'> 4
<class 'pytz.tzfile.US/Pacific'> 4
<type 'method-wrapper'> 5
<class 'twisted.python.deprecate._DeprecatedAttribute'> 5
<class 'jinja2.environment.TemplateModule'> 5
<class 'jinja2.ext.ExtensionRegistry'> 6
<class 'decimal.Decimal'> 6
<class 'codecs.CodecInfo'> 6
<class 'MySQLdb.DBAPISet'> 7
<class 'twisted.python.threadable.XLock'> 7
<class 'buildbotcustom.steps.unittest.MozillaPackagedMochitests'> 7
<type 'exceptions.KeyError'> 8
<type '_ctypes.CFuncPtrType'> 8
<class 'threading.Thread'> 11
<class 'jinja2.environment.Template'> 11
<class 'threading._Event'> 12
<class 'twisted.python.usage.CoerceParameter'> 13
<type 'collections.deque'> 13
<class 'twisted.python.versions.Version'> 14
<class 'email.LazyImporter'> 19
<class 'pkg_resources.Distribution'> 19
<type 'classmethod_descriptor'> 20
<class 'abc.ABCMeta'> 21
<type 'frozenset'> 23
<class 'jinja2.runtime.Macro'> 23
<type '_ctypes.SimpleType'> 26
<class 'twisted.conch.insults.insults._const'> 26
<type 'staticmethod'> 27
<class 'zope.interface.interface.Attribute'> 28
<type 'StgDict'> 37
<class 'threading._Condition'> 39
<type 'operator.itemgetter'> 43
<class 'jinja2.nodes.NodeType'> 63
<type 'classmethod'> 65
<type 'traceback'> 72
<class 'twisted.internet.tcp.Server'> 74
<class 'socket._socketobject'> 79
<type 'property'> 139
<class 'zope.interface.interface.InterfaceClass'> 142
<type 'frame'> 178
<class 'zope.interface.declarations.ClassProvides'> 187
<class 'zope.interface.declarations.Implements'> 194
<type 'member_descriptor'> 284
<type 'module'> 566
<class 'zope.interface.interface.Method'> 577
<type 'method_descriptor'> 637
<type 'getset_descriptor'> 840
<type 'classobj'> 939
<type 'wrapper_descriptor'> 1159
<type 'builtin_function_or_method'> 1443
<type 'cell'> 3931
<class 'twisted.internet.task.LoopingCall'> 5140
<class 'weakref.KeyedRef'> 9216
<type 'set'> 19207
<type 'instancemethod'> 21158
<type 'weakref'> 26733
<type 'function'> 64581
<type 'tuple'> 690993
<type 'type'> 1127856
<type 'dict'> 2554443
<type 'list'> 2595704

Comment 13

[Chris AtLee [:catlee]]

I think I'll start putting these in a spreadsheet...

Comment 14

[Chris AtLee [:catlee]]

So the reporting is a bit messed up because the dumping function should look like this: (the difference is t = o.__class__)

import gc
from types import InstanceType
objs_by_type = {}
for o in gc.get_objects():
    t = type(o)
    if t is InstanceType:
        t = o.__class__
    objs_by_type[t] = objs_by_type.get(t, 0) + 1

objs_by_type = objs_by_type.items()
objs_by_type.sort(key = lambda x:x[1])
for t, count in objs_by_type:
    print t, count


Nevertheless, the first set of data (labelled 'broken data' in the linked spreadsheet) indicates a high correlation with things like generator objects, various web status objects, as well as pulse related objects and memory usage.

Comment 15

[Chris AtLee [:catlee]]

OOP handling of pulse and logs doesn't fix the issue:

http://people.mozilla.org/~catlee/sattap/95de0d1a.png

Comment 16

[Chris AtLee [:catlee]]

I'm outta ideas.