catlee: mozilla-beta got updated catlee: so now every scheduler is iterating over every change
Something (the refresh of mozilla-beta?) created on the order of 60,000 changes. Now every buildbot scheduler has to examine each of the 60,000 changes, which takes time, and also thrashes the cache.
FTR, most schedulers, including mozilla-central and try, are stuck on changeid 469003: ea8b7e3f457baed08cdf980b98705beabfd7ff56. changeid 536797 is where e7cf36e12643 is introduced. I'm bumping up the change cache size on the main scheduler master.
via manhole: status.db._change_cache._max_size = 100000 since reconfig with c['changeCacheSize'] wasn't working
change cache maxed out at around 68k: >>> len(status.db._change_cache._cached_ids) 68028
We think we're out of the woods here, as all the schedulers have now processed the large chunk of changes out of mozilla-beta and the associated locales. And on new changes 2000 schedulers process pretty quickly. I'm assuming the other 1000 or so are now deprecated. Have to duck out now, but I'll drop the change cache size back down in a couple of hours.
I've done the same as comment #4 only setting it back to 10000, for the schedulers pm01-sm and pm02-sm. When new changes arrive they're being processed by 100-200 schedulers/second.