jobs not getting scheduled

RESOLVED FIXED

Status

Release Engineering
General
P1
normal
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: jrmuizel, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
catlee: mozilla-beta got updated
catlee: so now every scheduler is iterating over every change
(Assignee)

Comment 1

7 years ago
Something (the refresh of mozilla-beta?) created on the order of 60,000 changes. Now every buildbot scheduler has to examine each of the 60,000 changes, which takes time, and also thrashes the cache.
Assignee: nobody → catlee
Priority: -- → P1
Summary: Jobs not becoming pending when they should → jobs not getting scheduled
(Assignee)

Comment 2

7 years ago
FTR, most schedulers, including mozilla-central and try, are stuck on changeid 469003: ea8b7e3f457baed08cdf980b98705beabfd7ff56.

changeid 536797 is where e7cf36e12643 is introduced.

I'm bumping up the change cache size on the main scheduler master.
Bug 652858 is where the "cloning" of mozilla-beta happened.
Depends on: 652858
OS: Mac OS X → All
(Assignee)

Comment 4

7 years ago
via manhole:

status.db._change_cache._max_size = 100000

since reconfig with c['changeCacheSize'] wasn't working
(Assignee)

Comment 5

7 years ago
change cache maxed out at around 68k:
>>> len(status.db._change_cache._cached_ids)
68028
We think we're out of the woods here, as all the schedulers have now processed the large chunk of changes out of mozilla-beta and the associated locales. And on new changes 2000 schedulers process pretty quickly. I'm assuming the other 1000 or so are now deprecated.

Have to duck out now, but I'll drop the change cache size back down in a couple of hours.
I've done the same as comment #4 only setting it back to 10000, for the schedulers pm01-sm and pm02-sm.

When new changes arrive they're being processed by 100-200 schedulers/second.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.