1175726 - Test jobs not being scheduled...

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

... or something.

Comment 1

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

When I look at https://secure.pub.build.mozilla.org/buildapi/pending I see there's 33,549 entries in the pending jobs list.

When I look at http://builddata.pub.build.mozilla.org/reports/pending/pending.html I see there's 19431 pending ubuntu64-vm test jobs on fx-team, and 13939 pending winxp-ix test jobs on fx-team.

But I don't actually see any of these pending jobs in treeherder on the revisions that buildapi says has a bunch of pending revisions.


All trees closed.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

The pending count has gone up to 44k.

We think this is from http://hg.mozilla.org/build/buildbot-configs/rev/2de50c1598b4. There is a set of schedulers which have -<timeout> suffix. By changing the timeout value, we think an old set of schedulers was re-enabled, and the last code change they saw was on 2015-04-23. There are now processing every code push (change) since then, eg this from the tests scheduler master:

2015-06-17 14:32:23-0700 [-] tests-fx-team-xp-ix-debug-unittest-5-1800: triggering since we have 897/5 important changes

We're stopping the test scheduler master, and will cleanup the scheduler table in the db.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The schedulers are cleaned up, we're squashing the pending now.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Test scheduler is back up to accept sendchanges from builds job that are finishing. Still working on squashing the pending.

Comment 5

[Chris AtLee [:catlee]]

Still working on cleaning up the old pending jobs.

Comment 6

[Chris AtLee [:catlee]]

25k jobs left to clean up - ETA 15 minutes

Comment 7

[Chris AtLee [:catlee]]

most of the backlog has been cleared - looking to see what the fallout is like

Comment 8

[Chris AtLee [:catlee]]

releng - please file a bug to follow up on this so these old schedulers get deleted by the maintenance script.

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Reopened the trees. We'll see how the backlog goes.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris AtLee [:catlee] from comment #8)
> releng - please file a bug to follow up on this so these old schedulers get
> deleted by the maintenance script.

bug 1176132 for this.

Comment 11

[Kim Moir [:kmoir] ET]

Thanks to all of you who worked on fixed this yesterday. I didn't realize when I made a simple change to reduce the frequency of the SETA skipping that it would cause such scheduling issues. Lesson learned for the future :-)