Closed Bug 1132469 Opened 9 years ago Closed 9 years ago

Trees closed - Jobs not starting

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
All
Type:
task
Priority:
Not set
Severity:
blocker

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: cbook, Unassigned)

Details

seems b2g-i and fx-team suffer from a problem that prevent starting jobs like builds. Also buildapi shows nothing for the last 2 pushes on fx-team as example
and seems to spread with tests and build backlog. Integration Trees and Gaia are closed
A couple of confounding factors here this morning:

1) a buildbot db failover happened. This often causes masters and slaves to enter a state where they are connected to each other, but unable to take new work.
2) ongoing rolling downtimes to re-install buildbot masters due to GHOST. This has reduced capacity across the board.

A restart of the master (and occasionally a reboot of the attached slaves) will fix #1. Given that we need to restart masters anyway for #2, we will try to combine any new downtimes for #2 as part of the restart process for #1.

I've already started rebooting build masters, and jlund is looking at try masters. :dustin is currently restarting/reinstalling AWS test masters as part of #2, and jlund and I will look at the remaining once the build/try masters are back in service.
Further db-related issues are being tracked in bug 1132792.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.