Closed Bug 1218372 Opened 9 years ago Closed 7 years ago

monitor signing workers queue

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rail, Unassigned)

References

Details

Today all signing workers were stuck, not receiving any tasks from pulse. TC marked them as deadline-exceeded. After restarting the serves, signing workers started picking tasks without any issues.

We can probably monitor them for idleness and restart if they don't take tasks for too long.
I had to poke them again today. :(
Severity: normal → major
One of the theories is that the new rabbit server doesn't support heartbeats.

http://hg.mozilla.org/build/buildapi/file/f5b3577bf236/buildapi/lib/mq.py#l40 does some magic to explicitly reconnect.
Rail, I assume this is because of bug 1218976.
I believe this is fixed now.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.