983125 - self-serve agents eat exceptions, causing stalls

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The errors get logged, but then eaten, and the ReliableConsumer doesn't know that an error occurred.

Comment 1

[Carsten Book [:Tomcat]]

affects like retriggers don;t work at the moment

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug983125.patch

This also removes the system exit in response to a DB timeout - that doesn't seem like particularly good problem-solving!

Comment 3

[[e-mail] (:bhearsum)]

For posterity, these can be worked around by restarting the self serve agents: https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_BuildAPI

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8390467 [details] [diff] [review]
bug983125.patch

I bumped the version to 0.2.3 and put it on the puppet masters.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug983125-p1.patch

I should have posted this with the other patch - sorry to interrupt you twice :/

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8390699 [details] [diff] [review]
bug983125-p1.patch

Backed out:
https://hg.mozilla.org/build/puppet/rev/43dc6c60f0cd

No legit messages were being processed after we raised on the old messages from bug 980086.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I'll ship 0.2.4 for bug 980086 and then re-deploy here.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug983125-p2.patch

third patch is the charm..

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8392203 [details] [diff] [review]
bug983125-p2.patch

Checked in on 3/17/2014