Ignore "perspective_shutdown" and RuntimeError exceptions in twistd.log monitoring emails from masters

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: coop, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildmasters][cleanup][monitoring])

Attachments

(1 attachment)

(Assignee)

Description

7 years ago
We know the cause of both of these errors:

* perspective_shutdown won't go away until we update the buildbot masters to a more recent version of buildbot (>= 0.8.4). This won't be happening any time soon AFAICT.

* RuntimeError happens when users cancel their own jobs via self-serve. Not much we can do here.

To keep the exceptions emails useful, let's whitelist these exceptions. Patch coming up.
(Assignee)

Updated

7 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 1

7 years ago
Created attachment 559035 [details] [diff] [review]
Ignore exceptions we can do nothing about (perspective_shutdown and RuntimeError)

Tested successfully against some twistd.log files I downloaded from various masters.
Attachment #559035 - Flags: review?(catlee)

Updated

7 years ago
Attachment #559035 - Flags: review?(catlee) → review+
(Assignee)

Comment 2

7 years ago
Comment on attachment 559035 [details] [diff] [review]
Ignore exceptions we can do nothing about (perspective_shutdown and RuntimeError)

https://hg.mozilla.org/build/tools/rev/d0a86616776e
Attachment #559035 - Flags: checked-in+
(Assignee)

Comment 3

7 years ago
I've updated the tools checkout on all the masters. Will monitor exception emails this afternoon to make sure this is fixed.
(Assignee)

Comment 4

7 years ago
Didn't get a flood of exception email at 11:00am PDT, but still got mail from preproduction-master for valid exceptions. Looks fixed.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.