Closed Bug 1268952 Opened 10 years ago Closed 10 years ago

jobqueue[1-2].bugs.scl3.mozilla.com:Swap is WARNING

Categories

(bugzilla.mozilla.org :: Infrastructure, defect)

Product:
bugzilla.mozilla.org
Component:
Infrastructure
Version:
Production
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: sal, Assigned: fubar)

References

Details

@nagios-scl3> Fri 11:06:38 PDT [5232] bugzilla-rw-vip.db.scl3.mozilla.com:Schwartz Queue is WARNING: WARNING: Bugzilla::Job::BugMail: ct=88 max=584, Bugzilla::Job::Mailer: ct=1 max=584. (http://m.mozilla.org/Schwartz+Queue) @nagios-scl3> Fri 11:07:39 PDT [5234] bugzilla-rw-vip.db.scl3.mozilla.com:Schwartz Queue is CRITICAL: CRITICAL: Bugzilla::Job::BugMail: ct=1 max=644, Bugzilla::Job::Mailer: ct=1 max=644. (http://m.mozilla.org/Schwartz+Queue) @nagios-scl3> Fri 11:27:23 PDT [5247] jobqueue1.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 40% free (815 MB out of 2047 MB) (http://m.mozilla.org/Swap) @nagios-scl3> Fri 11:27:33 PDT [5248] jobqueue2.bugs.scl3.mozilla.com:Swap is CRITICAL: SWAP CRITICAL - 24% free (484 MB out of 2047 MB) (http://m.mozilla.org/Swap)
Well, Bug 627699 is very popular. It had five changes made to it so far today, and each change emails 207 people. 207 * 5 = 1035 emails for that bug alone. Note these are "hypothetical" emails, as not *all* 207 people would receive email on every update, it would depend on their user preferences. This also excludes component watchers and user watchers. The next in the list (for today only) is bug 869543. That bug only has 39 cc'd people, 195 hypothetical emails (once again, 5 events). Basically 627699 is pathological.
since the ct= was low on the schwartz queue and max= was high, *plus* oom_killer struck, I've restarted all of the runners on jobqueue[12]. swap is happy again, waiting to see if the schwartz clears.
Assignee: nobody → klibby
user had corrupted gpg key. :dkl removed the queue entries and will reach out to the user. have restarted daemons again.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
user tried to turn secure mail back on, with a corrupted gpg key. I've saved a copy of the key off for testing and I'm going to file a bug to resolve that. I cleared that user's entries from the jobqueue and restarted daemons. Hopefully it goes fine over the weekend.
It alerted again tonight: Fri 19:14:21 PST [5443] jobqueue2.bugs.scl3.mozilla.com:Swap is WARNING: SWAP WARNING - 44% free (881 MB out of 2047 MB) But cleared: Fri 19:35:28 PST [5464] jobqueue2.bugs.scl3.mozilla.com:Swap is OK: SWAP OK - 99% free (2015 MB out of 2047 MB)
You need to log in before you can comment on or make changes to this bug.