unable to send outbound email from pm-adm-bugs01

VERIFIED FIXED

Status

Infrastructure & Operations
NetOps
--
blocker
VERIFIED FIXED
6 years ago
4 years ago

People

(Reporter: whimboo, Assigned: adam)

Tracking

({dataloss})

Details

(Reporter)

Description

6 years ago
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111114 Firefox/10.0a2
Build ID: 20111114042039

Steps to reproduce:

I have a couple of qa watchers setup for different components. Since a couple of days I now don't get any email anymore whenever a bug gets resolved. This happens when I or anyone else resolves a bug. This is kinda annoying because it causes dataloss for me.


Actual results:

If someone resolves a bug no email gets send out.
(Reporter)

Updated

6 years ago
Severity: normal → major
Keywords: dataloss
(Reporter)

Comment 1

6 years ago
One example is bug 701676, which I have now marked as fixed. I never got an email. After changing the summary in another step, I received this specific email but still not the one for the state change.
i'm still receiving resolved mail.  i have to ask a silly question -- have you checked your spam folder?

i'll chat with IT to see if there's anything interesting in the logs.
ashish is looking into this; it looks like some mail to gmail are stuck in a queue.
Assignee: nobody → ashish
(Reporter)

Comment 4

6 years ago
Nope. Nothing is in the spam folder. But it looks like you have already identified the cause. I will wait for an update. Thanks.
-- 51046 Kbytes in 16612 Requests.

Mail to everywhere it timing out from pm-adm-bugs01.  Approximately 50% of outbound mail from bugzilla is blocked.  The other 50% is from pp-adm-bugs01, which seems to be working.
Assignee: ashish → network-operations
Severity: major → blocker
Component: General → Server Operations: Netops
Product: bugzilla.mozilla.org → mozilla.org
QA Contact: general → mrz
Summary: If bug state changes to RESOLVED no email gets send → unable to send outbound email from pm-adm-bugs01
Version: Current → other
This is 10.2.82.200

Should have outbound to port 25 anywhere.
FWIW, this IP is the designated outbound mail relay for vlan82 in sjc1.  All the other boxes in vlan82 use this one as an outbound relay.
17k emails is probably only a few hours worth (maybe a day if it's been slow), Bugzilla sends a lot of mail.  I can't find anything in the diffs in change-control that should have affected it going back a week.

default route on the box exists and is pingable (10.2.82.1)
Assignee: network-operations → adam
(Assignee)

Comment 9

6 years ago
I ran "clear xlate local 10.2.82.200" and now we are passing traffic without an issue.

One of these may be the case:
1.) the NAT thing fixed it for whatever reason. Odd issue with FW1?
2.) there was a reachability issue that resolved itself in coincidence to my action.
remaining queue on pm-adm-bugs01 is under 600 items.  The bulk of what's left is all to the same user on gmail that's getting an "over quota" tempfail, so I think we're good.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 11

6 years ago
Everything is solved for me. I got dozen of missing bug mails. Thank you all.
Status: RESOLVED → VERIFIED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.