703244 - unable to send outbound email from pm-adm-bugs01

Status: VERIFIED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0a2) Gecko/20111114 Firefox/10.0a2
Build ID: 20111114042039

Steps to reproduce:

I have a couple of qa watchers setup for different components. Since a couple of days I now don't get any email anymore whenever a bug gets resolved. This happens when I or anyone else resolves a bug. This is kinda annoying because it causes dataloss for me.


Actual results:

If someone resolves a bug no email gets send out.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

One example is bug 701676, which I have now marked as fixed. I never got an email. After changing the summary in another step, I received this specific email but still not the one for the state change.

Comment 2

[:glob ✱]

i'm still receiving resolved mail.  i have to ask a silly question -- have you checked your spam folder?

i'll chat with IT to see if there's anything interesting in the logs.

Comment 3

[:glob ✱]

ashish is looking into this; it looks like some mail to gmail are stuck in a queue.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Nope. Nothing is in the spam folder. But it looks like you have already identified the cause. I will wait for an update. Thanks.

Comment 5

[Dave Miller [:justdave]]

-- 51046 Kbytes in 16612 Requests.

Mail to everywhere it timing out from pm-adm-bugs01.  Approximately 50% of outbound mail from bugzilla is blocked.  The other 50% is from pp-adm-bugs01, which seems to be working.

Comment 6

[Dave Miller [:justdave]]

This is 10.2.82.200

Should have outbound to port 25 anywhere.

Comment 7

[Dave Miller [:justdave]]

FWIW, this IP is the designated outbound mail relay for vlan82 in sjc1.  All the other boxes in vlan82 use this one as an outbound relay.

Comment 8

[Dave Miller [:justdave]]

17k emails is probably only a few hours worth (maybe a day if it's been slow), Bugzilla sends a lot of mail.  I can't find anything in the diffs in change-control that should have affected it going back a week.

default route on the box exists and is pingable (10.2.82.1)

Comment 9

[Adam Newman [:adam]]

I ran "clear xlate local 10.2.82.200" and now we are passing traffic without an issue.

One of these may be the case:
1.) the NAT thing fixed it for whatever reason. Odd issue with FW1?
2.) there was a reachability issue that resolved itself in coincidence to my action.

Comment 10

[Dave Miller [:justdave]]

remaining queue on pm-adm-bugs01 is under 600 items.  The bulk of what's left is all to the same user on gmail that's getting an "over quota" tempfail, so I think we're good.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Everything is solved for me. I got dozen of missing bug mails. Thank you all.