Closed Bug 756259 Opened 14 years ago Closed 14 years ago

mx1.mail.corp.phx1.m.c is modifying message subjects based on spam scores

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

x86
macOS
task
Not set
critical

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Atoll, Assigned: limed)

Details

Attachments

(2 files)

Please consider adjusting the internal-only email server to not mangle messages it thinks are spammy - if there's any reason for it to be running spam checks at all, given that Postini handles that for external mail anyways. Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 17 May 2012 11:21:01 -0700 (PDT) Received: from mx1.mail.corp.phx1.mozilla.com (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id C28A5106604 for <rsoderberg@mozilla.com>; Thu, 17 May 2012 11:21:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: YES X-Spam-Score: 7.446 X-Spam-Level: ******* X-Spam-Status: Yes, score=7.446 required=6.31 tests=[HTML_MESSAGE=0.001, HTML_TAG_BALANCE_BODY=1.247, MIME_HTML_ONLY=2.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Received: from confluence1.webapp.phx1.mozilla.com (unknown [63.245.216.223]) by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 3BD10F2704 for <rsoderberg@mozilla.com>; Thu, 17 May 2012 11:21:01 -0700 (PDT) Date: Thu, 17 May 2012 11:21:01 -0700 (PDT) From: "Aaron Hill (Confluence)" <mana@mozilla.org> To: rsoderberg@mozilla.com Message-ID: <888859992.1278.1337278861237.JavaMail.confluence@confluence1.webapp.phx1.mozilla.com> subject: ***SPAM*** [mana] Netops > VLAN assignments
It's submitted mail on smtp.mozilla.org that's getting scanned (precisely because it didn't come in via postini). Correct fix here is to whitelist mana as a sender. We copied over the amavis config files from the old mail servers, it's possible we never added phx's IP range?
Assignee: server-ops → justdave
Component: Server Operations → Server Operations: Infrastructure
QA Contact: phong → jdow
Just to confirm, the IP range whitelists in amavis on the relays were WAY out of date. The ones in postfix did get updated. We do *not* have these files in puppet yet (they will be soon). I need to split for a few hours to take care of some kid stuff. If someone else wants to grab this before I get back, the affected files are /etc/amavisd.conf on mx1 and mx2.mail.corp.phx1. It goes in @mynetworks on line 45. The correct IP whitelist is available by running "postconf mynetworks" on either server.
"service amavisd restart" after editing.
Group: infra
OK, this should be set now
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
I've seen several emails like this since you closed the bug. Interestingly, none before that. (We get a lot of these update emails from this host while doing a release.) Not all messages from that host are so marked - I'll attach another one from the same time that did not get marked.
See attachments for spam/not spam on nearly identical emails from an internal server
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Nearly identical email within a few minutes of the other that was (correctly) not marked as spam
The one that got marked as spam was processed by mx2, the one that wasn't was processed by mx1. I *thought* I restarted amavis after copying the config over, but maybe I didn't. I'll go kick it again.
I've just got this in the buildbot logs on dev-master01: Failure: twisted.mail.smtp.SMTPDeliveryError: 451 4.6.0 Content scanner malfunction >>> DATA <<< 354 End data with <CR><LF>.<CR><LF> >>> . <<< 451 4.6.0 Content scanner malfunction $ grep ^DS /etc/mail/sendmail.cf DSmail.build.mozilla.org
You hit it in the 20 seconds between when we pushed a broken config change and it recovering with the fixed config. 4xx errors are non-fatal, just means "try again". Your best guard against that is to set your local sendmail or postfix to use smtp.mozilla.org for a relayhost and submit to 127.0.0.1 from your scripts, then it'll queue and try again if it gets errors without breaking your script.
This hit us again yesterday, with emails from buildbot about yesterday's FF13.0b6 release being flagged as spam.
I sent one example to cshields earlier today. Here's another example which hit our TB13.0b4 release in progress tonight. -------- Original Message -------- Subject: Fwd: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for releasetest testing Date: Wed, 30 May 2012 00:43:23 -0400 From: Rail Aliiev <rail@mozilla.com> To: thunderbird-drivers@mozilla.org I'm not sure if this passed to the mailing list. This means that QA can skip testing of updates on the betatest channel and use the releasetest channel instead. -------- Original Message -------- Subject: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for releasetest testing Date: Tue, 29 May 2012 21:32:32 -0700 From: release@mozilla.com To: Callek@..., akeybl@..., mbanner@..., release@mozilla.com Mirrors are absorbed enough for releasetest testing of Thunderbird 13.0b4 build1. -buildbot _______________________________________________ Thunderbird-drivers mailing list Thunderbird-drivers@mozilla.org https://mail.mozilla.org/listinfo/thunderbird-drivers
I need the full headers for it to be useful to me at all. (in particular all the X-Spam and Received stuff)
Return-Path: release@mozilla.com Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by zmmbox3.mail.corp.phx1.mozilla.com with LMTP; Tue, 29 May 2012 12:13:49 -0700 (PDT) Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 8918D1068F4; Tue, 29 May 2012 12:13:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: YES X-Spam-Score: 7.198 X-Spam-Level: ******* X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Received: from buildbot-master13.build.scl1.mozilla.com (unknown [63.245.222.66]) by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id 4AF42F22ED; Tue, 29 May 2012 12:13:49 -0700 (PDT) Content-Transfer-Encoding: base64 MIME-Version: 1.0 Content-Type: text/plain; charset="utf8" Date: Tue, 29 May 2012 12:13:45 -0700 subject: ***SPAM*** [release] Firefox 13.0b6 build1: ready for release From: release@mozilla.com To: Callek@gmail.com, akeybl@mozilla.com, release@mozilla.com Message-Id: <20120529191349.8918D1068F4@zmmta1.mail.corp.phx1.mozilla.com> TWlycm9ycyBhcmUgYWJzb3JiZWQgZW5vdWdoIGZvciByZWxlYXNlIG9mIEZpcmVmb3ggMTMuMGI2 IGJ1aWxkMS4KCi1idWlsZGJvdAoKCg==
Return-Path: release@mozilla.com Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by zmmbox3.mail.corp.phx1.mozilla.com with LMTP; Tue, 29 May 2012 21:32:34 -0700 (PDT) Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 1681C106425; Tue, 29 May 2012 21:32:34 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: YES X-Spam-Score: 7.198 X-Spam-Level: ******* X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Received: from buildbot-master34.srv.releng.scl3.mozilla.com (v-1030.fw1.releng.scl3.mozilla.net [63.245.214.82]) by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id C9804F240B; Tue, 29 May 2012 21:32:33 -0700 (PDT) Content-Transfer-Encoding: base64 MIME-Version: 1.0 Content-Type: text/plain; charset="utf8" Date: Tue, 29 May 2012 21:32:32 -0700 subject: ***SPAM*** [release] Thunderbird 13.0b4 build1: ready for releasetest testing From: release@mozilla.com To: Callek@gmail.com, akeybl@mozilla.com, mbanner@mozilla.com, release@mozilla.com Message-Id: <20120530043234.1681C106425@zmmta1.mail.corp.phx1.mozilla.com> TWlycm9ycyBhcmUgYWJzb3JiZWQgZW5vdWdoIGZvciByZWxlYXNldGVzdCB0ZXN0aW5nIG9mIFRo dW5kZXJiaXJkIDEzLjBiNCBidWlsZDEuCgotYnVpbGRib3QKCg==
thanks. short-term I'm going to throw these at the spam/ham trainer. Long term I'm seeing a bunch of settings to amavis that'll help, but they're potentially invasive, so I need to get a test server set up to play with first before I touch it on production.
(In reply to Dave Miller [:justdave] from comment #16) > thanks. short-term I'm going to throw these at the spam/ham trainer. Long > term I'm seeing a bunch of settings to amavis that'll help, (In reply to Dave Miller [:justdave] from comment #8) > The one that got marked as spam was processed by mx2, the one that wasn't > was processed by mx1. I *thought* I restarted amavis after copying the > config over, but maybe I didn't. I'll go kick it again. We're still seeing different behavior based on whether it's received by mx1 or xm2 first. Silly question - did the mx2 restart really "take" for these settings?
(In reply to Hal Wine [:hwine] from comment #17) > We're still seeing different behavior based on whether it's received by mx1 > or xm2 first. Silly question - did the mx2 restart really "take" for these > settings? Good question. The configs are auto-synced from mx1, and it's supposed to automatically restart when it picks up config changes. I just manually kicked mx2 just to make sure.
Alas, this is still occurring: Return-Path: release@mozilla.com Received: from zmmta2.mail.corp.phx1.mozilla.com (LHLO zmmta2.mail.corp.phx1.mozilla.com) (10.20.77.30) by zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 31 May 2012 15:43:47 -0700 (PDT) Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id A299F1020D8 for <release@mozilla.com>; Thu, 31 May 2012 15:43:47 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: YES X-Spam-Score: 7.199 X-Spam-Level: ******* X-Spam-Status: Yes, score=7.199 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_EQ_FM_DIRECT_MX=0.001, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Received: from buildbot-master13.build.scl1.mozilla.com (unknown [63.245.222.66]) by mx2.mail.corp.phx1.mozilla.com (Postfix) with SMTP id 46B5DF249F for <release@mozilla.com>; Thu, 31 May 2012 15:43:47 -0700 (PDT) Content-Transfer-Encoding: base64 MIME-Version: 1.0 Content-Type: text/plain; charset="utf8" Date: Thu, 31 May 2012 15:43:47 -0700 subject: ***SPAM*** [release] Tagging started for Firefox 13.0b7 build1 From: release@mozilla.com To: release@mozilla.com Message-Id: <20120531224347.A299F1020D8@zmmta2.mail.corp.phx1.mozilla.com> VGFnZ2luZyBzdGFydGVkIGZvciBGaXJlZm94IDEzLjBiNyBidWlsZDEKCi1idWlsZGJvdAoK ----- Also happens on mx1.
This is affecting release communication during release crunch time.
Severity: normal → major
Can we consider using a bigger hammer on this issue? Are there any significant inbound email flows that can bypass postini? If not, can we disable this level of spam checking? If it is needed, can we add all machines on the buildvpn to the whitelist? (I'd hope that could be done by subnet)
This is bypassing postini, it's our internal mail relay that's doing it. The "bigger hammer" is what I alluded to in comment 16 (and includes causing anti-spam being configured to not run on internal mail), but "experimenting" with said features on a production mail server could result in outright *loss* of mail rather than just labeling it. I need at least the weekend to experiment on a VM clone of the mail server to make sure everything is correct before I drop it in production. Or I can completely turn off the spam protection but then you'll be yelling at me about all the spam getting through.
This tripped us up today with the TB13.0 release. As a short term fix, can we explicitly whitelist certain senders, like "release@m.c" originating in build network, which was in each of the examples so far in this in bug? -------- Original Message -------- Subject: ***SPAM*** [release] Thunderbird 13.0 build1: ready for releasetest testing Date: Tue, 05 Jun 2012 10:54:19 -0700 From: release@mozilla.com To: Mirrors are absorbed enough for releasetest testing of Thunderbird 13.0 build1. -buildbot
I realize that Dave is on PTO right now but this is really an unacceptable state for us to be in. Release mail is absolutely crucial for hand-offs between RelEng, QA, and Release Management. Since this started happening we have repeatedly had multiple hours of delay during releases because of lost mail. Can we please get this fixed one way or another soon?
Severity: major → critical
I am not on PTO right now. But you'll notice there were dependencies outside of my jurisdiction on this bug which were just resolved within the last hour. This should be resolved within the next day.
(In reply to Dave Miller [:justdave] from comment #25) > there were dependencies outside of my jurisdiction on this bug which were just > resolved within the last hour. er, within the hour previous to your comment actually. (I suspect said bugmail about that resolution is what reminded you to comment here)
(In reply to Dave Miller [:justdave] from comment #26) > (In reply to Dave Miller [:justdave] from comment #25) > > there were dependencies outside of my jurisdiction on this bug which were just > > resolved within the last hour. > > er, within the hour previous to your comment actually. (I suspect said > bugmail about that resolution is what reminded you to comment here) Thunderbird says I didn't get bugmail about that resolution change :-( However it is nice if the issue is fixed :-)
bug 761869 is restricted, so we can't see what you were depending on
(In reply to Chris AtLee [:catlee] from comment #28) > bug 761869 is restricted, so we can't see what you were depending on Ah, my apologies. Was waiting on a machine to test configuration changes on without having to make them live in production and hope it wouldn't cause all mail to bounce. I have my test machine as of today.
There were several emails in the junk folder for the releases we have triggered since yesterday. I have dropped the domain from each email (not sure if necessary but meh) Here is one example: -------- Original Message -------- Subject: ***SPAM*** [release] Fennec 14.0b7 build2: completed fennec_source Date: Wed, 13 Jun 2012 13:52:19 -0700 From: release To: Callek, akeybl, release fennec_source complete for Fennec 14.0b7 build2 -buildbot
(In reply to Dave Miller [:justdave] from comment #25) > I am not on PTO right now. But you'll notice there were dependencies > outside of my jurisdiction on this bug which were just resolved within the > last hour. We didnt have access to the dep bug, but gtk you are now unblocked. > This should be resolved within the next day. Any news? We have 3 different high-urgency releases in flight, and these emails are how we coordinate across groups. When they disappear into spam, it's quite disruptive.
also, when you rollout fixes, will you need a downtime for this?
I'm running a little behind because I've been firefighting stuff all day. hopefully I'll have news yet tonight. It will be fairly instantaneous once I roll it out, just need to make sure on the staging server that it won't break things before I toss it at production.
this just from one of today's releases (with full headers this time!): -------- Original Message -------- Return-Path: release@mozilla.com Received: from zmmta1.mail.corp.phx1.mozilla.com (LHLO zmmta1.mail.corp.phx1.mozilla.com) (10.20.77.21) by zmmbox2.mail.corp.phx1.mozilla.com with LMTP; Thu, 14 Jun 2012 17:27:56 -0700 (PDT) Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta1.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 0D4AD10678E; Thu, 14 Jun 2012 17:27:56 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: YES X-Spam-Score: 7.198 X-Spam-Level: ******* X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Received: from buildbot-master12.build.scl1.mozilla.com (unknown [63.245.222.66]) by mx2.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id C784AF2707; Thu, 14 Jun 2012 17:27:55 -0700 (PDT) Content-Transfer-Encoding: base64 MIME-Version: 1.0 Content-Type: text/plain; charset="utf8" Date: Thu, 14 Jun 2012 17:27:45 -0700 subject: ***SPAM*** [release] Firefox 13.0.1 build1: completed repack_1/6 on win32 From: release@mozilla.com To: Callek@gmail.com, akeybl@mozilla.com, release@mozilla.com Message-Id: <20120615002756.0D4AD10678E@zmmta1.mail.corp.phx1.mozilla.com> repack_1/6 complete for Firefox 13.0.1 build1 on win32 -buildbot
X-Spam-Status: Yes, score=7.198 required=6.31 tests=[BAYES_50=2, MISSING_MID=1.199, TO_NO_BRKTS_DIRECT=3.999] autolearn=no Missing Message-ID is 2 points. That'll be enough to get you under 6.31 which will stop it from getting tagged as spam. Have your script add a Message-Id header. The one you pasted was added by the MTA after it got scanned. TO_NO_BRKTS_DIRECT is 4 points. You can avoid this one by putting names and brackets on the email addresses on the To line. This might be easier to fix than trying to come up with a unique message ID every time. To: Justin Wood <Callek@gmail.com>, Alex Keybl <akeybl@mozilla.com>, Release Team <release@mozilla.com> I'm still having fun issues with the bigger config change I was hoping to make getting it to not do spam scanning on internally-originating mail, but I just discovered a way to do spam score modifications by sender address, and release@mozilla.com and bootstrap@mozilla.com both get a -8 score boost in spamassassin now.
I have a script I run which emails from release-mgmt@mozilla.com to a combination of internal & external email addresses. The bulk of them end up in the Junk folder with this ** SPAM ** marking. Can internal email addresses be exempt from this spam evaluation? Is there a way to get away from being marked as spam without adding a Message-Id header?
I've added release-mgmt to the -8 score boost.
Just to validate the suggestion in comment #35 - adding the brackets does drop scores enough, in case the -8 score boost is spoofable. Return-Path: release@mozilla.com Received: from zmmta2.mail.corp.phx1.mozilla.com (LHLO zmmta2.mail.corp.phx1.mozilla.com) (10.20.77.30) by zmmbox4.mail.corp.phx1.mozilla.com with LMTP; Fri, 15 Jun 2012 12:45:03 -0700 (PDT) Received: from smtp.mozilla.org (zlb1.mail.corp.phx1.mozilla.com [10.20.77.200]) by zmmta2.mail.corp.phx1.mozilla.com (Postfix) with ESMTP id 1E38B1020C3 for <hwine@mozilla.com>; Fri, 15 Jun 2012 12:45:03 -0700 (PDT) X-Virus-Scanned: amavisd-new at mozilla.org X-Spam-Flag: NO X-Spam-Score: -8.301 X-Spam-Level: X-Spam-Status: No, score=-8.301 required=6.31 tests=[AM:BOOST=-8, BAYES_00=-1.5, MISSING_MID=1.199] autolearn=no Received: from dev-master01.build.scl1.mozilla.com (unknown [63.245.222.66]) by mx1.mail.corp.phx1.mozilla.com (Postfix) with ESMTPS id DF5FBF256D for <hwine@mozilla.com>; Fri, 15 Jun 2012 12:45:02 -0700 (PDT) Content-Transfer-Encoding: base64 MIME-Version: 1.0 Content-Type: text/plain; charset="utf8" Date: Fri, 15 Jun 2012 12:45:02 -0700 Subject: [staging-release] Firefox 12.0b3 build1: All win32 builds now available From: release@mozilla.com To: <hwine@mozilla.com> Message-Id: <20120615194503.1E38B1020C3@zmmta2.mail.corp.phx1.mozilla.com> QWxsIHdpbjMyIGJ1aWxkcyBub3cgYXZhaWxhYmxlCgpmdHA6Ly9kZXYtc3RhZ2UwMS5zcnYucmVs ZW5nLnNjbDMubW96aWxsYS5jb20vcHViL21vemlsbGEub3JnL2ZpcmVmb3gvbmlnaHRseS8xMi4w YjMtY2FuZGlkYXRlcy9idWlsZDEvd2luMzIKCi1idWlsZGJvdAoK
Anything left to do here?
Yes, the amavis config still needs to be redone so that internally-originating mail is treated differently as a category of mail rather than just screwing with spamassassin scores for frequent internal sender addresses.
Dave, I am sure this isn't the only thing on your plate, but I wanted to check in on this bug and get an update, if you have one.
Assignee: justdave → limed
We are planning on using policy banks[1] in amavis so that we can bypass spam checks if the email is originating from the local network. I just tested this out on mx3.mail.corp.phx1.mozilla.com and it looks like its working. I'm just going to have :justdave take a look at it one more time and then we'll flip the bit on mx1 and mx2 [1]http://www.ijs.si/software/amavisd/amavisd-new-docs.html#pbanks
Ok the change to amavis has been made to mx1 and mx2, the logs are showing that mail is getting delivered normally and mail from our local network is not getting tagged. I think we are good here
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → FIXED
Component: Server Operations: Infrastructure → Infrastructure: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: