For the past two weeks mozilla.support.firefox, mozilla.support.thunderbird, and mozilla.support.seamonkey have been getting bunches of spam on a daily basis. It's clear that this isn't going to stop. All of it is coming via Google Groups; and the long Giganews reaction time with regards to removing the spam, makes that option useless.
mozilla.support.seamonkey should be included in this move to read-only, I think even adding mozilla.dev.apps.seamonkey would be good, given that it got quite an amount of spam as well recently.
So here's what I'm attempting currently... --- /var/www/html/newsgroups.txt 2008-03-26 22:01:17.000000000 -0700 +++ /root/.listconfig/newsgroups.txt 2008-03-26 22:47:48.000000000 -0700 @@ -17 +17 @@ -mozilla.dev.apps.seamonkey +mozilla.dev.apps.seamonkey moderated @@ -99 +99 @@ -mozilla.support.firefox +mozilla.support.firefox moderated @@ -103,2 +103,2 @@ -mozilla.support.seamonkey -mozilla.support.thunderbird +mozilla.support.seamonkey moderated +mozilla.support.thunderbird moderated This configuration is not in mailman, it's hardcoded into the script that generates the newsgroups.txt file. Assuming Google's robot correctly picks this up, it should start mailing any attempts to post via Google into the list posting address instead of directly posting them. That'll let us moderate Google posts via the mailman interface. The downside of this is the current list configurations on the support lists means anyone who isn't actually subscribed to some list at mozilla via mailing list is going to end up getting a reject notice outright.
Google usually scrapes that file at midnight PDT every night. I'd give it a couple hours after they scrape before expecting it to show up that way on their site, assuming they actually deal with it correctly. If they pick it up correctly it should be working in 3 or 4 hours.
I can turn off auto-rejection, and see what the signal to noise ratio is like. Looking at the date of your comment, I assume this Google moderation is already in place, correct? Thanks a lot Dave. If we ever meet in person, I'm buying you a steak dinner.
(In reply to comment #4) > Looking at the date of your comment, I assume this Google moderation is already > in place, correct? It *should* be. I don't see any way to tell without trying to post and see if it ends up with mailman headers on it and news.mozilla.org X-Trace info.
I posted a message to mozilla.support.firefox via Google Groups. <news://news.mozilla.org/mailman.678.1206640076.6508.support-firefox@lists.mozilla.org> I did have to approve it via the mailman interface, and it arrived at nmo. However, the Google Group doesn't seem to contain any posts, since the change was applied (ie. not archiving). <http://groups.google.com/group/mozilla.support.firefox>
ah crap, you know why... Mailman doesn't know it's moderated, so it's not adding the headers to tell Google they're okay. Pah. So I just enabled the "open list, moderated newsgroup" thing in mailman, which should mean the list controls apply... but does that mean the mailing list will ignore stuff posted to giganews?
hmm, but stuff posted via Giganews wouldn't go to Google either without said headers. :( what a mess :(
I subscribed to the mailing list, and posted <news://news.mozilla.org/F5idnYvs4LKNZHbanZ2dnUVZ_gudnZ2d@mozilla.org> via nmo. It arrived in my mailbox, but still hasn't shown up on Google.
yeah, exactly.
I've opened a ticket with Giganews specific to the pseudo-moderation issue, to see if there's any kind of magic they can pull off on their end, or if they've got any more bright ideas we can try. :)
I've switched auto-reject back on, for support-firefox and support-thunderbird. If a person posts through Google, and someone on nmo replies, the OP will not see the reply on Google. So it's basically the same reason why I turned off non-member posting in the first place.
ok, Giganews is thinking the safest option is probably going to be to lock everything down to being moderated on their end as well and funnel everything through our mailman. The down side of this is we're going to have to turn off the auto-reject and have people actually moderate the lists actively (and add common posters to the auto-accept lists) or people without mailman accounts are gonna get upset quick.
Hmm... I don't know. I would rather go back to the original setup of Google being read-only. Obviously there's resistance to asking anything from Google. Nir, Q, Robert, what do you think? The auto-approve list is going to be huge. On the other hand, we can set to autoaccept nonmembers, and set certain addresses to hold for moderation or auto-discard.
I agree with Chris that posting from the newsgroup side of things should not need moderation by default.
I think we should be able to set the list to auto-approve non-members, and set any messages with a certain header (eg. Return-Path: <news@google.com> ) to hold for moderation at <https://lists.mozilla.org/admin/support-firefox/privacy/spam>. Dave, that should work, right?
maybe auto-approve stuff coming from giganews, hold stuff from google, and auto-reject non-members for everything else?
Okay, so in theory, if I set: generic_nonmember_action to Reject. Spam Filter Rule 1: Return-Path: <poster@giganews.com> Action: Accept Spam Filter Rule 1: Return-Path: <news@google.com> Action: Hold ...then it should work. My only worry is if generic_nonmember_action takes precedence over the spam filters. If the above should work, then let's try it.
I think we should experiment with one newsgroup (say mozilla.test? :) ) first then do the rest if it works the way we expect. :)
(In reply to comment #16) > I think we should be able to set the list to auto-approve non-members, and set > any messages with a certain header (eg. Return-Path: > <news@google.com> ) to hold for moderation at > <https://lists.mozilla.org/admin/support-firefox/privacy/spam>. Dave, that > should work, right? > Isn't the real target posts from Google Groups, not posters using GG/Gmail as their mailboxes? Seems like flagging on message-ids along the lines of *@google*.com would have the desired effect...
With a moderated newsgroup, the news server emails the posted message to the moderator address. I'm pretty sure in such cases the return path (the address the message will bounce to if it bounces) is set to an address at the news server, not the person who posted.
Okay, the above settings are applied to mozilla.test.
Except for "Open list, moderated group".
ok, I've requested Giganews switch mozilla.test over to moderated, and switched the setting in mailman so it'll show up as moderated to Google on tonight's scrape.
I just did some tests. So far, so good. Giganews clearly hasn't configured mozilla.test as a moderated group yet.
(In reply to comment #14) > Hmm... I don't know. I would rather go back to the original setup of Google > being read-only. Obviously there's resistance to asking anything from Google. > > Nir, Q, Robert, what do you think? The auto-approve list is going to be huge. Q here (late). I think having Google read-only as before would confuse people, since Google's interface isn't set up to make read-only status very clear to users. If the call were mine alone, I'd drop Google entirely, just stop feeding them and ask them to stop archiving the groups. If the support groups are to be moderated, even if there's a pass-through for non-Google posts, I think you'll need more than one moderator, so that legit Google posts can get through in a timely manner. I'm afraid I'm not willing to be part of that effort.
mozilla.test appears to be a moderated group on news.mozilla.org now. I just tried some tests, and it appears generic_nonmember_action is taking precedence over the spam filter, that accepts posts with Return-Path: <poster@giganews.com>. I also tried X-Complaints-To: abuse@mozilla.org, got the same result (auto-rejected, when posting via nmo).
I did something darn similar to this on the careers mailing list for HR... hang on a sec, let me see what I did there... generic_nonmember_action = Accept spam filters: ^(to|cc):.*careers@mozilla\.com = Accept .* = Discard Not quite the same (we lose the matching against the subscriber list) but it does eliminate messages that are Bcced (which most spams originating via email are).
The problem with that is, that it doesn't address users mistaking the list address for private support (ie. not subscribed, so not going to see any replies), which is why auto-rejection in on.
I've been doing some testing today, with the filters in comment 28. I managed to get it working properly, by turning off require_explicit_destination; but auto-accepted messages aren't making it to the news server. (List members receive the message via email, but it doesn't show up on news.mozilla.org, or Google.)
Appearantly that batch of posts took around 2 hours to show up. Further tests had very little latency. But messages from Google are being auto-accepted now. :-(
Okay, I think I fixed it, by moving the return-path filters before the ^(to|cc):.*test@lists.mozilla\.org filter.
Using the above method, we actually got a couple of spam messages last week in mozilla.test. I've been moderating *everything* from nonmembers and Google Groups, and have not seen the typical spam from Google in a while, that prompted this bug report. I'd be willing to set things back to normal on one or all the support groups listed in this bug, and see if we still get the frequent spam. Right now, things are bad for Google Groups posters who aren't seeing all replies. I'm still in favour of making the Google Groups end read-only. Q, I think the interface for read-only access is fine. For an example, see <http://groups.google.com/group/mozilla.support.general>.
(In reply to comment #33) > Q, I think the interface for read-only access is fine. > For an example, see > <http://groups.google.com/group/mozilla.support.general>. I don't think that notice is shown for active groups, just for groups that don't exist any more. E.g., net.news, discontinued over 20 years ago, <http://groups.google.com/group/net.news>. I don't recall clearly what Google presented no posting via Google was allowed for mozilla.support.* -- I only remember thinking it wasn't clear enough. And if the notice at the above links /is/ shown for active groups, I think it's a problem because it implies that there is no way to post to the group, via Google or otherwise. Maybe the best thing to do would be to make the groups read-only at Google, then if the interface isn't good, start lobbying Google to improve it.
*** Bug 430707 has been marked as a duplicate of this bug. ***
mozilla.dev.tech.svg and mozilla.dev.platform have been spammed heavily recently. So this spam is coming via Google Groups from spammer-controlled Google accounts? That sucks, especially since I post using Google Groups myself :-(.
I request similar relief for mozilla.dev.tech.crypto which has been heavily spammed from google groups. I moderate the associated mailing list dev-tech-crypto@lists and have cut off all messages from google at the news->mail gateway, which has helped the list immensely but not the newsgroup. Is there any reason not to offer similar protection for all Mozilla's newsgroups?
I've mailed our contact at Google (management side rather than the grunts at the abuse desk) to see if I can get it escalated on their end at all.
Got a reply back from Google already, they said they're already on it actually, and have been playing a game of whack-a-mole on spammer account signups. I gave him a list of the harder-hit newsgroups so they can keep a closer eye on them.
Renaming bug because: - newsgroups other than the support group are being included in this bug - making the groups read-only from Google is one of two or three solutions proposed.
Can they (or someone else) remove the spam from the newsgroups that have been/are being spammed?
Dave, After-the-fact whack-a-mole reactive handling of spam from googlegroups is not an acceptable level of spam control for dev-tech-crypto@lists.mozilla.org Given that Mozilla's news->mail gateway does NOT apply its configured spam filters to messages that it forwards from news to mail, a one-way gateway to google groups (mail->news only) is the only acceptable solution. I would prefer to be able to apply that only to google groups; that is, I would prefer to filter out only postings that originate at google groups and not those that originate on other news servers, but mailman doesn't give me any choice. It's all news spam or none. I choose none.
That's your prerogative. Glad you found something that works for you.
I wasn't very clear in comment 42. If the NNTP link between mozilla's news server and the rest of google groups was one way, sending news from Mozilla's server to google groups, but not the other way, then one could allow news that originated on Mozilla's news server itself to be gatewayed to mailing lists without much worry about spam. That would be desirable. It's a shame to have to disallow news from Mozilla's news server to enter the list, just to avoid google groups noise from entering the list.
So, has some action been taken that has once again restored the link between mozilla's news server and google-groups to a one-way link, disallowing traffic to flow in from google groups? Or is the mole just away for a little while recovering from his bruises? Also, can I unilaterally turn m.d.t.crypto into a moderated newsgroup using the list server admin UI? I mean, would that work? Or is there some other distributed change that needs to take place to maker that group moderated?
Another idea: drop Google and have Gmane provide the web interface for the groups. Gmane provides two web interfaces, and their archives are searchable. Folks who really want Google's interface could still subscribe to the lists with Gmail accounts. I read several MLs through Gmane, and there's never been a spam problem with them. Downsides: Gmane.org is less discoverable than Google.com, for users seeking support. I don't know how much less. Also, Gmane provides nntp access as well, which may be a problem depending on what the understanding with Giganews is wrt exclusivity. Gmane already carries some of the dev lists, e.g. <http://dir.gmane.org/gmane.comp.mozilla.devel.svg>. They gate user and dev lists for other FLOSS projects, so I assume they'd be happy to do so for Mozilla.
The spam from Google groups seems to have slowed way down or stopped. Do we know why? Was some action taken to prevent future Spammers from doing the same as the guy selling apparel? Or did they just spank that one guy, and no-body else has tried since? If something has been done to actively prevent future recurrences of the apparel spam, then I'll re-enable the news->mail gateway for m.d.t.crypto, but otherwise, I'll probably leave well enough alone (gateway disabled).
I haven't seen any spam in the support groups for a while. This seems to be fixed in the support newsgroups.
yeah, agreed, the abuse@mail is back down to the normal levels, and the moderation settings on the newsgroups are all back where they're supposed to be for a few weeks (with the exception of the crypto newsgroup - Nelson can do what he wants there).
> the moderation settings on the newsgroups are all back where they're > supposed to be Oh, where were they before? The question in comment 45 was never answered.
(In reply to comment #50) > > the moderation settings on the newsgroups are all back where they're > > supposed to be > > Oh, where were they before? The support groups were all previously unmoderated on both servers. We had experimented with making them moderated on Google in order to clear the spam problem. They are now unmoderated again as of the 19th. > The question in comment 45 was never answered. I know that Google tweaked up their anti-spam measures a bit. In addition to that, we may have discouraged that particular spammer with our moderation experiment and he stopped. Spammers aren't exactly predictable so we won't really know. It's the Internet, these things happen. Changing the moderated newsgroup setting in your list's admin panel will change it in mailman and on Google Groups (effective midnight PDT or shortly thereafter the following morning), but not news.mozilla.org. We have to open a ticket with Giganews to change that one.
We still need mozilla.test changed back to normal on news.mozilla.org.
(In reply to comment #51) > Changing the moderated newsgroup setting in your list's admin panel will > change it in mailman and on Google Groups (effective midnight PDT or > shortly thereafter the following morning), but not news.mozilla.org. Thanks for that info. That sounds like exactly the solution I wanted a month ago.
The Spam problem is back with a vengeance. (In reply to comment #51) > Changing the moderated newsgroup setting in your list's admin panel will > change it in mailman and on Google Groups (effective midnight PDT or > shortly thereafter the following morning), but not news.mozilla.org. The admin panel says clearly that that setting affects only traffic going from list to newsgroup, and not from newsgroup to list. Is it wrong?
Yes, because Mailman doesn't know that Google is polling our config and changing their settings to match every night. Do note that messages posted via news.mozilla.org will not make it to Google Groups after midnight the night you change that setting to moderated newsgroup unless we've also notified Giganews to make the same change (they aren't polling our config yet). Once the news servers have been changed to moderated, they email the post to the list submission address instead of immediately posting it, so it'll go into the list's moderator queue.
(I just got Cc'd here -- Chris, was there anything you needed me to do?)
(In reply to comment #56) > (I just got Cc'd here -- Chris, was there anything you needed me to do?) Because all the spam is being posted via Google Groups, there's a growing impetus to switch Google Groups access to the Mozilla newsgroups backing to being read-only. Considering it was you that filed bug 327251 to allow posting via Google Groups and had many comments in favour of it, I thought this was a bug should not only know about but also might have some counter-arguments. On second thought, posting about this on bug 327251 might have been better than simply CCing you. Sorry. :-[
I don't think there's much of a counter-argument needed -- nntp is a really poor way of letting people get involved in mozilla discussions, as is a mailing list; not being able to participate via google groups would make me argue that we should just dump nntp altogether. We should be able to put together the technology to do spam filtering. I would really be happier with something like a real google groups setup (non-nntp), with either explicit subscription or whitelisting by moderators on first message posted.
(In reply to comment #58) > We should be able to put together the technology to do spam filtering. Sure. Bug 403970.
Just to be clear, as long as Google Groups is involved, there's pretty much zero we can do about spam. They won't remove anything after the fact unless it's a DMCA violation. And we can all see how well their before-the-fact spam filtering is doing.
(In reply to comment #58) > I don't think there's much of a counter-argument needed -- nntp is a really > poor way of letting people get involved in mozilla discussions, as is a mailing > list; not being able to participate via google groups would make me argue that > we should just dump nntp altogether. > > We should be able to put together the technology to do spam filtering. I would > really be happier with something like a real google groups setup (non-nntp), > with either explicit subscription or whitelisting by moderators on first > message posted. Yeah, the solution I proposed earlier today in mozilla.dev.mozilla-org is somewhere along those lines. It was to move off of Google Groups, and use an in-house web interface/archive, that can peer with the mailing lists/newsgroups.
(In reply to comment #58) > I don't think there's much of a counter-argument needed -- nntp is a really > poor way of letting people get involved in mozilla discussions, as is a mailing > list; You may think that; but there are a lot of both NNTP users (e.g. me) and mailing list users who disagree. I find news to be the perfect interface for reading the Mozilla lists. The reason we spent time setting up a bi-directional mail/news gateway and connecting it to Google Groups is that people have strong preferences for one of the three methods of participating (NNTP/mail/web), and eliminating any method upsets its users. "Let's dump NNTP and mailing lists, and all use Google Groups" isn't an option. Gerv
(In reply to comment #62) > The reason we spent time setting up a bi-directional mail/news gateway and > connecting it to Google Groups is that people have strong preferences for one > of the three methods of participating (NNTP/mail/web), and eliminating any > method upsets its users. "Let's dump NNTP and mailing lists, and all use > Google Groups" isn't an option. In fact, doing that doesn't even solve the problem, since Google Groups is where all the spam is coming from. If we drop all the other methods, the spam will still be there. Google Groups is the piece that has to go if we want to get rid of the spam. So the basic problem is we need to find some other web-based solution for accessing our newsgroups/mailing lists.
There is a recent increase in SPAM level particularly in m.s.thunderbird. At least 10 SPAM are posted within 4 days using Google Groups. Some measure needs to be taken to deal with this SPAM problem in mozilla newsgroup.
*** Bug 479949 has been marked as a duplicate of this bug. ***
I think GMane is fully open source. Maybe setting up a separate instance of that for Mozilla would make sense?
I've had good experience with Newsportal <http://floh.gartenhaus.net/newsportal/index-english.php3>. You can see it live in action on www.newsoffice.de (German-speaking website).
Reed's had the last couple communications with the Google folks on this, so I might as well assign this to him for now.
Any updates on the progress here?
Now scanning all mail with SpamAssassin before forwarding on. Will modify scores appropriately as needed as time goes on. Let's see how this does.
Spam sent via Google Groups was posted in the newsgroup today in mozilla.support.thunderbird. news://news.mozilla.org/6b7175ee-4593-4af0-8b57-c69a1056dec3@o7g2000yqb.googlegroups.com
(In reply to comment #71) > Spam sent via Google Groups was posted in the newsgroup today in > mozilla.support.thunderbird. > news://news.mozilla.org/6b7175ee-4593-4af0-8b57-c69a1056dec3@o7g2000yqb.googlegroups.com Ah, so, my solution just dealt with people subscribed to the mailing list. Let me contact Google and see what can be done about spam in general like that.
In all the years that I've been moderating mozilla/netscape mailing lists that were also tied to newsgroups, spam from list subscribers has NEVER been a problem. There is a known spammer email address subscribed to dev-tech-crypto, and we have NEVER gotten a single spam from it. All the spam comes into the lists through the news->mail gateway, against which mailman apparently has no defenses. I just don't understand why it's so difficult to treat all messages that come from the news->mail gateway like any other messages received by email, and run them through the very same spam filters that messages received by email go through. That would require almost NO extra configuration effort by the list owner/admin/moderator, and would completely cure the problem for the mailing lists.
The changes that have been recently made to lists.mozilla.org to attempt to do spam filtering have broken a fundamental capability of that list server. Whenever a message is flagged as spam by spam assasin, it goes into the moderation queue, EVEN IF the list owner has configured the list to DISCARD all emails from non-list subscribers. So, the option to discard all email from non-subscribers no longer works. This is a big regression.
The new messages about Spam Assassin show "scores" that differ from SA's own scores. Here are some examples: > SpamAssassin gave the message a score of 5.4 for the following reasons: > X-Spam-Score: 5.549 > X-Spam-Level: ***** > X-Spam-Status: No, score=5.549 tagged_above=2 required=6.31 > SpamAssassin gave the message a score of 7.8 for the following reasons: > X-Spam-Score: 5.862 > X-Spam-Level: ***** > X-Spam-Status: No, score=5.862 tagged_above=2 required=6.31 > SpamAssassin gave the message a score of 9.7 for the following reasons: > X-Spam-Score: 4.991 > X-Spam-Level: **** > X-Spam-Status: No, score=4.991 tagged_above=2 required=6.31 tests=[AWL=0.749, 7.8 vs 5.8? 9.7 vs 4.9? Is this the "new math" ? :)
(In reply to comment #74) > The changes that have been recently made to lists.mozilla.org to attempt to > do spam filtering have broken a fundamental capability of that list server. > Whenever a message is flagged as spam by spam assasin, it goes into the > moderation queue, EVEN IF the list owner has configured the list to DISCARD > all emails from non-list subscribers. So, the option to discard all email > from non-subscribers no longer works. > > This is a big regression. Indeed. I'll see about getting this fixed as soon as I can. (In reply to comment #75) > The new messages about Spam Assassin show "scores" that differ from SA's > own scores. Here are some examples: > > > SpamAssassin gave the message a score of 5.4 for the following reasons: > > X-Spam-Score: 5.549 > > X-Spam-Level: ***** > > X-Spam-Status: No, score=5.549 tagged_above=2 required=6.31 I don't know where you're getting those X-Spam-* headers from... Where are you seeing them? Those aren't matching up with what is set on the server-side, so it almost seems like they may be your local settings or something? or at least I'm not seeing them for some reason.
> I don't know where you're getting those X-Spam-* headers from... > Where are you seeing them? They're in the message itself, as seen on the mailman web page that shows the message content to the moderator
(In reply to comment #77) > > I don't know where you're getting those X-Spam-* headers from... > > Where are you seeing them? > > They're in the message itself, as seen on the mailman web page that shows > the message content to the moderator Interesting. They seem to be stripped out before the messages make it to users. Ok, thanks.
This bug's lost momentum and I can't figure out what has to happen to close it. Someone care to help me?
At this point, I would say that this has become a dupe of bug 403970, since there don't seem to be any other proposed solutions than finding a different host for the web-end. But that's just my opinion.
(In reply to comment #80) > At this point, I would say that this has become a dupe of bug 403970, since That bug is about archiving USENET on a Mozilla hosted NNTP server. This one is about about spam which we appear to have little control over with Google. Still not sure how to fix this bug.
Inclined to CANTFIX this bug - any disagreements?
http://ejohn.org/blog/google-groups-is-dead/
I suggest this gets resolved as INVALID, really, as it's not something we can control, yeah.
and the bug count drops by one.
(In reply to comment #84) > I suggest this gets resolved as INVALID, really, as it's not something we can > control, yeah. It could be controlled by cutting off peering with Google Groups. If that's not a good option because Mozilla wants/needs to rely on Google Groups' web interface, IMO this should be resolved WONTFIX rather than INVALID.
Fair enough. What we have now is, as far as we can tell having looked at things quite closely, the least worst option which meets all of the design criteria. Gerv
or
