348858 - auto-discard messages that would be held for moderator approval, and improve the generated message

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, P1)

Description

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

Moderation of our mailing lists is simply not working -- it forces the moderators to wade through a vast amount of spam, and, in the exceedingly-rare case that a human has their "real" message held, it sets an unfortunate expectation that a moderator will let their message through in a reasonable time frame.

I propose the following changes be made:
1) reject all messages that would currently be held for moderation (not discard)

2) replace the rejection message with one that points to the appropriate google group and the subscription URL, as well as whatever documentation we have on using the news server

3) remove the recipient-limit from our lists (this is the only remaining case for moderated "ham" that I've seen in any meaningful quantity)

We already send out "held for moderation" messages in response to spam that makes it to mailman, so I'm not concerned about giving additional data to spammers who are "fishing" for addresses, or fallout from mail sent to forged sender addresses.

I believe these changes would improve the experience for "legitimate" users caught in our moderation trap today, as well as removing the onerous (and therefore often ignored or carelessly discharged) duty from our moderators.

Comment 1

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

http://www.mail-archive.com/[e-mail]/msg40124.html outlines a similar approach to what I suggest, I think, though not quite identical.

Comment 2

[Jeremy Orem [:oremj]]

Reassigning to Dave.  I believe he was working on it.

Comment 3

[Dave Miller [:justdave]]

(In reply to comment #2)
> Reassigning to Dave.  I believe he was working on it.

I didn't know that. ;)

But I'll be happy to.  This makes good sense for any lists still moderated by [e-mail], because that pile is just too big to deal with.  Probably be a per-admin decision for the lists that actually have real admins.  I'll make the change on the unmanaged lists after I get back from vacation and email the remaining list admins to let them know what to change if they want to do the same.

Comment 4

[Deb Richardson [:dria] (plz NEEDINFO)]

This depends how you define "real admin". :)

I'm admin on a fair number of lists, but should only be considered a real admin of dev-mdc. Even then, however, the dev-mdc list moderator requirements would largely be taken care of through automation, and I'd appreciate that.

Comment 5

[Dave Miller [:justdave]]

OK, we were brainstorming about a bunch of other things on #mozwebtools just now, and came up with a kick-ass idea for how to deal with this.

I *really* don't like the idea of creating lots of backscatter from forged sender addresses on spam coming into the system, so I dislike the idea of sending bounce notices back to the sender (even to tell them it's been held for moderation).

But...  if we could tie postfix into mailman so that postfix could look up who the subscribers were and reject the mail during the SMTP process, then the sending mail server would be responsible for the bounce, not us, and if the sending mailserver was a spammer (and not using an open relay) then they'll usually just give up instead of generating one.  Either way, that makes it "not our problem" at that point.

It just so happens you can actually do this in postfix.  Postfix has this neat little concept called policy daemons.  You run a program that listens on some arbitrary port.  Postfix will feed it some information about the email, and it'll reply with whether or not to accept the mail.  Postfix has generic policy daemon skeletons in a variety of languages on their website, and one of them is in python.  Mailman has python modules you can load to obtain access to said data.  Put it all together and we have a nice clean solution to this problem.

>>> MAIL FROM:<[e-mail]>
<<< 250 Ok
>>> RCPT TO:<[e-mail]>
<<< 550 [e-mail] must subscribe to this list first before posting to it.

Of course, we can make it do the "subscribed to any list on this server" thing like we're currently doing instead of just that list.

Comment 6

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

I hear through the grapevine that we did something for some mailing lists -- any update?

Comment 7

[Dave Miller [:justdave]]

There was discussion on the governance list briefly about it.  That's about where it sits.  I've got other projects on my plate at the moment, and hoping to get to this after the current stuff is done.  It's likely to be mid-December before I get to it though.

Comment 8

[Jeremy Orem [:oremj]]

Changing QA Contact.

Comment 9

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(In reply to comment #7)
> There was discussion on the governance list briefly about it.  That's about
> where it sits.  I've got other projects on my plate at the moment, and hoping
> to get to this after the current stuff is done.  It's likely to be mid-December
> before I get to it though.

Any update?  The unsubscribed-posting experience still sucks pretty badly, as does the moderator experience.

Comment 10

[Mike Beltzner [:beltzner, not reading bugmail]]

poke/prod/pry/bribe

Comment 11

[matthew zeier [:mrz]]

Grabbing as something to look at for Q1 2009.  (P1 == Q1)

Comment 12

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

We gonna get this this quarter?

11:22 < justdave> Subject: 1741 Sheriffs moderator request(s) waiting

Comment 13

[matthew zeier [:mrz]]

Yes.  Reed has committed to having it done.

Comment 14

[Reed Loden [:reed]]

(In reply to comment #0)
> 1) reject all messages that would currently be held for moderation (not
> discard)

Done. Note that existing messages in the queue have not been touched.

> 2) replace the rejection message with one that points to the appropriate google
> group and the subscription URL, as well as whatever documentation we have on
> using the news server

Text is now as follows -- feel free to submit copy changes:
"
Greetings,

In order to protect our mailing lists from spam, we require that all
posters be subscribers of the list to which they are posting. We
support several different ways of submitting your post. Pick the one
that works best for you.

If you wish to continue using e-mail to post, please visit
https://lists.mozilla.org/listinfo/test and subscribe to the list.
Once you have been subscribed, you may resubmit your post.

If you wish to use NNTP to post, you can submit directly to
nntp://news.mozilla.org/mozilla.test. More information about our news
server and how to use it can be found at
http://www.mozilla.org/community/developer-forums.html.

If you wish to use Google Groups to post, you may submit using a
Google Account at http://groups.google.com/group/mozilla.test.

We're sorry for this inconvenience, and we appreciate your patience
and understanding in helping us prevent spam from reaching our lists.

Thanks.
"

> 3) remove the recipient-limit from our lists (this is the only remaining case
> for moderated "ham" that I've seen in any meaningful quantity)

I can't easily completely remove it, but I raised it from 5 to 20. I can raise it higher if you think 20 is still problematic.

Note that all these changes have been made only to the newsgroup mailing lists (those on lists.mozilla.org) and not to any of the mailing lists on mail.mozilla.org.