658972 - Implement global blacklist for mailman

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Gervase Markham [:gerv]]

One of the key aspects of the new newsgroups system:
https://wiki.mozilla.org/Discussion_Forums/Proposal
is a global sender blacklist, which helps distribute the work of keeping spammers out. I manage 5 lists, and I would love it if banning a sender from one list banned them from them all.

This bug is a request to implement a global sender blacklist, in parallel to the current global sender whitelist, which is made up of the sum of the blacklists of all the lists.

Gerv

Comment 1

[Shyam Mani [:fox2mike]]

Over to Dave for comment, punt back to server-ops if needed for implementation? CCing Rob as well.

Comment 2

[Dave Miller [:justdave]]

Needs a little python scripting done.  Pulling in dumitru since he's handling most of the newsgroup stuff these days.  There's a script on notorious that does global whitelist based on all of the list subscribers, should be pretty similar to that.  Will need to have Mailman itself patched to honor the list once it's built.

Comment 3

[Gervase Markham [:gerv]]

mrz: can this get into the plan for Q3?

It would make spam handling for the discussion forums an order of magnitude easier, which would ease the burden on moderators, which would mean we could more easily recruit moderators, which would mean we could switch more groups to moderated, which would mean we could have better discussions and control the signal-to-noise ratio.

Gerv

Comment 4

[matthew zeier [:mrz]]

Corey's handling q3 scheduling.

Comment 6

[Gen Kanai [:gen]]

Corey, any update on when this can be accomplished? This would significantly help control spam across all Mozilla mailing lists.

Comment 7

[Chris Ilias [:cilias]]

If I understand correctly, this is the bug that is blocking progress on converting more forums. Is there anything I can do to help get this fixed?

Comment 8

[Gervase Markham [:gerv]]

Chris: I talked to mrz about this bug at the Mozilla Festival. IT is really busy right now, but I wrote an email to Phong Tran making the case for prioritizing this. We'll have to see what he says :-) There is a possibility that IT's excellent moves to increase community involvement may lead to a community member having sufficient access to do this hacking. We'll have to see about that too :-)

Gerv

Comment 9

[Rob Tucker [:rtucker]]

I'm happy to volunteer to write the python scripts.

Comment 10

[Jean-Marc Desperrier]

Can't, as part of the increase of community involvement, the *tools* be open sourced and available for external hacking, even if for the time being only a small team has access to the platform to deploy them ?

I'd love to open a discussion about that, don't know where to do it though.

Comment 11

[matthew zeier [:mrz]]

Yes.  I don't think we actually write any tools that aren't assumed to be open.

If you were to start on this right now, what would you need?

Comment 12

[Jean-Marc Desperrier]

Some instructions that allows to setup an environment that correctly replicates what Mozilla uses, and allows testing locally if any change works properly.

I think this means :
- knowing which tools are used, if mailman really is the only thing, what version of mailman and if they are any patch
- a description of how the platform works, and how it's configured, how the spam filtering and various white/black list are plugged in currently

If actually Mailman 2.1.14 is all you use, maybe the description is not so long, it's just that's it's a bit hard to judge from the outside.

The best would be for all that to be described somewhere in a link accessible from https://wiki.mozilla.org/Discussion_Forums

Comment 13

[Dave Miller [:justdave]]

It is Mailman, but we do have local patches to it.  It is deployed as an RPM, so we probably ought to be able to make the SRPM available somewhere.

Comment 14

[Jean-Marc Desperrier]

OK, if nobody's available to create the package, would it likely be worth it if I try to install a default Mailman 2.1.14, and play with it to see if I already can find out how it should be done and try to write a patch ?

If a patch is there, who would be available to carry it until production ?
(which includes doing the tests required so that you trust the patch enough to add it to production)

Comment 15

[Gen Kanai [:gen]]

Can we consider hiring a Mailman consultant to work on this? Jean-Marc's offer is generous but getting a subject-matter expert to help would be faster and this bug has been open for over 6 mo. Mozilla lists affect the entire project.

Comment 16

[Rick Bryce [:rbryce]]

Update:  I am working on a solution to our mailing list woes.  In the next few days I will push out a global blacklist mechanism that will allow moderators to blacklist/whitelist email before they make it to the mailing list.

Comment 17

[Gen Kanai [:gen]]

Rick, glad to hear that. Could you provide more details as to the solution you are architecting? I'd like to make sure that we all agree that the solution you are building is the one we _all_ want.

Comment 18

[Rick Bryce [:rbryce]]

Gen -- The idea is to give the moderators a mechanism to block/allow email address without having to login to the mailman admin or go to many lists to block one address. I have setup 2 email address for moderators to forward spam and false-positive to.  From there a script will check these emails and process them accordingly.  Basically, the script will feed the tools already in place to handle spam and populate the black/white list files.

Comment 19

[Gervase Markham [:gerv]]

rbryce: can we have a status update? Are you still working on this, or did you get pulled off to work on other things? How far have you got?

Thanks,

Gerv

Comment 20

[Rick Bryce [:rbryce]]

:gerv 

I haven't touched this in while.  I knew that server hardware and software versions were going to change as part of the data center migrations.  Now is a great time to get back at this. I will provide more details very soon.

Comment 21

[Gen Kanai [:gen]]

Rick, any updates on this? All of us who moderate mailing lists at Mozilla are eager to have this implemented.

Comment 22

[Brendan Dahl [:bdahl]]

I manage the pdf.js mailing list and also would like to see this implemented.  

However, instead of a global block list why don't we get a better spam filtering solution?  I routinely get 3-4 messages a day and then nag messages from mailmain about this spam.  With the amount of lists we have we're wasting a lot peoples time filtering out so much spam manually.

Comment 23

[Dan Mosedale (:dmosedale, :dmose)]

As a manager on multiple mailing lists; I would also find this very helpful -- it would allow me spend less time doing mental context switches and more time doing real work.

Comment 24

[Luke Crouch [:groovecoder]]

+1 from moderator of dev-mdn list

Comment 25

[Rick Bryce [:rbryce]]

Im closing this bug wont-fix.

A global blacklist for all mailing list is a scaling nightmare. As well as, not being a complete solution to the problem (GigaNews & GoogleGroups)   Currently spam assasain is doing a better job since migrating mailman servers to PHX.  The IT/Infra group will be addressing this with the community @ the IT work week in 2 weeks.  There are a collection of bugs containing a miriad of problems and complaints.  We are going back to the drawing board as it pertains for mailinglists, googlegroups, giganews, and the spam that comes along with the task.  We will be making a company and community wide address when we hammer out the plan in 2 weeks.  If you have any questions or comments please email me directly.  I will be sure to bring any concern you might have to our meeting.