353279 - Hundreds of bounces daily from "private message reminders"

Status: RESOLVED FIXED

(Websites Graveyard :: spreadfirefox.com, defect)

Description

[Dave Miller [:justdave]]

Every morning, Spreadfirefox sends out reminders to people who have unread private messages.  Spreadfirefox has been around for a while, and email addresses can get discontinued or other problems over time.  We get LOTS of bounce messages from that daily reminder coming back every morning for email addresses that are dead.

Does Drupal's reminder system have any way of trapping the bounce notices?  If so, we can configure an email address to feed into that, and I can change that cron job to use that email address as the sender so we don't get them all on root all the time.

Comment 1

[Paul Booker]

>>Does Drupal's reminder system have any way of trapping the bounce notices? 

I am not sure how that could work Dave  ...

The privatemsg module sends out its emails every morning via privatemsg_cron()  >> _privatemsg_mailalert() informing sfx subscribers of any unread private messages. 

These emails are sent out from drupal and are then divorced from drupal to the best of my knowledge they then bounce from one mail server who can't deliver the email back to the mail server from whence it came ? 

>>I can change that
>>cron job to use that email address as the sender so we don't get them all on
>>root all the time.

We can easily adjust the sender from '[e-mail]' to something else
if that helps.

Perhaps we should modify _privatemsg_mailalert() so that it only sends out reminders for a fixed period of time that should reduce the ammount of 
emails that are currently bouncing.

Regards
Paul

Comment 2

[Paul Booker]

Hi Again

Ian has asked me to suggest if its possible (In reply to comment #1)
> >>Does Drupal's reminder system have any way of trapping the bounce notices? 
> 
> I am not sure how that could work Dave  ...
> 
> The privatemsg module sends out its emails every morning via privatemsg_cron() 
> >> _privatemsg_mailalert() informing sfx subscribers of any unread private
> messages. 
> 
> These emails are sent out from drupal and are then divorced from drupal to the
> best of my knowledge they then bounce from one mail server who can't deliver
> the email back to the mail server from whence it came ? 
> 
> >>I can change that
> >>cron job to use that email address as the sender so we don't get them all on
> >>root all the time.
> 
> We can easily adjust the sender from '[e-mail]' to something else
> if that helps.
> 
> Perhaps we should modify _privatemsg_mailalert() so that it only sends out
> reminders for a fixed period of time that should reduce the ammount of 
> emails that are currently bouncing.
> 
> Regards
> Paul
> 

Is it possible for sendmail,.. to record in a database what emails bounce we can
then assume these accounts are no longer active and not send out any further reminders ? 

Comment 3

[Dave Miller [:justdave]]

(In reply to comment #1)
> We can easily adjust the sender from '[e-mail]' to something
> else if that helps.

The sender address on the mails is currently [e-mail].  That defaults to the user running the cron job I believe.  It would need to be overridden by using the -f option to /usr/sbin/sendmail  (this is the envelope sender, not the one in the From: line in the email)

(In reply to comment #2)

> Is it possible for sendmail,.. to record in a database what emails bounce we
> can then assume these accounts are no longer active and not send out any
> further reminders ? 

Sendmail can easily be set up to mail them somewhere (to a script even) which is free to record it somewhere...

Comment 4

[Paul Booker]

> Sendmail can easily be set up to mail them somewhere (to a script even) which
> is free to record it somewhere...
> 

This sounds like something that i can get working

So if we change the sender from  [e-mail]
to [e-mail] then when we get a  bounce
and the email comes back looking for  [e-mail]
we can get sendmail to redirect this mail to script as a 'post' (??) which will then parse the email to extract email address of user which is unique to the SFx user and from there we can determine|write information to a table about  problematic users 

How does this sound Dave ?

Best Regards
Paul

 

Comment 5

[Dave Miller [:justdave]]

(In reply to comment #4)

> So if we change the sender from  [e-mail]
> to [e-mail] then when we get a  bounce
> and the email comes back looking for  [e-mail]
> we can get sendmail to redirect this mail to script as a 'post' (??) which will
> then parse the email to extract email address of user which is unique to the
> SFx user and from there we can determine|write information to a table about 
> problematic users 
> 
> How does this sound Dave ?

Sounds good to me.  The email will come in on standard input, full headers and body.  Parsing bounces isn't easy because they're all different, and you're likely to get spam on that address as well (especially since it's been mentioned on a public bug here) so make sure to just discard anything you can't tell is a bounce.

Comment 6

[Paul Booker]

Hi Dave.

The emails sent out by privatemsg.module  _privatemsg_mailalert()  are now
coming from [e-mail] for SSFx [see notes]

privatemsg.module  _privatemsg_mailalert() 
132   $from = variable_get('site_mail', ini_get('sendmail_from'));
      $from = "[e-mail]"; 

Would you please redirect all bounced emails coming to [e-mail] to bounced_email_parser.php ?

I'll then get my ereg() out and attempt to write a script to parse the bounced emails.

I need to give some thought with regards to setting up an environment on SSFx
so that i get a manageable number of bounced emails to handle and to inspect 
so i can see what i am actually getting as i have never done this before.

I did hammer google for some information but i didn't find anything really.



Notes : All paths are relative to drupal root /home/paul/spreadfirefox on the sandbox server willow.mozilla.org (SSFx)

Regards Paul 

Comment 7

[Paul Booker]

Hi Dave

Always looking for a simple solution.

How about we have a a second cron schedule that runs once every 6 months say which sets all private messages as having been read except for the most recent (past week lets say)

By choosing how long the second cron schedule should run we should be able to get the number of bounced emails down to zero/very low number 

Shall we do this Dave ?

Regards
Paul

Comment 8

[Paul Booker]

Apologies , i haven't kept up with this ticket
(Paul adds it to his tasksheet)

Do you still need assistance here Dave ?

Regards Paul .

Comment 9

[Dave Miller [:justdave]]

(In reply to comment #8)
> Apologies , i haven't kept up with this ticket

Er, same here, sorry about that.

> Do you still need assistance here Dave ?

I do actually.  But I think the next task in the list is in my court (getting the mail alias set up to redirect the bounces back in to you guys).  Did you have a cron job that marks everything older than 6 months as read?  That might be a good idea to do as well.  If you still need examples of bounces I can probably get you a bunch.

Comment 10

[Paul Booker]

Ok  , 

I'll now write some code so that all private messages older that 6 month as marked  as read (which should alleviate matters) and if you would send me a bunch of example of bounced emails that couldn't be sent ill evaluate how easy it will be to parse these generally to extract the emails for further processsing so that we can then mark all account messages as read and flag the account as inactive  .



-Paul

Comment 11

[Paul Booker]

Please let me know if the code added to the privatemesg.module
has alleviated matters.
https://bugzilla.mozilla.org/show_bug.cgi?id=369840

All private messages old than a month should now be marked as having been read .


-Paul

Comment 12

[Ian Hayward]

@ Dave, I'm tidying up some SFx related bugs, is this one fixed for you Dave or do we need to keep it open?

thanks, Ian

Comment 13

[Paul Booker]

Closing ticket .