209181 - Bugzilla should track bugmail that bounces and automatically disable bugmail on accounts that continually bounce.

Status: NEW

(Bugzilla :: Email Notifications, enhancement)

Description

[Jason M. Abels]

I think it would help in the triaging of bugs to know if a reporter's email
address is still valid. If we get a bounce message from the reporter, a flag or
notation should be added to the bug telling us that the address appears to no
longer be valid. This is helpful especially when dealing with somewhat old bugs,
and we are waiting on confirmation from the reporter. If the reporter's address
is dead, we'll be waiting a long time.

Comment 1

[Matthew Cline]

Seems like a good idea; confirming.

Also, Hardwars/OS -> All/All

Comment 2

[J. Paul Reed [:preed]]

This is not as easy as it sounds.

Currently, Bugzilla mail comes from <webservergroup>@hostname; this address
typically doesn't go anywhere, unless the administrator set up an alias to
handle this mail somehow. So, the bounces don't go anywhere unless the
administrator sets up an alias to handle this mail somehow.

To do what you're suggesting, there would have to be some bugzilla script that
would listen to that alias, be able to detect a bounce message (difficult, if
not impossible, due to all the different formatting of bounces; although, there
may be an RFC standard for X-Message-Status or something like that), and then do
something with that information.

There's also no authentication with SMTP, so people could start forging bounce
messages to the alias, and get prompt the alias to do whatever it normally does
(post comments to the bug? Close the account? What?) without any authentication.

The security issues raised by fixing this are not insignificant, and involves
coopting the webservergroup address, which other applications (PHP scripts,
etc.) may use, or administrators may not want to redirect to a script (maybe
they use it for something else.)

Email has always been considered a secondary notification (i.e. the Bugzilla
installation itself is the final version of the status information for the bug),
so email should *never* be relied upon to get bug information.

Unless anyone can point out some huge point I'm missing, I propose marking
RESOLVED/WONTFIX.

Comment 3

[Daniel Berlin]

Actually, it's not as hard as you make it out to be.
In fact, I do it on gcc bugzilla.

1. Use a return-path to make the bounces go to whatever alias you want.
2. See RFC1892 for delivery status notifications.
3. There's a perl module that handles most formats of bounce messages out there, standard and 
non-standard.  See Mail::DeliveryStatus::BounceParser.  We can always extend the messages 
4. Nobody forges bounce messages. If they do, you should block bugzilla from allowing them 
anyway.  This is a people problem.  We aren't disabling or deleting accounts, just marking a 
possible bounce.  We can record the ip address that sent the bounce if you are really that 
concerned, if this is a problem.
5. If you are really super concerned about bounces, we could record the message id of each 
message we send out (for 30 days or something), as well as who it was sent to, and match the 
bounce back to it, and not mark any address as possibly bouncing we can't do this for.

Comment 4

[J. Paul Reed [:preed]]

Ok... that's good to know how it's done elsewhere.

But you hit the nail on the head: this is a people problem.

As I noted, email was ALWAYS considered a secondary notification tool; it should
not be relied upon to get BZ information out to people.

Still propose RESOLVED-WONTFIX.

Justdave?

Comment 5

[Dave Miller [:justdave]]

the other thing you can do is set the return path with a + extension, which
sendmail and other MTAs will ignore anything following the +, but it'll still be
included in the headers for the benefit of the recipient.  Do you could do
something like

MAIL
FROM:<[e-mail]>

where Bxxx is the bug number, Txxxx is the unix timestamp used in the activity
log/long descriptions for the bug change, and the rest is obvious.

Majordomo 2 does it in a similar fashion to track mailing list bounces.

Not sure the fact that it bounced should be noted in the bug, but some sort of
bounce tracking has been on the idea list for Bugzilla in the back of my head
for some time now.

It's an easy way to detect dead accounts.

Comment 6

[J. Paul Reed [:preed]]

Well, then I'll accept.

We still need to come up with what to do with a bounce report though... that's
not really clear from the bug report.

Comment 7

[Stefan Baebler]

It can easily happen sooner or later that some (too) strict mail providers will
block _all_ emails from being delivered to his mail server if there are too many
mails that bounce being sent from one server (eg bugzilla) to his system.

yahoo.com is one (of many?) that enforces: "if you send me too many mails that i
can't deliver, i won't accept your mails at all, not even the ones i could
deliver" as their way of fighting spam.
http://help.yahoo.com/help/us/mail/spam/spam-18.html
Their reasoning is that you must be sending spam to some outdated list of email
addresses.

They suggest Variable Envelope Return Paths to detect bounces:
http://cr.yp.to/proto/verp.txt

Comment 8

[Stephen Lee]

See also bug 94247, where the proposal was that the reporter should be able to 
ABANDON a bug (giving it a blank, or otherwise specially marked reporter field 
to indicate that they are no longer involved).

Comment 9

[Wayne Mery (:wsmwk)]

-> default assign (JP no longer works on bz)

(In reply to comment #7)
> ... 
> They suggest Variable Envelope Return Paths to detect bounces:
> http://cr.yp.to/proto/verp.txt

VERP is interesting suggestion - implmented in debian
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=305324

additional references
http://packages.debian.org/stable/perl/libmail-verp-perl
http://search.cpan.org/src/GYEPI/Mail-Verp-0.04/Verp.pm
http://packages.debian.org/testing/perl/libmail-verp-perl
http://www.courier-mta.org/?sendmail.html
http://www.postfix.org/VERP_README.html

Comment 10

[Frédéric Buclin]

*** Bug 299795 has been marked as a duplicate of this bug. ***

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to comment #4)
> ... : this is a people problem.
> 
> As I noted, email was ALWAYS considered a secondary notification tool; it should
> not be relied upon to get BZ information out to people.

Paul, please explain - if bz e-mail is secondary then what is primary notification method?

By "this is a people problem" do you mean one should assume bz mail won't be delivered, that one either calls or pings the person (immediately or after some period of time)?

Don't most people rely upon bugzilla as THE controlling process to get information to and from people? E-mail is the only identifier in the bug (eg. bz doesn't have irc nicknames) so it certainly is a *major* piece of bugzilla. Even if it is as you say, secondary, bz email surely is still *very* important. I have been told by developers and I have seen in documentation that one should request info from others via bugzilla rather than use direct email (i.e. between two people) - to perserve tracking of conversation and activity within the bug.

Further observations:
* If this is difficult to implement it doesn't make it any less important
* Developers and triagers have a different takes on usage of bz, might this bug be one of them?
* Don't modern tracking tools tout e-mail notification and update of issue/bug via e-mail as a major feature/selling point?
* this is big time waster for the triager
* this contributes to delays in making progress on bugs both new and old
* this is not just a problem for old bugs - email addresses expire for new reporters and commenters too

UI complexity need not increase. Examples:
* strike-through name+address if it's not getting mail
* mousoever could indicate date of last successful delivery or failed delivery


Same functionality for cc: and commentors would be equally helpful.

Does Bug 249121 add complexity to this bug?

Comment 12

[Denis Roy]

When we get bounces, we often configure the user's EMail Prefs to "Disable All Mail", then we disable the account with a message saying "Mail bouncing - contact [e-mail]". It's been working good so far, except the whole process is manual. 

As for comment 2, to prevent forging, set a threshold for bounces.  User's mail bounced 5 times? Disable all mail going to them. Optionally disable their account with [insert message here].

Ignoring bounces is bad in that mail server admins on the receiving end can end up putting busy bugzilla sites on SPAM RBL's because they continuously send them mail to invalid addresses.

D.

Comment 13

[Marc Schumann [:Wurblzap]]

This seems to be a thing that'd need to be very site-specific in its implementation. Proposing either WONTFIX (allowing a site to handle it however it wishes it to be handled), or moving to b.m.o component.

Comment 14

[Denis Roy]

> This seems to be a thing that'd need to be very site-specific

I'm not sure I agree.  Add this to the email parameters:

bounced_email_processing

Consider user is AWOL after [5] bounced messages
Action(s) to take when someone is declared AWOL:
  [x] Disable account
  [x] Append "AWOL" to real, so everyone knows about it
  [x] Disable all bugmail to this person
  [ ] Notify site admin

Comment 15

[Denis Roy]

Should be:
>   [x] Append "AWOL" to realname, so everyone knows about it

Comment 16

[Nikolay Shopik]

To make it 100% accurate we can check only our local MTA bounces which generated during SMTP session but not when remote server accept email and bounce mail later. This slightly decrease amount of dead addresses because there still front-end servers which accept mail and bounce later.
To accomplish this we need just small shell script, which can be written within hours. I will look into bugzilla code and see if I can help with this.

Comment 17

[Nikolay Shopik]

Currently field "Reporter accessible?" used for same thing but changed manually, it can be used for this purpose on automatic manner

Comment 19

[Dave Miller [:justdave]]

Hmm, BMO does this. I don't know if it's before or after we forked it back to harmony though. If before, we may have this in Harmony already.

Comment 20

[chrizilla]

(In reply to Denis Roy from comment #12)

When we get bounces, we often configure the user's EMail Prefs to "Disable All Mail", then we disable the account

Why disable the account, isn't "disable all mail" enough ?

Comment 21

[Wayne Mery (:wsmwk)]

(In reply to chrizilla from comment #20)

(In reply to Denis Roy from comment #12)

When we get bounces, we often configure the user's EMail Prefs to "Disable All Mail", then we disable the account

Why disable the account, isn't "disable all mail" enough ?

In BMO it's because we want to know there is someone on the other end (at that address) who will respond.