44393 - develop an extension that provides comment-only inbound email support

Status: RESOLVED WONTFIX

(bugzilla.mozilla.org :: General, task, P3)

Description

[Ben Bucksch (:BenB)]

From bug 6419:
------- Additional Comments From Dave Miller 2000-05-20 09:54 -------
Something to point out is that with bug_email.pl as far along as it is now (we 
can already submit via email and add comments), this would make queries via
email easy to implement, too.

------- Additional Comments From Dave Miller 2000-06-30 18:36 -------
[...]
Replying to the bugzilla spam to add comments is already possible (but doesn't 
really have much to do with this particular bug).  Look at bug_email.pl and 
appendbugmail.pl (or whatever their real names are) and the associated README in 
the contrib/ directory.

------- Additional Comments From Dave Miller 2000-06-30 19:09 -------
Ah, that's right...  mozilla.org doesn't use it.  I think this is because the 
error messages and confirmation notices it sends out still leave a lot to be 
desired, and they'd tend to scare people off.

---
Please fix the remaining problems and enable this for Mozilla's bugzilla. Email
apps are so much more comfortable (larger window, save sa draft etc.) than a
<textarea> and also fast, because I don't have to click on the link, scroll down
etc..

BTW: How did you solve the authentication problem?

Comment 1

[Dave Miller [:justdave]]

Authentication problem?  Hmm, yeah, that's a good one.  It's easy to spoof an 
email address.  I haven't fixed it on my system other than make the incoming 
email for new bugs go to a pending product with restricted visibility, then 
moving them where they go after they come in.  Easier on the users, more work for 
us.  Well, we could always make users register to be able to submit stuff via 
email, and make them upload their public PGP key to their user profile.  They 
would then have to sign their emails for them to be accepted.

Comment 2

[Ben Bucksch (:BenB)]

Yep, that was my idea as well. But you should support both PGP and S/MIME, if
possible.

Comment 3

[Dave Miller [:justdave]]

I should point out this is a Mozilla.org-specific bug...  Bugzilla in general can 

already do what this bug asks for, Mozilla.org just doesn't have the option 

enabled right now.



I'd ask if the people responsible for bugzilla.mozilla.org (I just added them to 

the CC) could post here what their requirements would be for getting bug_email.pl 

into a state that they would be willing to use it.  Then we can make new bugs for 

the individual things that need to be done to get it that way (easier to do it in 

small pieces) and treat this as a tracking bug.

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

<news://news.mozilla.org/[e-mail]> mentions some
issues with the bug_email.pl script.  I think we'd probably at least want to see
points 2 and 5 addressed.  Perhaps this bug should be considered a tracking bug
for those issues.

Comment 5

[Dawn Endico]

1. How does it handle vacation messages? Obviously we don't want
vacation message spam added to bug comments. Right now we have 
at least a half dozen big bug owners on 6 week vacations.

2. This seems like it will promote bad citizenship. Replying via email means
you're replying to the last comment and haven't read the entire bug.

3. I'm against turning on automatic account creation via email.

4. General uneasy feelings about inventing a bug input language. (handwave,
   handwave). I guess this is similar problem to what importxml.pl does
   except that the syntax for xml input is well defined (see
   http://bugzilla.mozilla.org/bugzilla.dtd). What's the syntax for
   submitting bugs?

Comment 6

[Tara Hernandez]

Dawn--going ahead and reassigning to you.

Comment 7

[David Krause]

*** Bug 46998 has been marked as a duplicate of this bug. ***

Comment 8

[David Krause]

I agree with endico on this one.  Vacation programs and auto-responders are
going to be a big problem and it is very difficult to filter out these type of
messages from being added to the database.  Also, I agree people may respond to
a message before reading all the following messages.  I've experienced that a
lot on mailing lists where people reply before reading the rest of the thread.

This would be a cool thing to have but I think it could turn out very difficult
to implement correctly.

Comment 9

[Ben Bucksch (:BenB)]

Dawn, re 1. and 2.:

I haven't figured out a concrete implementation architecture, but this is no
problem in newgroups/mailinglists either (1.), or it is tolerated (2.). We might
actually be able to leverage this same code by accepting email comment via
mailinglist software and processing it after it only ran through that
mailing-list software or so. Anyway, doesn't seem unsolveable to me - it is just
like mailing-lists.

> I'm against turning on automatic account creation via email.

Same as below.

> What's the syntax for submitting bugs?

I see that as an optional enhancement. Most of my comments are without changes
to the field. Cover 70% first.

Comment 10

[David Krause]

Another thing I just thought about.  How would authentication work?  Because I
could possibly:

Send an email back to [e-mail] with a faked from email address and a
faked bugzilla change spam quoted in the body along with whatever changes I
wanted to make.

Comment 11

[Ben Bucksch (:BenB)]

David, we sdiscusse dthis already in the ng thread, iirc. Basically, you need
crypto or hacks.

Comment 12

[Dawn Endico]

note that this bug is a request for bugzilla.mozilla.org to turn on
bugzilla's feature that lets people reply to bugzilla spam. I'm
marking this wontfix. I don't want to turn this on for the reasons
noted below. (authentication and spam)

Some day when the code is more mature we can look at this again.

Comment 13

[Ben Bucksch (:BenB)]

endico, no, this bug is also about implementing those parts of the feature that
would be necessary to enable it in bugzilla.mozilla.org. REOPENing.

From inital description:
> Please fix the remaining problems and enable this for Mozilla's bugzilla.

Please explain what you mean with "spam". Bugzilla cannot be spammed (in the
traditional way), because we would require authentication.
Replying without having read all of the comments is a problem usenet and
mailing-lists lived for years with, and well - I don't see that as a reason not
to activate this feature.

Comment 14

[Dawn Endico]

ben, the previous comments in this bug explain what spam is. This bug was
assigned to me because people wanted the current bugzilla reply option
turned on. I'm not turning it on because of the problems noted above. As
far as fixing the feature goes, I dislike this whole idea and think its
doomed to failure. I'm not going to work on it. Reassigning back to Tara
and updating summary although this should probably be assigned to the person
who wrote that feature in the first place.

Comment 15

[Christopher Aillon (sabbatical, not receiving bugmail)]

*** Bug 97059 has been marked as a duplicate of this bug. ***

Comment 16

[Christopher Aillon (sabbatical, not receiving bugmail)]

Moving to new product Bugzilla, component Creating/Changing Bugs

Comment 17

[Matthew Cline]

For both security and ignoring auto-responders, how about this:

Concatenate the bug number with the user's password (or password
hash, however bugzilla does it) and make a message digest of that (say
with MD5); put the digest at the beginning of the message.  The
response must have that digest in the message, and the line in which
it is contained will be deleted (in case it's quoted with a "> " or
something).  If some script kiddie is able to get their hands on a
bugzilla spam for bug 1234 to [e-mail], then he'll be able to send
forged messages from [e-mail] for bug 1234, but only for that
combination.

To take care of auto-responders, a line saying "remove this line to
respond" would be put in right after the digest, and the user would
have to manually remove it.  Any auto-responders that included the
entire text of the original message would include that as well,
causing bugzilla to ignore it.

Comment 18

[Sebastian Biallas]

*** Bug 172399 has been marked as a duplicate of this bug. ***

Comment 19

[Wayne Woods]

Is this bug still in consideration? :) I was only just made aware of it, but I
see it hasn't had any serious discussion since 2002.

Lately I've had a number of bug reporters try to reply to my comments by
replying directly to the bugzilla-daemon email. Unfortunately their mail gets
swallowed up. I have no idea that they said anything and they have no idea that
their message wasn't posted. While it's not going to crash any systems, I'd
consider this a pretty horrible problem.

cheers

Wayne

Comment 20

[Ben Bucksch (:BenB)]

Mail to <[e-mail]> used to bounce. It seems this is no longer
the case, and it's a bug of the mozilla.org installation. I filed bug 236087
about that.

Comment 21

[Wayne Woods]

Great, thanks Ben :)

Comment 22

[Frédéric Buclin]

(In reply to comment #12)
> note that this bug is a request for bugzilla.mozilla.org to turn on
> bugzilla's feature that lets people reply to bugzilla spam.

So this bug is in the wrong product -> mozilla.org

Comment 23

[Ben Bucksch (:BenB)]

I don't know how this feature is implemented in bugzilla, but an idea for authentication: bugzilla could include a unique value in the notification email, and only if the email reply contains that, it's valid. The value could be a hash of user+date+server secret, or somthing like that. The value could be in the body or in the message-ID, which can be generated by bugzilla and is free-form (apart from hostname), and which email clients quote in References: and/or In-Reply-To: headers of the reply, so bugzilla can check there.

Comment 24

[Frédéric Buclin]

(In reply to comment #23)
> I don't know how this feature is implemented in bugzilla, but an idea for
> authentication: bugzilla could include a unique value in the notification
> email, and only if the email reply contains that, it's valid. The value could
> be a hash of user+date+server secret, or somthing like that. The value could be
> in the body or in the message-ID

You read my minds, see bug 419203. :)

Comment 26

[:glob ✱]

as we have the user's public keys via securemail, we should just require signed email in the email-in interface.

Comment 27

[Jonathan Watt [:jwatt]]

We were just talking about using signed email for this at the rendering work week in paris. In the session on bug triage and general bugzilla productivity this was the top item that we thought would help. As well as skipping bmo loading and submit slowness, replying to email is just a much faster workflow anyway.

Comment 28

[:glob ✱]

in order to better deal with stray comments ("billy is out of the office"), i'd like comment tagging to land before this.

Comment 30

[:glob ✱]

we have a plan for finally allowing users to comment on bugs by replying to bugmail:

  https://wiki.mozilla.org/BMO/inbound_email

as our requirements are different from upstream this will be developed as an extension rather than using bugzilla's native implementation.


security: can you please review the planned use of tokens to verify the server.
details are on the wiki page, but in summary:

- limited to adding comments to existing bugs only (no attachments, no editing, no bug creation)
- limited to users with editbugs
- not available to confidential bugs which trigger secure-mail encryption

- create a token which links with the message-id
- build a reply-to address containing the token
- emails sent to this address must have a "in-reply-to" header matching the associated message-id
- tokens are single-use, and expire after 7 days

Comment 31

[Ben Bucksch (:BenB)]

> the reply-to header is set to <reply_$[e-mail]>

This is going to mess up processing in email software. Email software collects email addresses I sent email to, so it will collect a junk address each time I use the "reply to bugmail" feature and fill up the store (which also autocompletes, so will be annoying).

Given that your inbound scheme insists on the "In-Reply-To:" header to match, you require this to work in any case. Why not use only this? I don't see an advantage in insisting on both the header and the To address. Just use the In-Reply-To: header (and maybe "References:" header).

The token could even be cryptographically signed, so it should be sufficient.

Comment 32

[:glob ✱]

(In reply to Ben Bucksch (:BenB) from comment #31)
> > the reply-to header is set to <reply_$[e-mail]>
> 
> This is going to mess up processing in email software. Email software
> collects email addresses I sent email to, so it will collect a junk address
> each time I use the "reply to bugmail" feature and fill up the store (which
> also autocompletes, so will be annoying).

this is a common practice and is already used by github, facebook, etc, so you are likely already experiencing this issue.

> Given that your inbound scheme insists on the "In-Reply-To:" header to match, you require this
> to work in any case. Why not use only this?

the message-id that bugzilla generates when a bug is created is generated from the bug data (bug-$bug_id-$user_id@..).  bugmail relating to updates to bugs set the "in-reply-to" and "references" headers to this message-id.

if we were to store tokens as part of the message-id, we'd have to keep track of all tokens on a per-user base, even if that token has been used, in order to generate correct in-reply-to/references headers.  this isn't a practical solution.

Comment 33

[Ben Bucksch (:BenB)]

> this is a common practice

That's never been an argument for me. If it's broken, it's broken, no matter how many sites do it.
Fact is, this *will* mess up collected addresses.

> if we were to store tokens as part of the message-id

You'd put in the same that you now put into the reply-to address. Same work, just a different place.

Comment 34

[:glob ✱]

(In reply to Ben Bucksch (:BenB) from comment #33)
> > if we were to store tokens as part of the message-id
> 
> You'd put in the same that you now put into the reply-to address. Same work,
> just a different place.

what we will implement will result in messages with:

new bugs:
   message-id: bug-$bug_id-$[e-mail]
     reply-to: reply_$[e-mail]

bug updates:
   message-id: bug-$bug_id-$user_id-$[e-mail]
  in-reply-to: bug-$bug_id-$[e-mail]
     reply-to: reply_$[e-mail]

are you proposing:
   message-id: reply_$[e-mail]
     reply-to: [e-mail]

unfortunately this won't work because in order to generate correct in-reply-to headers for *subsequent* messages we'd need to store the original message's message-id, for each recipient, forever.

if i've miss-understood your proposal can you please lay it out in with more details?

> That's never been an argument for me. If it's broken, it's broken, no matter
> how many sites do it. Fact is, this *will* mess up collected addresses.

that's a fair point, however we are unfortunately limited with regards to fields that we're able to store this information.

Comment 35

[Ben Bucksch (:BenB)]

> are you proposing:
>    message-id: reply_$[e-mail]
>      reply-to: [e-mail]

Yes.

> in order to generate correct in-reply-to headers for *subsequent* messages

You mean to allow threading (using in-reply-to) of each bug, one thread per bug? I assume we need flat threads, not ever increasing thread depth.

You can do that with:
  In-Reply-To: bug-$[e-mail]
   message-id: reply_$[e-mail]
     reply-to: [e-mail]
whereby In-Reply-To is always the same value for all notifications for a given bug. Reasonable email software (all that I know) will thread it together, even if it never saw a message with this ID, because it happens all the time in practice that I get CCed on a thread I haven't been on from the start.

BTW (I don't think we need it, but it might be good for you to know): The In-Reply-To and References headers (both are essentially the same) can hold several message IDs, the whole chain of parent messages up to the thread starter. And a message ID that is not in the store will be ignored by all mailers, because it can always happen that the user doesn't see one of the messages in the threads, even if it's one of the mails in-between. You might achieve what you want by simply passing two "parent" message-IDs: The one with the token, and a static one (always the same for this ticket).

http://www.faqs.org/rfcs/rfc2822.html RFC 2822 Section 3.6.4 Identification fields
http://www.jwz.org/doc/threading.html
http://cr.yp.to/immhf/thread.html

Comment 36

[:glob ✱]

thanks for the clarification ben.  i agree with you that using the message-id/in-reply-to fields to store the token is better than the reply-to address.

in order for threading to work we'll have to set the in-reply-to/references headers to <bug-$bug_id-$user_id@$url_base> even for notifications of new bugs (which currently do not set threading headers).

i've updated the wiki to reflect these changes.

Comment 37

[Ben Bucksch (:BenB)]

Thanks, glob!
+1 for constructive discussion and cooperation :)

Comment 38

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

So interesting corner case. Bug is public, a comment goes out and I get the mail. The bug is then changed to a confidential/security bug. The token for the email I get is not yet expired and I reply via email. What will happen? Will the comment succeed or will the mail bounce or...?

We're also assuming that the sent mail goes through the same sanitation routines that a normal bugzilla comment does.

For the most part I think this sounds sensible, but we should also look over the code just to be sure.

Comment 39

[:glob ✱]

(In reply to Curtis Koenig [:curtisk] from comment #38)
> So interesting corner case. Bug is public, a comment goes out and I get the
> mail. The bug is then changed to a confidential/security bug. The token for
> the email I get is not yet expired and I reply via email. What will happen?
> Will the comment succeed or will the mail bounce or...?

the current plan is to allow replying to comments for any bug unless they are secured by a group that has secure-mail (ie. email encryption) enabled.

so if you reply to a bug which is then put into the moco-employee group (which doesn't have securemail), then the comment will be added to the bug.

as per https://wiki.mozilla.org/BMO/inbound_email#inbound if you reply to a bug which is then put into a securemail enabled group, your comment will be rejected and the email will bounce.

> We're also assuming that the sent mail goes through the same sanitation
> routines that a normal bugzilla comment does.

yes; the code just adjusts the headers on normal outbound email, and uses the normal bug objects to add comments.

Comment 41

[Lu Wei]

If Bugzilla could be participated by 3 forms: web, mail, newsgroup, like Gmane... Maybe too luxurious.

Comment 42

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Because of the issues with responding to security bugs over email, I'm shelving this.