Open Bug 44393 Opened 20 years ago Updated 29 days ago

develop an extension that provides comment-only inbound email support

Categories

(bugzilla.mozilla.org :: General, task, P3)

Tracking

()

People

(Reporter: BenB, Unassigned)

References

()

Details

(Keywords: bmo-big)

From bug 6419:
------- Additional Comments From Dave Miller 2000-05-20 09:54 -------
Something to point out is that with bug_email.pl as far along as it is now (we 
can already submit via email and add comments), this would make queries via
email easy to implement, too.

------- Additional Comments From Dave Miller 2000-06-30 18:36 -------
[...]
Replying to the bugzilla spam to add comments is already possible (but doesn't 
really have much to do with this particular bug).  Look at bug_email.pl and 
appendbugmail.pl (or whatever their real names are) and the associated README in 
the contrib/ directory.

------- Additional Comments From Dave Miller 2000-06-30 19:09 -------
Ah, that's right...  mozilla.org doesn't use it.  I think this is because the 
error messages and confirmation notices it sends out still leave a lot to be 
desired, and they'd tend to scare people off.

---
Please fix the remaining problems and enable this for Mozilla's bugzilla. Email
apps are so much more comfortable (larger window, save sa draft etc.) than a
<textarea> and also fast, because I don't have to click on the link, scroll down
etc..

BTW: How did you solve the authentication problem?
Severity: normal → enhancement
Authentication problem?  Hmm, yeah, that's a good one.  It's easy to spoof an 
email address.  I haven't fixed it on my system other than make the incoming 
email for new bugs go to a pending product with restricted visibility, then 
moving them where they go after they come in.  Easier on the users, more work for 
us.  Well, we could always make users register to be able to submit stuff via 
email, and make them upload their public PGP key to their user profile.  They 
would then have to sign their emails for them to be accepted.
Yep, that was my idea as well. But you should support both PGP and S/MIME, if
possible.
I should point out this is a Mozilla.org-specific bug...  Bugzilla in general can 

already do what this bug asks for, Mozilla.org just doesn't have the option 

enabled right now.



I'd ask if the people responsible for bugzilla.mozilla.org (I just added them to 

the CC) could post here what their requirements would be for getting bug_email.pl 

into a state that they would be willing to use it.  Then we can make new bugs for 

the individual things that need to be done to get it that way (easier to do it in 

small pieces) and treat this as a tracking bug.

<news://news.mozilla.org/00071714011602.14256@jarjar.imall.com> mentions some
issues with the bug_email.pl script.  I think we'd probably at least want to see
points 2 and 5 addressed.  Perhaps this bug should be considered a tracking bug
for those issues.
1. How does it handle vacation messages? Obviously we don't want
vacation message spam added to bug comments. Right now we have 
at least a half dozen big bug owners on 6 week vacations.

2. This seems like it will promote bad citizenship. Replying via email means
you're replying to the last comment and haven't read the entire bug.

3. I'm against turning on automatic account creation via email.

4. General uneasy feelings about inventing a bug input language. (handwave,
   handwave). I guess this is similar problem to what importxml.pl does
   except that the syntax for xml input is well defined (see
   http://bugzilla.mozilla.org/bugzilla.dtd). What's the syntax for
   submitting bugs?
Dawn--going ahead and reassigning to you.
Assignee: tara → endico
*** Bug 46998 has been marked as a duplicate of this bug. ***
I agree with endico on this one.  Vacation programs and auto-responders are
going to be a big problem and it is very difficult to filter out these type of
messages from being added to the database.  Also, I agree people may respond to
a message before reading all the following messages.  I've experienced that a
lot on mailing lists where people reply before reading the rest of the thread.

This would be a cool thing to have but I think it could turn out very difficult
to implement correctly.
Dawn, re 1. and 2.:

I haven't figured out a concrete implementation architecture, but this is no
problem in newgroups/mailinglists either (1.), or it is tolerated (2.). We might
actually be able to leverage this same code by accepting email comment via
mailinglist software and processing it after it only ran through that
mailing-list software or so. Anyway, doesn't seem unsolveable to me - it is just
like mailing-lists.

> I'm against turning on automatic account creation via email.

Same as below.

> What's the syntax for submitting bugs?

I see that as an optional enhancement. Most of my comments are without changes
to the field. Cover 70% first.
Another thing I just thought about.  How would authentication work?  Because I
could possibly:

Send an email back to bugzilla@mozilla.org with a faked from email address and a
faked bugzilla change spam quoted in the body along with whatever changes I
wanted to make.
David, we sdiscusse dthis already in the ng thread, iirc. Basically, you need
crypto or hacks.
note that this bug is a request for bugzilla.mozilla.org to turn on
bugzilla's feature that lets people reply to bugzilla spam. I'm
marking this wontfix. I don't want to turn this on for the reasons
noted below. (authentication and spam)

Some day when the code is more mature we can look at this again.
Status: NEW → RESOLVED
Closed: 20 years ago
Resolution: --- → WONTFIX
endico, no, this bug is also about implementing those parts of the feature that
would be necessary to enable it in bugzilla.mozilla.org. REOPENing.

From inital description:
> Please fix the remaining problems and enable this for Mozilla's bugzilla.

Please explain what you mean with "spam". Bugzilla cannot be spammed (in the
traditional way), because we would require authentication.
Replying without having read all of the comments is a problem usenet and
mailing-lists lived for years with, and well - I don't see that as a reason not
to activate this feature.
Status: RESOLVED → REOPENED
Resolution: WONTFIX → ---
ben, the previous comments in this bug explain what spam is. This bug was
assigned to me because people wanted the current bugzilla reply option
turned on. I'm not turning it on because of the problems noted above. As
far as fixing the feature goes, I dislike this whole idea and think its
doomed to failure. I'm not going to work on it. Reassigning back to Tara
and updating summary although this should probably be assigned to the person
who wrote that feature in the first place.
Assignee: endico → tara
Status: REOPENED → NEW
Summary: Make replies to bugzilla spam possible → improve replies to bugzilla spam feature
Target Milestone: --- → Future
*** Bug 97059 has been marked as a duplicate of this bug. ***
Moving to new product Bugzilla, component Creating/Changing Bugs
Assignee: tara → myk
Component: Bugzilla → Creating/Changing Bugs
Product: Webtools → Bugzilla
Version: other → unspecified
For both security and ignoring auto-responders, how about this:

Concatenate the bug number with the user's password (or password
hash, however bugzilla does it) and make a message digest of that (say
with MD5); put the digest at the beginning of the message.  The
response must have that digest in the message, and the line in which
it is contained will be deleted (in case it's quoted with a "> " or
something).  If some script kiddie is able to get their hands on a
bugzilla spam for bug 1234 to john@foobar.com, then he'll be able to send
forged messages from john@foobar.com for bug 1234, but only for that
combination.

To take care of auto-responders, a line saying "remove this line to
respond" would be put in right after the digest, and the user would
have to manually remove it.  Any auto-responders that included the
entire text of the original message would include that as well,
causing bugzilla to ignore it.

*** Bug 172399 has been marked as a duplicate of this bug. ***
Is this bug still in consideration? :) I was only just made aware of it, but I
see it hasn't had any serious discussion since 2002.

Lately I've had a number of bug reporters try to reply to my comments by
replying directly to the bugzilla-daemon email. Unfortunately their mail gets
swallowed up. I have no idea that they said anything and they have no idea that
their message wasn't posted. While it's not going to crash any systems, I'd
consider this a pretty horrible problem.

cheers

Wayne
Mail to <bugzilla-daemon@mozilla.org> used to bounce. It seems this is no longer
the case, and it's a bug of the mozilla.org installation. I filed bug 236087
about that.
Great, thanks Ben :)
QA Contact: mattyt-bugzilla → default-qa
Target Milestone: Future → ---
Assignee: myk → create-and-change
(In reply to comment #12)
> note that this bug is a request for bugzilla.mozilla.org to turn on
> bugzilla's feature that lets people reply to bugzilla spam.

So this bug is in the wrong product -> mozilla.org
Assignee: create-and-change → nobody
Component: Creating/Changing Bugs → Bugzilla: Other b.m.o Issues
Product: Bugzilla → mozilla.org
QA Contact: default-qa → other-bmo-issues
Version: unspecified → other
I don't know how this feature is implemented in bugzilla, but an idea for authentication: bugzilla could include a unique value in the notification email, and only if the email reply contains that, it's valid. The value could be a hash of user+date+server secret, or somthing like that. The value could be in the body or in the message-ID, which can be generated by bugzilla and is free-form (apart from hostname), and which email clients quote in References: and/or In-Reply-To: headers of the reply, so bugzilla can check there.
Summary: improve replies to bugzilla spam feature → Enable email interface on b.m.o.
(In reply to comment #23)
> I don't know how this feature is implemented in bugzilla, but an idea for
> authentication: bugzilla could include a unique value in the notification
> email, and only if the email reply contains that, it's valid. The value could
> be a hash of user+date+server secret, or somthing like that. The value could be
> in the body or in the message-ID

You read my minds, see bug 419203. :)
Whiteboard: [bmo-post-4.0]
Component: Bugzilla: Other b.m.o Issues → General
Product: mozilla.org → bugzilla.mozilla.org
Duplicate of this bug: 769697
as we have the user's public keys via securemail, we should just require signed email in the email-in interface.
We were just talking about using signed email for this at the rendering work week in paris. In the session on bug triage and general bugzilla productivity this was the top item that we thought would help. As well as skipping bmo loading and submit slowness, replying to email is just a much faster workflow anyway.
in order to better deal with stray comments ("billy is out of the office"), i'd like comment tagging to land before this.
Depends on: 942725
No longer depends on: 793963
we have a plan for finally allowing users to comment on bugs by replying to bugmail:

  https://wiki.mozilla.org/BMO/inbound_email

as our requirements are different from upstream this will be developed as an extension rather than using bugzilla's native implementation.


security: can you please review the planned use of tokens to verify the server.
details are on the wiki page, but in summary:

- limited to adding comments to existing bugs only (no attachments, no editing, no bug creation)
- limited to users with editbugs
- not available to confidential bugs which trigger secure-mail encryption

- create a token which links with the message-id
- build a reply-to address containing the token
- emails sent to this address must have a "in-reply-to" header matching the associated message-id
- tokens are single-use, and expire after 7 days
Assignee: nobody → glob
Status: NEW → ASSIGNED
Flags: sec-review?
Keywords: bmo-big
Priority: P3 → P2
Summary: Enable email interface on b.m.o. → develop an extension that provides comment-only inbound email support
> the reply-to header is set to <reply_$token@reply.bugzilla.org>

This is going to mess up processing in email software. Email software collects email addresses I sent email to, so it will collect a junk address each time I use the "reply to bugmail" feature and fill up the store (which also autocompletes, so will be annoying).

Given that your inbound scheme insists on the "In-Reply-To:" header to match, you require this to work in any case. Why not use only this? I don't see an advantage in insisting on both the header and the To address. Just use the In-Reply-To: header (and maybe "References:" header).

The token could even be cryptographically signed, so it should be sufficient.
(In reply to Ben Bucksch (:BenB) from comment #31)
> > the reply-to header is set to <reply_$token@reply.bugzilla.org>
> 
> This is going to mess up processing in email software. Email software
> collects email addresses I sent email to, so it will collect a junk address
> each time I use the "reply to bugmail" feature and fill up the store (which
> also autocompletes, so will be annoying).

this is a common practice and is already used by github, facebook, etc, so you are likely already experiencing this issue.

> Given that your inbound scheme insists on the "In-Reply-To:" header to match, you require this
> to work in any case. Why not use only this?

the message-id that bugzilla generates when a bug is created is generated from the bug data (bug-$bug_id-$user_id@..).  bugmail relating to updates to bugs set the "in-reply-to" and "references" headers to this message-id.

if we were to store tokens as part of the message-id, we'd have to keep track of all tokens on a per-user base, even if that token has been used, in order to generate correct in-reply-to/references headers.  this isn't a practical solution.
> this is a common practice

That's never been an argument for me. If it's broken, it's broken, no matter how many sites do it.
Fact is, this *will* mess up collected addresses.

> if we were to store tokens as part of the message-id

You'd put in the same that you now put into the reply-to address. Same work, just a different place.
(In reply to Ben Bucksch (:BenB) from comment #33)
> > if we were to store tokens as part of the message-id
> 
> You'd put in the same that you now put into the reply-to address. Same work,
> just a different place.

what we will implement will result in messages with:

new bugs:
   message-id: bug-$bug_id-$user_id@https.bugzilla.mozilla.org
     reply-to: reply_$token@bugzilla.mozilla.org

bug updates:
   message-id: bug-$bug_id-$user_id-$random@https.bugzilla.mozilla.org
  in-reply-to: bug-$bug_id-$user_id@https.bugzilla.mozilla.org
     reply-to: reply_$token@bugzilla.mozilla.org

are you proposing:
   message-id: reply_$token@https.bugzilla.mozilla.org
     reply-to: reply@bugzilla.mozilla.org

unfortunately this won't work because in order to generate correct in-reply-to headers for *subsequent* messages we'd need to store the original message's message-id, for each recipient, forever.

if i've miss-understood your proposal can you please lay it out in with more details?

> That's never been an argument for me. If it's broken, it's broken, no matter
> how many sites do it. Fact is, this *will* mess up collected addresses.

that's a fair point, however we are unfortunately limited with regards to fields that we're able to store this information.
> are you proposing:
>    message-id: reply_$token@https.bugzilla.mozilla.org
>      reply-to: reply@bugzilla.mozilla.org

Yes.

> in order to generate correct in-reply-to headers for *subsequent* messages

You mean to allow threading (using in-reply-to) of each bug, one thread per bug? I assume we need flat threads, not ever increasing thread depth.

You can do that with:
  In-Reply-To: bug-$bug_id@https.bugzilla.mozilla.org
   message-id: reply_$token@https.bugzilla.mozilla.org
     reply-to: reply@bugzilla.mozilla.org
whereby In-Reply-To is always the same value for all notifications for a given bug. Reasonable email software (all that I know) will thread it together, even if it never saw a message with this ID, because it happens all the time in practice that I get CCed on a thread I haven't been on from the start.

BTW (I don't think we need it, but it might be good for you to know): The In-Reply-To and References headers (both are essentially the same) can hold several message IDs, the whole chain of parent messages up to the thread starter. And a message ID that is not in the store will be ignored by all mailers, because it can always happen that the user doesn't see one of the messages in the threads, even if it's one of the mails in-between. You might achieve what you want by simply passing two "parent" message-IDs: The one with the token, and a static one (always the same for this ticket).

http://www.faqs.org/rfcs/rfc2822.html RFC 2822 Section 3.6.4 Identification fields
http://www.jwz.org/doc/threading.html
http://cr.yp.to/immhf/thread.html
thanks for the clarification ben.  i agree with you that using the message-id/in-reply-to fields to store the token is better than the reply-to address.

in order for threading to work we'll have to set the in-reply-to/references headers to <bug-$bug_id-$user_id@$url_base> even for notifications of new bugs (which currently do not set threading headers).

i've updated the wiki to reflect these changes.
Thanks, glob!
+1 for constructive discussion and cooperation :)
So interesting corner case. Bug is public, a comment goes out and I get the mail. The bug is then changed to a confidential/security bug. The token for the email I get is not yet expired and I reply via email. What will happen? Will the comment succeed or will the mail bounce or...?

We're also assuming that the sent mail goes through the same sanitation routines that a normal bugzilla comment does.

For the most part I think this sounds sensible, but we should also look over the code just to be sure.
Flags: needinfo?(glob)
(In reply to Curtis Koenig [:curtisk] from comment #38)
> So interesting corner case. Bug is public, a comment goes out and I get the
> mail. The bug is then changed to a confidential/security bug. The token for
> the email I get is not yet expired and I reply via email. What will happen?
> Will the comment succeed or will the mail bounce or...?

the current plan is to allow replying to comments for any bug unless they are secured by a group that has secure-mail (ie. email encryption) enabled.

so if you reply to a bug which is then put into the moco-employee group (which doesn't have securemail), then the comment will be added to the bug.

as per https://wiki.mozilla.org/BMO/inbound_email#inbound if you reply to a bug which is then put into a securemail enabled group, your comment will be rejected and the email will bounce.

> We're also assuming that the sent mail goes through the same sanitation
> routines that a normal bugzilla comment does.

yes; the code just adjusts the headers on normal outbound email, and uses the normal bug objects to add comments.
Flags: needinfo?(glob)
Flags: sec-review? → sec-review?(dveditz)
Duplicate of this bug: 1120142
Assignee: glob → nobody
Status: ASSIGNED → NEW
Priority: P2 → --
Priority: -- → P3
Flags: sec-review?(dveditz)
If Bugzilla could be participated by 3 forms: web, mail, newsgroup, like Gmane... Maybe too luxurious.
Type: enhancement → task
You need to log in before you can comment on or make changes to this bug.