94850 - (bz-email-in) Integrate bug_email into the main code. (inbound submission and reply)

Status: RESOLVED FIXED

(Bugzilla :: Incoming Email, enhancement, P1)

Description

[Matthew Tuck [:CodeMachine]]

We need to clean up bug_email and move it from contrib to the main code.

Comment 1

[Tobias Burnus]

Besides the feature presently possible. I'd like to see:
a) Allow product/component selection based on the "To:" address.
   (We have only one product and a few compontents a
    support-unix@ and support-win@ is easier than bugs@ + @product one +   
    @component unix
b) Allow the automatical addition of the user to the database (it the account 
   doesn't exist yet). This would provide a simpler access for in-house support
   and would be like Debian's bug tracking system (minus "reportbug")
c) It should be possible to separate mailserver and webserver.
   .procmailrc is run on the mailserver and
   ConnectToDatabase(); has to be run on the bugzilla-webserver.

Comment 2

[Dave Miller [:justdave]]

This bug is listed on the 2.18 goals list.  This is now a meta bug for all
issues required to get bug_email into a condition that we can officially support
it.  If anyone knows of any other bugs related to bug_email which didn't get set
as dependencies here, please add them.

Comment 3

[Bradley Baetz (:bbaetz)]

The fix for this bug involves ignoring whats there entirely, and instead moving
the backend for bug creation/modification out of post_bug.cgi, and into a
backend module which the procmail script shares.

Comment 4

[Klaas Freitag]

Attachment: Proposal Mail Interface 2

This is a very alpha version of my proposal of a new bugzilla mail interface to
create bugs from mail- or other input. It is independant from the Bugzilla
server (needs not to run on the same computer) and generates a http-request to
post_bug.cgi with the input data. The script is not limited to mail input.
See README in the attachment tarball for few further notes.

Comment 5

[Joel Peshkin]

The approach of running the mail parser on another machine and communicating by
POST is appealing, though I am not sure how you are handling authentication.

Before you go too far with this, I am not sure if the existing form variables
are the best way to seperate the front end and back end.  Should this be
something done by POSTing an XML description of the bug?

Comment 6

[Matthew Barnson]

Since this hasn't been touched since September, I'd like to take a stab at
owning it for a couple of weeks and see what I can do with it.  I planned on
updating the bug email documentation anyway, and in doing so discovered how
unmaintainable existing inbound mail handling was.  

Comment 7

[Dave Miller [:justdave]]

note all the dependencies as well :)

Comment 8

[Klaas Freitag]

Well, I am still here to help with a new version of the mail interface :) I have
the new version running in production now. Unfortunately it can only create bugs
and append comments and there is still much room for improvement. Please drop me
a note if you want me to attach the latest tarball ;-)

I fully agree that it would be a much better way to send xml description of the
bug and not to form a post request filling formular data. But in 2.16, there is
no such XML backend. Is it here in 2.17.x.? This is the reason why there is no
bug manipulation in my mail interface yet.

For authentication, the bugzilla password must send as a tag in the mail that
creates a bug. Thats bad, of course, but not as bad as having no authentication.
A future extension could use PGP and signed mails.

Comment 9

[Gervase Markham [:gerv]]

No longer a 2.18 blocker.

Gerv

Comment 10

[Brian G. Peterson]

Attachment: bug_email.pl patched to work on Bugzilla 2.17.4

Comment 11

[Brian G. Peterson]

Attachment: bugzilla_email_append.pl patched to work on Bugzilla 2.17.4

Comment 12

[Brian G. Peterson]

I recently upgraded my bugzilla install at work to Bugzilla 2.17.4.

We immediately noticed that bug_email and bugzilla_email_append were broken by 
the integration of the venerable processmail.pl into the main code.

We fixed bug_email and bugzilla_email_append to work with the corified 
processmail functions.

I've attached the patched versions of the code.  Hopefully somebody with CVS 
acess can test them and put them in contrib, since it seems that Gerv is 
abandoning this bug once again.

Comment 13

[Christian Reis]

Brian, can I ask you to file a new bug on updating them to the trunk and
attaching cvs diffs of the files? 

This bug is a tracker, and shouldn't be used to house any specific development.

Comment 14

[Dave Miller [:justdave]]

Brian: Gerv doesn't own this bug, so he's not abandoning it.  Matt obviously
abandoned it a long time ago though, so changing the owner to reflect reality.

There's already FIXED bugs for the processmail changes.  It was fixed in 2.17.5.
 (see bug 192512)

Comment 15

[Joel Peshkin]

This really needs a full rewrite....

What I'd like to see is something that does the following...

1) Accepts mail in text/plain or multipart/alternative with a text/plain section.

2) Configurable to permit new bugs to be assigned to products/components by
either   
   a) Mapping the email address to a command-line for bug_email.pl 
           (e.g.   BugzillaBugs: "| bug_email.pl Bugzilla Bugzilla-General"
             or    BugzillaBugs: "| bug_email.pl Bugzilla --default
Bugzilla-General" )
   b) Parsing the subject line
        Subject: [Bugzilla-General] Integrate bug_email into the main code
   c) Parsing the body
        Component: Bugzilla-General

3) Permit mailed-in bugs to be assigned to a default owner for classification
   a) Either all incoming bugs or only those incompletely specified by parsing
   b) It should also be possible to have ambiguous/incomplete mails forwarded by
email to a default owner for repair.  The default owner should be able to
forward the fix message back to bug_email and have it processed as an action by
the original author rather than the person who fixes the message.

4) Permit bugs to be updated by email following mailto: links in bugmail
     (e.g.  Subject: [bug 94850] ....)
   a) Comments should simply be appended.  
   b) Alternative HTML representations should be ignored.
   c) Attachments should be attached 
      i) Needs protection against silly attachments like Lotus Notes PCX files,
probably by requiring that commands be placed in the text of the mail. 
     (e.g.   ATTACH: filename ) - otherwise the file is ignored.

  d) Commands should include 
       REASSIGN: user-email
       RESOLVE: FIXED
       ACCEPT:
       DUPLICATE: number

5) All mail-in actions should be confirmed by email automatically

6) In order to make this viable, outbound bugmail should be easily customizable
to indicate the group permissions of a bug. (So users don't accidentally mail-in
 would-be private comments on public bugs)

7) Private comment mechanisms need to be relatively goof-proof.


 

Comment 16

[Klaas Freitag]

I think the points mentioned here can be fullfilled with my proposal for the 
mail interface 2 in comment #4. I am willing to help if you agree. 

Comment 17

[Joel Peshkin]

I think some early decisions are needed on the following questions....

1) How should the script interact with the back-end?  My suspicion is that the
right thing to do is to POST the entire SMTP message and let all the parsing
happen on the server where the database is accessible. This would minimize
issued of syncing form variables and solve many problems with permissions.

2) Should we always take the word of the user in the email From: field about the
user's identity or is more validation needed?

Comment 18

[Dave Miller [:justdave]]

I think a user preference for how mail from them is handled would be in order.

When a new user signs up, the system would default to not allowing mail from
them.  In their preferences is three (or more?) choices for how to deal with
mail received with their address on the From: line:
1) don't allow mail from me (default)
2) allow mail from me and trust the From: line
3) allow mail from me and require it to be GPG-signed with this key: (textbox)

Another possibility is temporarily stashing received email somewhere and issuing
a token for it, which is mailed back to the user to confirm (this is what most
mailing list systems do)

Comment 19

[Joel Peshkin]

(In reply to comment #16)
> I think the points mentioned here can be fullfilled with my proposal for the 
> mail interface 2 in comment #4. I am willing to help if you agree. 

We'd love the help, though I think some further refinement is needed on the
proposal in comment 4.  

We certainly want to be able to create and alter bugs by email.  These changes
must be handled under the bugzilla account appropriate to the user.  Using a
password in the email would be a pretty poor security approach.  The suggestions
justdave made in comment 18 are a good start, though I'd like to come up with a
good way to validate users who would never figure out GPG.  One alternative
would be to permit justdave's option 2 require that the mail come from certain
hosts.  Anyone have a better idea??  So far, justdave and I have come up with a
whole array of partial solutions.   A real solution would be handy :-)

Comment 20

[Peter Kay]

I would like to add a suggestion to this:

Any implementation that handles this should have two parts:

Part 1:
Parse e-mail (in whatever form) into a hash containing all the information for
creating/updating the bug (I'm more concerned with creating them myself), call
Part 2 with this hash and handle results.

Part 2:
HTTP/hash API:  A funtion callable from anywhere that:
Gets a hash with information for creating/updating the bug.  Takes that hash,
forms the HTTP requests, sends them off, gets a result, parses a result, puts
together a new hash ({bug_number}=>..., etc) and returns that hash.

This way, when you have anyone who wants to, say, migrate a different bug
database to bugzilla, automatically add 80 bugs, etc, they don't have to send 80
e-mails, they can just write a script around this API, pass in a hash, and deal
with the results.

It shouldn't be too much trouble to seperate these functionality, and would make
extending it much easier.

Comment 21

[Dave Miller [:justdave]]

That sounds like Bug.pm, if only it could write out changes. :)

Comment 22

[Joel Peshkin]

Rumor has it that someone is working on this...  anyone?

Comment 23

[Erik Stambaugh]

Sorry abbout the essay, but here is what I see for this:

A lot of the bug manipulation routines need to be put in a module that makes
it easier for other parts of the program, like mail-in, to use.

Now, I think it should be considered beyond the scope of this bug to go
delving into Bug.pm and similar places to make everything go along with this.
It needs to be done, but I think a good shove in the right direction would be
to create duplicate functionality in a more generic library on an as-needed
basis, then work on migrating the old code over in another bug.

I'm a really big fan of trying to get these patches in in small steps, so I'd
like to keep the capabilities of an initial mail-in system to a minimum, while
leaving it wide open for expansion later.

A simple version of inbound mail would have a set of options defined
line-by-line, like:

Command: Search
Product: Bugzilla
Component: Bugzilla-General

...and so forth.  Optionally, a Command could be listed on the subject line,
along with a default option.  In Search's case, I would assume short_desc
would be the default field, so an email message coming in that has:

Subject: Search bug_email

...would search the Summary (short_desc) for "bug_email"

Various commands should be handled by their own modules.  It should probably   
just include anything that's in Bugzilla/Mail/Inbound.  Function refs for each
command are kept in a hash, keyed by command name.  We pass the options and
values to it as a hash ref.

Functions should also have access to all of the MIME portions of a message, 
along with any uuencoded data.

Inbound messages have to pass through some authorization tests, including host
mask (authorize any host on 192.168.0.0/16, if all relays occurred there),
confirmation key (unauthorized mail is sent back to its origin with a key
string inside, saying "please just respond to authorize").  Something 
involving GPG signatures or a persistent key would be nice in the future.

Help should allow you to request a template message for any command, with all  
possible options listed and brief explanations so a user may fill in the
blanks.

To start with, I would like to include very simple Help and New-Bug commands.

Internationalization kind of scares me, since the commands and options will 
probably want to live in a template somewhere, then be fed back into the code

Anyhow, due to the fact that I need this, I'm willing to get rolling on it,
unless I hear objections.

Comment 24

[Igor Jese]

Small steps, minimal functionality, separate modul that gradually takes over the 
work of spagetti --> this surely is the most reasonable (and by-the-book) path 
to go - couldn't agree more with Erik.

Regarding the interface though, this should be done very carefully. Utmost 
simplicity is what is wanted of posting a new bug. At least some advanced 
capabilities (like in quicksearch.html) are wanted from searching through bugs. 
Other commands like help, user-mgmt, product-mgmt are far less important and 
could probably wait for interface to stabilize a bit, or they could exist in 
some most basic form. Internationalization could probably also be postponed and 
added (or "refactored") later.

Of course, my oppinions are based on needs and experience of mine, my colleagues 
and bugzilla users on the same system. From this experience, e-mail interface is 
most badly needed when I finish session with end-user testers and want to enter 
30 bugs at once. Bugzilla via html interrupts me too much when I write all the 
data I could gather and try to comment as much as possible. So I write all in 
advance in text editor, and then *slowly* post problem descriptions and 
attachments. This process is very tedious and leeds to situation that sometimes 
people post bugs "in chunks": one bug that containes descriptions of 10 
associated bugs.

Sorry, I also got a little lengthy, just wanted to help.
Thanks, Erik.

Comment 25

[Dave Miller [:justdave]]

I like the idea of (optional) RFC2822-style headers within the body of the
message.  Everyone's used to it already from email, and it's also the way
Debian's mail interface works (which seems to be the most well-liked of the bug
trackers with inbound email support currently [at least by people who like to
work via email - everyone else hates it]).

Comment 26

[Erik Stambaugh]

OK, I'll make this bug my burden.  Not today, but soon, because it's something I
need.

I want to write a brand-new modular system for this, and it's probably going to
require a *lot* of params, so I'm putting bug 46296 (multi-panel editparams) on
as a dependency.  Otherwise the process of setting it up will just be ridiculous.

Comment 27

[Dave Miller [:justdave]]

Bugzilla 2.20 feature set is now frozen as of 15 Sept 2004.  Anything flagged
enhancement that hasn't already landed is being pushed out.  If this bug is
otherwise ready to land, we'll handle it on a case-by-case basis, please set the
blocking2.20 flag to '?' if you think it qualifies.

Comment 28

[Dave Miller [:justdave]]

Adding a few keywords to the summary to make it easier to find.

Comment 29

[Dave Miller [:justdave]]

*** Bug 304390 has been marked as a duplicate of this bug. ***

Comment 30

[Frédéric Buclin]

The trunk is now frozen to prepare Bugzilla 2.22. Only bug fixes are accepted, no enhancement bugs. As this bug has a pretty low activity (especially from the assignee), it's retargetted to ---. If you want to work on it and you think you can have it fixed for 2.24, please retarget it accordingly (to 2.24).

Comment 31

[Max Kanat-Alexander]

Okay, in present time, the real blocker of this bug is bug 122922 -- we need to have methods in Bug.pm of creating/modifying bugs before we can integrate the email interface into the main code.

We shouldn't get into too much re-writing of the system; let's just make it work.

This bug will just be for the final cvs add/cvs remove of the files out of contrib/.

Comment 32

[Frédéric Buclin]

This bug is in the new roadmap for 2.24.

Comment 33

[Colin Ogilvie [:cso]]

(In reply to comment #32)
> This bug is in the new roadmap for 2.24.

Per the discussions in the meeting, and the roadmap document, I said I'd take this.

Comment 34

[Max Kanat-Alexander]

I'll take this. I have it entirely done, for 2.22.

Comment 35

[Max Kanat-Alexander]

Attachment: Inbound Email Interface for Bugzilla 2.22

Okay, here's the inbound email interface that I wrote for Akamai, for 2.22.

The script itself is very nice. The modifications to attachment.cgi, process_bug.cgi, and post_bug.cgi are all sort of a hack, although they work very nicely for 2.22. For upstream I'd like to do something a little smoother.

Comment 36

[Max Kanat-Alexander]

Just so everybody knows the patch I attached is basically a port of bug_email's interface to entirely new code. 

That is, the code is all mine, but it still uses the same sort of email format as bug_email did, mostly because the email format made sense. It requires users to specify the fields at the top of the email, but the field names have to start with @.

That's the only reliable way to tell if somebody's sent us an email with just a comment or not. It also allows an unlimited amount of whitespace at the top of the email.

The script does allow more fields to be specified than bug_email did. It allows anything that post_bug or process_bug allows, in terms of field names.

It does handle multipart/alternative mails just fine, although you can't include an attachment with a multipart/alternative mail (but only because of a bug in Email::MIME::Attachment::Stripper).

The script *does* require a lot of the Email:: modules. If you're on RHEL, the ones that aren't available at Dag I've used cpan2rpm to make RHEL4 modules of:

http://landfill.bugzilla.org/perl-email/

Validating the "From" address is something we should do in a future bug.

Adding the ability to do things other than update bugs is something we could possibly do in a future bug.

Comment 37

[Max Kanat-Alexander]

Attachment: Basic Framework for Bugzilla 3.0

Okay, and here's the basic framework for Bugzilla 3.0. This is just the email_in.pl script itself and the changes to Bugzilla required for it to compile.

We're going to split this into three bugs:

1) The ability to create bugs
2) The ability to post bugs.
3) The ability to add attachments.

Basically, I'll just be stripping out parts of this script and putting them in separate patches. Each patch will also require other changes to parts of Bugzilla, in order to work, so each patch will also include additional code.

Comment 38

[JK]

(In reply to comment #35)
> Created an attachment (id=236290) [edit]
> Inbound Email Interface for Bugzilla 2.22
> Okay, here's the inbound email interface that I wrote for Akamai, for 2.22.
> The script itself is very nice. The modifications to attachment.cgi,
> process_bug.cgi, and post_bug.cgi are all sort of a hack, although they work
> very nicely for 2.22. For upstream I'd like to do something a little smoother.

Can you briefly explain how this patch works and how to install it.
Many thanks JK

Comment 39

[Max Kanat-Alexander]

(In reply to comment #38)
> Can you briefly explain how this patch works and how to install it.

  Sorry, the patch itself is unsupported. I'm sure you can find some tutorials online about how to apply a patch using the standard "patch" utility, though.

  If you want to know how the email interface works, you can do "perldoc email_in.pl" after you apply the patch.

  Make sure that you don't ask any more questions about it in this bug, though, since it's unsupported. It's possible you could ask questions about it on the support list, if you want to know any more about it.

Comment 40

[Max Kanat-Alexander]

Bugzilla 3.2 is now frozen. Only enhancements blocking 3.2 or specifically approved for 3.2 may be checked in to the 3.2 branch. If you would like to nominate your enhancement for Bugzilla 3.2, set "blocking3.2" tp "?", and either the target milestone will be changed back, or the blocking3.2 flag will be granted, if we will accept this enhancement for Bugzilla 3.2.

Comment 41

[Max Kanat-Alexander]

I would say that this is in fact fixed since Bugzilla 3.0 and email_in.pl. We just need to add attachment support, but that's a separate bug.