381912 - (bz-email_in-attach) [email_in] Add the ability to add attachments by email

Status: RESOLVED FIXED

(Bugzilla :: Incoming Email, enhancement, P1)

Description

[some user]

User-Agent:       Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Build Identifier: Bugzilla 3.0

According to the documentation, email_in.pl will add MIME attachments to bugs that it creates. But when I use it, attachments are parsed and then ignored.

Reproducible: Always

Steps to Reproduce:
1. Craft a multipart MIME message with @parameters in the body, and an attachment.
2. perl email_in.pl < the_email_message

Actual Results:  
Bug is created, but attachment is not added.

Expected Results:  
Bug is created and attachment is added to the bug.

It looks like it has something to do with the way post_bug.cgi processes attachments - email_in.pl adds the attachments to the 'attachments' field/cgi param, but that field isn't used by post_bug.cgi.

Comment 1

[some user]

Attachment: Example message with MIME attachment

Here's the testcase I was using.

Comment 2

[Magnus Holmgren]

It's pretty obvious that it can't work; the attachments are returned by get_body_and_attachments() as a reference to an array of references to hashes, and there is simply no code in place to make anything useful of it.

Comment 3

[some user]

selected a more specific component.

Comment 4

[fedor ezeev]

I've got the same problem on Bugzilla 3.0.1

Comment 5

[Max Kanat-Alexander]

Um, no, according to the docs it can't handle HTML emails with attachments. It doesn't say that it actually adds them.

email_in.pl doesn't yet support attachments, but for some reason I hadn't yet filed the enhancement to add them, so this bug is becoming that enhancement.

We have to wait for Attachment->create to be finished, basically.

Comment 6

[some user]

I don't mean to nitpick, but the 3.0.1 docs state: "If you send multiple attachments in one email, they will all be attached, but Bugzilla may not send an email notice out for all of them." (http://www.bugzilla.org/docs/3.0/html/api/email_in.html)

To me, that implies that it would add them. Someone may want to clarify or remove that text from the documentation.

Comment 7

[Max Kanat-Alexander]

(In reply to comment #6)
> To me, that implies that it would add them. Someone may want to clarify or
> remove that text from the documentation.

  Yeah, that was my mistake. See, I had a version of email_in that was more hacky than the version we finally committed, and it had attachment support, but didn't make it into 3.0. However, that text remained in the POD.

  Please file another bug to remove that text. Thank you for volunteering to write the patch to remove it. :-)

Comment 8

[fedor ezeev]

(In reply to comment #5)
> We have to wait for Attachment->create to be finished, basically.

Very sad news.

(In reply to comment #7)
> See, I had a version of email_in that was more
> hacky than the version we finally committed, and it had attachment support, 

Can you attach your version of email_in.pl to this bug? I'm really need attachment support, becourse I've got this ability in every previous version of bugzilla: from 2.18 to 2.22.

Comment 9

[Max Kanat-Alexander]

(In reply to comment #8)
> Can you attach your version of email_in.pl to this bug?

  It's attached to bug 94850, but it comes with no support. Please don't ask questions about it anywhere in Bugzilla or on the support list.

Comment 11

[fedor ezeev]

Attachment: Add attach from email for Bugzilla 3.0.1

(In reply to comment #9)
> (In reply to comment #8)
> > Can you attach your version of email_in.pl to this bug?
>   It's attached to bug 94850, but it comes with no support. Please don't ask
> questions about it anywhere in Bugzilla or on the support list.

Thank you. This is a patch for 3.0.1, that hacky fix this problem. There is no my code, only yours from both patches from bug 94850 :)

Comment 12

[fedor ezeev]

Attachment: Add multiple attaches from email for Bugzilla 3.0.1

There is two problems in attachment.cgi:
1. 'exit;' at the end of main code part. This 'exit;' does impossible use of attachment.cgi from the loop (as email_in.pl do)
2. (not so critical) attachment.cgi create UI html even in USAGE_MODE_EMAIL.

Comment 13

[fedor ezeev]

(In reply to comment #7)
>   Please file another bug to remove that text. Thank you for volunteering to
> write the patch to remove it. :-)

Bug 395727

Comment 14

[Vincent Castellano]

(In reply to comment #12)
> Created an attachment (id=280354) [details]
> Add multiple attaches from email for Bugzilla 3.0.1
> 

Thanks for the patch, seems to work mostly, however, I have run into a problem where an attachment in an html email with a text/plain part is rejected for having no plain text.

When looking at the verbose output, and dumping the $email structure using Data::Dumper, it looks fine in $email, but $message, from $stripper, is nearly empty. Checking $email again after the call to Email::MIME::Attachment::Stripper, it seems to be also empty. Is this known behavior, and is there anything more I can do to help? I'll continue to poke about, but I'm not sure what is the problem. I can provide an mbox file with just the message if it helps.

[bugmail@dev01 ~]$ cat mbox | /var/www/html/bugzilla/email_in.pl -vvv
Parsing Email
Splitting Body and Attachments [Type: multipart/mixed; boundary="----_=_NextPart_001_01C82623.4855670D"]...
Part Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C82623.4855670D"
Your message did not contain any text.Bugzilla does not
    accept HTML-only email, or HTML email with attachments.

Comment 15

[fedor ezeev]

(In reply to comment #14)
> Your message did not contain any text.Bugzilla does not
>     accept HTML-only email, or HTML email with attachments.

I think that this is not a bug, but feature. And Bugzilla says it to you :)
Don't send html email to bugzilla at all.

Comment 16

[Vincent Castellano]

(In reply to comment #15)
> (In reply to comment #14)
> > Your message did not contain any text.Bugzilla does not
> >     accept HTML-only email, or HTML email with attachments.
> 
> I think that this is not a bug, but feature. And Bugzilla says it to you :)
> Don't send html email to bugzilla at all.
> 

Kind of, Bugzilla says it's HTML only, (HTML email with attachments, which is this case, but...) there is a text/plain section still in the email, I don't see why this section cannot be extracted and used, just like if it was an HTML email, without attachments, as long as it has a text/plain part. If it's done in one case, and not the other, I see it as a bug, or at least, something that could use enhancing.

Comment 17

[Max Kanat-Alexander]

The problem you're experiencing is a known limitation of Email::MIME::Attachment::Stripper. If anybody wants to submit a patch upstream to rjbs to fix it, please, please do.

Comment 19

[Zach van Schouwen]

Attachment: Patch for HTML mail with attachments

(In reply to comment #17)
> The problem you're experiencing is a known limitation of
> Email::MIME::Attachment::Stripper. If anybody wants to submit a patch upstream
> to rjbs to fix it, please, please do.

I've got a patch for this, which we've been using for our in-house Bugzilla install -- if stripper sends back a useless body, it takes the first text/plain section (or, failing that, a text/html section). I don't know maintainers, though, who should I be sending this to? I'm attaching it too.

Not sure this is 100% elegant, but this issue has been driving us nuts, and this fix works well on our end.

Comment 20

[Frédéric Buclin]

Comment on attachment 323083 [details] [diff] [review]
Patch for HTML mail with attachments

review+ is for official reviewers only.

Comment 21

[Max Kanat-Alexander]

Comment on attachment 323083 [details] [diff] [review]
Patch for HTML mail with attachments

This patch should go on a totally different bug--this bug is about having attachments in email_in at all, which we don't have.

This patch should go on a bug that depends on this one, about having HTML email with attachments.

Comment 22

[Zach van Schouwen]

Does that bug exist? Since people were discussing it on _this_ bug, it seemed to be the most appropriate place. I don't see another one.

Comment 23

[Max Kanat-Alexander]

(In reply to comment #22)
> Does that bug exist? Since people were discussing it on _this_ bug, it seemed
> to be the most appropriate place. I don't see another one.

  It doesn't exist. You should file it! :-)

Comment 24

[Alexander]

patch:
"Add multiple attaches from email for Bugzilla 3.0.1"
doesn't work for:
"Example message with MIME attachment"

version:
--snip
GUI: version 3.0.3
$ cat /etc/issue
Fedora Core release 5 (Bordeaux)
Kernel \r on an \m
$ cat /proc/version
Linux version 2.6.20-1.2320.fc5 ([e-mail]) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #1 Tue Jun 12 18:50:38 EDT 2007
--snap

The problem is that after applying that patch it began posts only text attachments:
--snip
Content-Type: text/plain; name="TODO"
Content-Disposition: attachment; filename="TODO"
Content-Transfer-Encoding: base64
--snap
but not:
--snip
Content-Type: JPEG image image/jpeg; name="ad_05.jpg"
Content-Disposition: attachment; filename="ad_05.jpg"
Content-Transfer-Encoding: base64
--snap

Does anybody experienced such a problem already?
Is there any solution for this?

Comment 25

[fedor ezeev]

Attachment: add multiple attachments from email for 3.2.2

Just the same as attachment #280354 [details] [diff] [review], but for 3.2.2 version.
There is no heavy difference between them, just linenumbers.

Comment 26

[Vitaliy Filippov]

Attachment: Implementation of improved mail handler for 3.4.1

Attaching a modified incoming Email handler.
Features:
* Autoregistering disabled users with unknown emails.
* Posting attachments via email.
* Handling HTML email.
* Handling user mail aliases + script to get aliases from LDAP db.
* Ability to call "next-in-pipe" command _before_ processing to get all mail saved.
* Adding CC from email CC: header.
* Default field values for different incoming bugzilla addresses ("helpdesk" ones probably with defaults Product.Component = ...) - editemailin.cgi

Comment 27

[Max Kanat-Alexander]

(In reply to comment #26)
> Created an attachment (id=399713) [details]
> Implementation of improved mail handler for 3.4.1

  Thanks! However, we can't take a patch that does so many things, and particularly not in a tar. You'll want to see our Development Process for information on how to get these features into Bugzilla itself:

  http://wiki.mozilla.org/Bugzilla:Developers

  In particular, the "small patches" part is important.

Comment 28

[Vitaliy Filippov]

OK, I'll begin with Bug 518402 ... :)

Comment 29

[Vitaliy Filippov]

Continue with Bug 518404 ...

Comment 30

[Vitaliy Filippov]

Attachment: A patch for handling attachments

This patch is dedicated to handling attachments.
And Adding CC is Bug 518407.

Comment 31

[Frédéric Buclin]

Comment on attachment 402406 [details] [diff] [review]
A patch for handling attachments

I don't know which code do you use, but this is not Bugzilla 3.5 (tip code).

Comment 32

[Max Kanat-Alexander]

Attachment: v1

Okay, here's a very simple patch that does the necessary functionality, provided that all of the dependent bugs are applied first.

The only thing that this doesn't do is send emails correctly. Instead of adding even more hacks like there are in post_bug.cgi for adding comments when an attachment is added, I'm going to fix bug 452919, and un-hackify things there. So this patch won't be *fully* functional until that bug is fixed.

Comment 33

[Max Kanat-Alexander]

Attachment: v2

Okay, this version correctly adds/updates the comments, and depends on bug 452919. I actually didn't test this version (because the maze of interlocking, dependent, not-checked-in patches became a bit too much to deal with) but I did test that it compiles.

If you want to test a working version without bug 452919 being checked in, you can use v1. (It just doesn't add/update comments correctly.)

Comment 34

[Max Kanat-Alexander]

Attachment: v3

I noticed and fixed a small bug in the last patch.

Comment 35

[David Lawrence [:dkl]]

Comment on attachment 410474 [details] [diff] [review]
v3

[dkl@localhost trunk]$ perl -wcT -I. email_in.pl 
Bareword "CMT_ATTACHMENT_CREATED" not allowed while "strict subs" in use at email_in.pl line 242.
Bareword "CMT_ATTACHMENT_CREATED" not allowed while "strict subs" in use at email_in.pl line 246.
email_in.pl had compilation errors.

-use Bugzilla::Constants qw(USAGE_MODE_EMAIL);
+use Bugzilla::Constants qw(USAGE_MODE_EMAIL CMT_ATTACHMENT_CREATED);

Comment 36

[Max Kanat-Alexander]

Comment on attachment 410474 [details] [diff] [review]
v3

dkl: This patch requires every single blocker in its dependency tree in order to compile and work.

Comment 37

[Frédéric Buclin]

Comment on attachment 410474 [details] [diff] [review]
v3

>+    return ($bug, $bug->longdescs->[0]);

>+        $added_comment = $bug->longdescs->[-1];

s/longdescs/comments/ for both.



>+sub handle_attachments {

>+    Bugzilla::BugMail::Send($bug->id, { changer => Bugzilla->user->login });

Not sure this has any effect about sending bugmail. From what I can see, only the initial comment is emailed, and the fact that attachments are also added is simply ignored. But I guess that's not a major problem.


Otherwise the patch looks good and is working great. Great new feature!! :) r=LpSolit with the fix on checkin above.

Comment 38

[Max Kanat-Alexander]

Ah, thanks! I fixed the Send() problem on checkin, though it currently can't be fixed for process_bug.cgi. I'll attach a patch that shows what I did, which has a comment explaining the situation.

Checking in email_in.pl;
/cvsroot/mozilla/webtools/bugzilla/email_in.pl,v  <--  email_in.pl
new revision: 1.30; previous revision: 1.29
done

Comment 39

[Max Kanat-Alexander]

Attachment: Checked-In Version

Comment 40

[Max Kanat-Alexander]

Added to the release notes in bug 547466.