774217 - attachment corrupted on drag-and-drop (data of Thisbodypartwillbedownloadedondemand is generated for part by Drag&Drop of attached application/pdf to composing mail)

Status: VERIFIED FIXED

(MailNews Core :: Networking: IMAP, defect)

Description

[harald.dunkel]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1
Build ID: 20120617215007

Steps to reproduce:

I dragged a pdf attachment from a received EMail to the compose window to forward it to somebody else. (Of course I verified that this attachment is OK.)


Actual results:

The receiver got a corrupted attachement with 27 bytes binary data.


Expected results:

The pdf should have been attached.

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #0)
> Actual results:
> The receiver got a corrupted attachement with 27 bytes binary data.

Sounds similar phenomenon to bug 570914.

Can you attach message source of the corrupted pdf part of sent mail?
　Save the sent mail as .eml file, edit the .eml file by text editor,
  remove all data other than the pdf part corresponds to "27 bytes binary data",
  and attach the .eml file to this bug.
  (For multipart/xxx; boundary="!!!", lines from --!!! to next --!!! or --!!!--)
  (for pdf part of Content-Type: application/pdf, application/octet-strem etc.)

> Steps to reproduce:
> I dragged a pdf attachment from a received EMail
> to the compose window to forward it to somebody else.
> (Of course I verified that this attachment is OK.)

What kind of "verification" did you do with Tb?
Was "Save As" of Tb + open by Adobe Reader, "Open" of Tb using Adobe Reader, normally executed for the PDF part of received mail?

Local mail folder(POP3 or Local Folders)? Or IMAP folder?
If IMAP, Offline-use=On? Off? (Folder Properties/Synchronization)

Can you show us mail structure of the received mail?
Simple multipart/mixed mail with message body text part + pdf part of Content-Type: application/pdf?
　Save the sent mail as .eml file, edit the .eml file by text editor,
  remove all data except following,
     First Content-Type: header before first null line.
       (multipart/xxx; boundary="???", ??? depends on mail/part)
       (a part of multipart/xxx is lines from --??? to --??? or --???--)
     First null line
     Header data of any part(--??? to first null line of the part) 
  and attach the .eml file to this bug.
Long data in each non "Content-Type: multipart/xxx" part is not required.
If a part is also multipart/xxx, different boundary="###" is used. In this case, see lines from --### to next --### or --###--.

If IMAP, can you reproduce your problem with the mail copied to local mail folder?
  Copy the received mail to a mail folder of "Local Folders", and do same steps.

Comment 2

[harald.dunkel]

Attachment: good attachment

Comment 3

[harald.dunkel]

Attachment: bad attachment

Comment 4

[harald.dunkel]

Attachments are online. Please note the "Thisbodypartwillbedownloadedondemand". Sounds like this happens on purpose.

The EMails were created following this procedure:

uuencode tux.pdf tux.pdf | mailx -s tux.pdf `whoami`

Next I have used TB to forward the "tux.pdf" attachment to me (using drag and drop). That is the "good attachment" eml. Finally I have dragged and dropped tux.pdf from the "good" email to another EMail to myself. Thats the "bad attachment" Email showing the problem.

IMAP server is dovecot, maildir is kept on the server.

If I drag the good email to a local mbox folder and try again, then the forwarded attachment is OK.

Hope this helps

Comment 5

[harald.dunkel]

PS: These steps have been done using TB14.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Attachment: Draft mail of Thisbodypartwillbedownloadedondemand generated by Tb's inline Forward of HTML mail with embed image

(In reply to harald.dunkel from comment #3)
> bad attachment

Thisbodypartwillbedownloadedondemand of image could be observed by Tb's inline Forward of HTML mail with embed image, with utilizing wrong RFC822.SIZE by Yahoo! IMAP.

(0) Offline-use=Off IMAP folder.
    View/Message Body As/Original HTML.
    View/Display Attachments Inline=Off(Unchecked).
    Drafts folder == Drafts of Local Folders for ease of test.
    With Yahoo! IMAP.
    Bug 740453 occurs even on MS Win(line ending=CRLF),
    because Yahoo1 IMAP rturns wrong RFC822.SIZE.
    Disk Cache/Memory Cache of Tb is cleared before test.
(1) Original mail = simple HTML mail with a embed PNG.
    multipart/related
      text/html, <img src="cid:cid-url-of-image">
      image/png, Content-ID: <cid-url-of-image>
(2) In mail didplay, the embed PNG is not shown(shown as broken), even with
    Original HTML mode, unless Display Attachments Inline is requsted.
    This is a phenomenon of Bug 740453
(2-1) Because of Offline-use=Off and mail.imap.mime_parts_on_demand=true, message data is held in Disk Cache and PNG image part is initially held in Disk Cache like next; 
> --Bdy
> X-Mozilla-IMAP-Part: 2
> Content-Type: image/png
> Content-Transfer-Encoding: base64
> Content-ID: <image-png-part>
> Content-Disposition: inline; filename="magenta-32x32.PNG"
>
> This body part will be downloaded on demand.
> --Bdy--
(2-2) Because of View/Message Body As Original HTML, the PNG image part is retrieved after cid: URL resolution. However, due to wrong RFC822.SIZE from Yahoo! IMAP, Bug 740453 happens, and Tb fails to obtain the PNG image data.
=> "embed PNG image in HTML mail" is not rendered.
Note-1:
If Gmail IMAP, this broken image is not observed because of correct RFC822.SIZE. 
Note-2:
If Display Attachments Inline=On(Checked), Tb tries to show not-used part of multipart/related as if attachment and the PNG image part is re-fetched, then the PNG image embed in HTML is shown.
(3) Forward As Inline of the mail(UID=577 in this case).
Image location at Forwarding mail is;
> imap://[e-mail]:993/fetch%3EUID%3E/INBOX/Inbox2%3E577?part=1.2&filename=magenta-32x32.PNG
(4) Save As draft 
Because PNG image data for Tb is currently one at step (2-1), Thisbodypartwillbedownloadedondemand is used as PNG part data of forward mail.
(space and period looks removed because of base64 part.)

Comment 7

[WADA:World Anti-bad-Duping Agency]

If application/pdf part, Tb doesn't have capability to display PDF in inline. So PDF part data is not fetched by mime-parts-on-demand until user requests Save As, or Open(by application like Adobe Reader), or View/Message Source, when Offline-use=Off.

Does Tb request fetch for the PDF part or fetch of entire mail data upon your Drag&Drop operation?

Getting IMAP log for initial mail display, Drag&Drop of PDF, Save as draft to local Drafts folder, will help your problem analysis.
  See bug 402793 comment #28.
  Following parameter(Win example) is prhaps useful.
    SET NSPR_LOG_MODULES=timestamp,imap:5,MsgCopyService:5
  If you need to open log data to public, remove/replace personal info, please.

Another possible cause of Thisbodypartwillbedownloadedondemand.
If PDF part under multipart/related or multipart/alternative, recent Tb shows the non-displayed part as if attachment, in order to provide a way to save the PDF part to disk file. Save As/Open for such part currently works well, except in special/rare case such as broken mime type. However, referring to the non-displayed part(for example, Forward in inline) may not always be successful.

Does your IMAP only problem occur even when "Drag&Drop of application/pdf file at attachment pane of valid/normal multipart/mixed mail"?

Comment 8

[harald.dunkel]

Attachment: log file of sample session

Attached you can find the requested log file. Hope this helps

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #8)
> log file of sample session

Log for what operation? Which log lines does corrspond to a your operation?
- Which mail(in what Mbox, of what UID, of what Message-ID:/Subject:) is 
  original mail which has valid PDF attachment data, on which Drag&Drop of
  attachec PDF is initiated?
- Which mail(in what Mbox, of what UID, of what Message-ID:/Subject:) is
  composed & sent mail, on which the Drag&Drop finished?

Comment 10

[harald.dunkel]

I have dragged and dropped the pdf attachment from a received EMail on the Dovecot server to the compose window, as requested. The received EMail has the id '[e-mail]'. Thats the "good attachment" eml included in this bug report (see above).

The composed EMail has the ID '[e-mail]'. See line 105 in the log file.

Please note that even the copy in the local Drafts folder (mbox in my $HOME) shows the problem.

I am sorry for sending too many unrelated lines in the log. I tried to avoid removing something that might be important.

Hope this helps

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #10)
> I have dragged and dropped the pdf attachment from a received EMail on the
> Dovecot server to the compose window, as requested. 
> The received EMail has the id '[e-mail]'.
> Thats the "good attachment" eml included in this bug report (see above).

(1) UID=533888 is assiged to the mail in INBOX by your server.
    Following is log for UID FETCH 533888 (BODY.PEEK[1.MIME] BODY.PEEK[2.MIME])
> 2012-07-19 08:14:44.566964 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:SendData: 14 UID fetch 533888 (BODYSTRUCTURE)
> 2012-07-19 08:14:44.567976 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: * 1350 FETCH (UID 533888 BODYSTRUCTURE (("text" "plain" ("charset" "ISO-8859-1") NIL NIL "7bit" 0 0 NIL NIL NIL NIL)("application" "pdf" ("name" "tux.pdf") NIL NIL "base64" 761812 NIL ("attachment" ("filename" "tux.pdf")) NIL NIL) "mixed" ("boundary" "------------070206050407080503050708") NIL NIL NIL))
> 2012-07-19 08:14:44.573087 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: 14 OK Fetch completed.
> 2012-07-19 08:14:44.573134 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:SendData: 15 UID fetch 533888 (BODY.PEEK[HEADER] BODY.PEEK[1.MIME] BODY.PEEK[2.MIME])
> 2012-07-19 08:14:44.574210 UTC - -260049152[7fd4f42e6bc0]: ReadNextLine [stream=f08d1110 nb=45 needmore=0]
> 2012-07-19 08:14:44.574232 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: * 1350 FETCH (UID 533888 BODY[HEADER] {751}
> 2012-07-19 08:14:44.574248 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:STREAM:OPEN Size: 0: Begin Message Download Stream
> (snip)
> 2012-07-19 08:14:44.574320 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: Message-ID: <[e-mail]>
> 2012-07-19 08:14:44.574407 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: Subject: forwarded tux
> 2012-07-19 08:14:44.574430 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: Content-Type: multipart/mixed;
> 2012-07-19 08:14:44.574489 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket:  BODY[1.MIME] {81}
> (snip)
> 2012-07-19 08:14:44.574505 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: Content-Type: text/plain; charset=ISO-8859-1
> 2012-07-19 08:14:44.574580 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket:  BODY[2.MIME] {141}
> 2012-07-19 08:14:44.574597 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: Content-Type: application/pdf;
> 2012-07-19 08:14:44.574609 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket:  name="tux.pdf"
> (snip)
> 2012-07-19 08:14:44.574667 UTC - -260049152[7fd4f42e6bc0]: f0cfa000:mailhost.sub.mydomain.de:S-INBOX:CreateNewLineFromSocket: )

(2) As seen in above log for UID FETCH 533888 ( ... BODY.PEEK[1.MIME] BODY.PEEK[2.MIME]), your server looks to always return <<origin octet>>(offset of partial mail/part for a chunk), even though Tb doesnt requet it in FETCH command.
This may be an RFC incomplience of your server.
> http://tools.ietf.org/html/rfc3501#section-7.4.2
>  BODY[<section>]<<origin octet>>
>     A string expressing the body contents of the specified section.
>     The string SHOULD be interpreted by the client according to the
>     content transfer encoding, body type, and subtype.
>
>     If the origin octet is specified, this string is a substring of
>     the entire body contents, starting at that origin octet.  This
>     means that BODY[]<0> MAY be truncated, but BODY[] is NEVER
>     truncated.
>
>        Note: The origin octet facility MUST NOT be used by a server
>        in a FETCH response unless the client specifically requested
>        it by means of a FETCH of a BODY[<section>]<<partial>> data
>        item. 

Fetching of application/pdf part may be affected by this <<origin octet>> in FETCH response.

> Please note that even the copy in the local Drafts folder (mbox in my $HOME)
> shows the problem.

Is mail of UID=533888, Message-ID: <[e-mail]>, in server's INBOX actually same as mail data you call "good attachment eml"?
(If IMAP folder, UID of mail is value of "Order Received" column)

Is copy of the mail(UID=533888) in local mail folder actually same as "good attachment eml"?

Comment 12

[harald.dunkel]

<[e-mail]> in dovecot maildir is almost identical to the good attachment eml (except for the CR-LF added by TB). diff -uw shows

--- 1342596216.32273_0.mailhost.sub.mailhost.de:2,Sa	2012-07-18 09:23:36.000000000 +0200
+++ forwarded tux.eml	2012-07-18 09:26:16.170623817 +0200
@@ -1,7 +1,3 @@
-Return-Path: <[e-mail]>
-Received: from host082.sub.mailhost.de (host082.sub.mailhost.de [172.19.97.128])
-	by mailhost.sub.mailhost.de. (8.14.3/8.14.3/Debian-9.4) with ESMTP id q6I7NZi9032269
-	for <[e-mail]>; Wed, 18 Jul 2012 09:23:36 +0200
 Message-ID: <[e-mail]>
 Date: Wed, 18 Jul 2012 09:23:35 +0200
 From: Xxxxx User <[e-mail]>
@@ -12,8 +8,6 @@
 X-Enigmail-Version: 1.4.3
 Content-Type: multipart/mixed;
  boundary="------------070206050407080503050708"
-X-Virus-Scanned: clamav-milter 0.97.3 at mailhost.sub.mailhost.de
-X-Virus-Status: Clean
 
 This is a multi-part message in MIME format.
 --------------070206050407080503050708

The copy in the local mbox is identical to the good attachment eml, except for some additional information kept in the mbox. diff -uw:

--- mbox	2012-07-24 11:11:00.710941874 +0200
+++ forwarded tux.eml	2012-07-18 09:26:16.170623817 +0200
@@ -1,7 +1,3 @@
-From - Wed Jul 18 09:38:26 2012
-X-Mozilla-Status: 0001
-X-Mozilla-Status2: 00000000
-X-Mozilla-Keys:                                                                                 
 Message-ID: <[e-mail]>
 Date: Wed, 18 Jul 2012 09:23:35 +0200
 From: Xxxxx User <[e-mail]>



Hope this helps

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #12)
> <[e-mail]> in dovecot maildir is almost identical to the
> good attachment eml (except for the CR-LF added by TB). diff -uw shows
>(snip)
> The copy in the local mbox is identical to the good attachment eml, except
> for some additional information kept in the mbox. diff -uw:

If mail in IMAP folder of offline-use=Off, Thisbodypartwillbedownloadedondemand in composed mail by Drag&Drop of PDF attachment may occur, because "This body part will be downloaded on demand." is written in Disk Cache for the application/pdf part of the mail.
However, if mail in local mail folder, Thisbodypartwillbedownloadedondemand can not be generated by Tb, because entire mail data is held at local file and because such data never exists in mail data.

Question to avoid wrong result due to mistake.
Is "data in application/pdf part of the good attachment eml used for the diff" actually correect/sufficiently long base64 encoded PDF data?

Do you still persistently reproduce your problem with mail in local mail folder of Tb?
Or you Drag&Drop PDF attachmen at stand alone message window for good attachment eml file?

Comment 14

[WADA:World Anti-bad-Duping Agency]

You said following in comment #4, so I thought IMAP only problem... 
> If I drag the good email to a local mbox folder and try again, 
> then the forwarded attachment is OK.

Comment 15

[harald.dunkel]

(In reply to WADA from comment #14)
> You said following in comment #4, so I thought IMAP only problem... 
> > If I drag the good email to a local mbox folder and try again, 
> > then the forwarded attachment is OK.

I cannot say if it is an imap-only problem, since my Dovecot doesn't have mbox configured. If I copy the source EMail with the good attachment to a local mbox in my $HOME, then I can drag-and-drop the attachment to a new EMail without this problem.

> Question to avoid wrong result due to mistake.
> Is "data in application/pdf part of the good attachment eml used for the diff" 
> actually correect/sufficiently long base64 encoded PDF data?

I am not sure if I got you correctly. The pdf attachment looks OK, except for the CR-LF at the end of each line. If I save the good attachment, then the md5sum is correct.

Comment 16

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #15)
> I cannot say if it is an imap-only problem, since my Dovecot doesn't have
> mbox configured. If I copy the source EMail with the good attachment to a
> local mbox in my $HOME, then I can drag-and-drop the attachment to a new
> EMail without this problem.

What do you call by "my Dovecot" and "doesn't have mbox configured"?

What do you call by "local mbox in my $HOME"?
Tb's local mail folder(POP3 account or Local Folders)? (I meant this by my "local mail folder")
Or other file for mail repository for your local Dovecot server?

Comment 17

[harald.dunkel]

Its "my Dovecot" since I am responsible for the configuration. Its the only Dovecot involved. It doesn't have mbox enabled (and I won't).

"local mbox in my $HOME" is an mbox in $HOME/.thunderbird/*.default/Mail/Local\ Folders. For testing I had copied the "good attachment" email to the local "Drafts" mbox. If I drag the attachment from there to a compose window, then it works fine.

Comment 18

[WADA:World Anti-bad-Duping Agency]

(In reply to harald.dunkel from comment #17)
> "local mbox in my $HOME" is an mbox in $HOME/.thunderbird/*.default/Mail/Local\ Folders.

I now understand your following in comment #4.
> If I drag the good email to a local mbox folder and try again, 
> then the forwarded attachment is OK.
       ---------------------------------       ----------------------------
       "good attachment email" location
       where Drag&Drop of PDF is started       This bug's problem
       ---------------------------------       ----------------------------
  (i)  Your own Dovecot server's Mbox(IMAP)    Occurs
  (ii) Your Tb's mail foler of Local Folders   Doesn't occur.
       ---------------------------------       ----------------------------
This is same as my initial understanding of report.
This indicates IMAP only problem.

If so, what phenomenon with what operation do you call by following in your comment #10?
> Please note that even the copy in the local Drafts folder (mbox in my $HOME) shows the problem.
Following on already broken draft mail, instead of on "good attachment eml"?
- problem occurs by (i).
- composed mail is saved in IMAP Drafts folder, and Thispart..ondemand is seen.
- when the draft mail in IMAP folder is copied to Tb's local mail folder,
  Thispart..ondemand is seen in the copied draft mail too.

Comment 19

[WADA:World Anti-bad-Duping Agency]

Attachment: Minimu test case, eml with PDF attachment, without CRLF after close boundary

multipart/mixed mail with application/pdf part. Eml file's line ending = CRLF. CRLF after close boundary is intentionally removed.
Note: 
base64 data in application/pdf part is dummy data. Never open it with application such as Adobe Reader, please. 

This eml is similar to mail which reproduces bug 740453 even with following;
- IMAP server returns correct RFC822.SIZE (checked with Gmail IMAP)
- IMAP server uses CRLF as line ending in IMAP data stream
- line ending of OS is also CRLF(MS Win, no mismatch with IMAP server's one)

[Steps to reproduce]
(1) create new IMAP Mbox with Offline-use=Off (for test with single mail only)
(2) Drag&Drop the eml file to thread pane of the IMAP Mbox. (import by Dra&Drop)
(3) "Repair Folder", to force clean up of local data
(4) Clear Disk Cache/Memory Cache
(5) Open the test mail
(6) Open mail composition window
(7) Dra&Drop attached PDF of the test mail to composition window
(8) Save As draft
    => Thisbodypartwillbedownloadedondemand in application/pdf part

Comment 20

[WADA:World Anti-bad-Duping Agency]

Above is check result with Tb 14.0.0 on Win-XP, Gmail IMAP.
Confirming.

This is similar problem to bug 740453(on image part). Can you see problem of bug 740453 consistently in your environment, with your Dovecot, Tb, and OS?

Comment 21

[harald.dunkel]

Since 740453 is supposed to be fixed in TB16, I tried the most recent nightly build 
(thunderbird-16.0a2.en-US.linux-x86_64.tar.bz2). Using this version the drag-and-drop works fine. The 'Thisbodypartwillbedownloadedondemand' attachment is gone; I see the expected PDF instead.

Seems OK to me.

Comment 22

[WADA:World Anti-bad-Duping Agency]

I also can't reproduce problem by STR of comment #19 in Tb trunk 2012/7/18 build which I had.

Comment 23

[YS1]

This problem still occurs with TB 16.0.2 (from Ubuntu 12.04 repo).

mail.imap.mime_parts_on_demand is set to false