543076 - TB 3.0.1 does not handle IMAP Attachments correctly (bug of "Attachment size" add-on or some others)

Status: RESOLVED INVALID

(MailNews Core :: Networking: IMAP, defect)

Description

[mad.engineer]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

TB 3.0.1 does not handle IMAP Attachments correctly. I've searched bugzilla and found other similar bugs reported on this issue, but none have been fixed so far.

Reproducible: Sometimes

Steps to Reproduce:
I ran into this issue while on TB 3.0 but thought that this may be an isolated incident, but ran again while on 3.0.1 and going through the similar bugs makes me believe that something is broken on TB 3.x for sure. Here's what I've seen on this issue:

First instance:
---------------
1) Fresh install of TB 3.0, Win XP Pro SP2
2) IMAP setup against my work email, no MS-Exchange involved
3) Not using off-line setup to save/sync mails locally. Keeping all mails on the IMAP server
4) A user sent me an email from his MS-Exchange account with a .ppt attachment. The email came in, I could see the total email size in the size column ok.
5) Able to open the email and see the message body/headers OK
6) Attachment file name show up in the bottom pane OK, but the size of the attachment showed as Zero bytes. When tried to open it, got error that the file is corrupt or not available.
7) Tried Compacting the Folder, no change
8) I accessed the same IMAP email from my other email client (Pine/Alpine). Was able to see and extract the same attachment without any issues.

2nd Instance:
-------------
1) Fresh install of TB 3.0.1 on Win XP Pro SP2
2) IMAP setup against my work email, no MS-Exchange involved
3) Not using off-line setup to save/sync mails locally. Keeping all mails on the IMAP server
4) Received an email from a non MS-Exchange user with a MS-Word attachment. Total email size around 650Kb, attachment file size around 450Kb.
5) Able to open the email and see the message body/headers OK
6) Attachment file name show up in the bottom pane OK, but the size of the attachment showed as Zero bytes. When tried to open it, got error that the file is corrupt or not available.
7) Tried Compacting the Folder, no change
8) I accessed the same IMAP email from my other email client (Pine/Alpine). Was able to see and extract the same attachment without any issues.
Actual Results:  
Attachments should come across fine.

Expected Results:  
Attachments should come across fine.

I've a long user of UNIX Pine and never had any issues. Since moving to TB 3.x, it's nothing but frustration. I've spent more time searching/posting questions on various TB/Mozilla forums than actually using TB itself. Very disappointed.

Comment 1

[Ludovic Hirlimann [:Usul]]

Can you provide a imap log when you reproduce the issue , that will help us debug the issue. see https://wiki.mozilla.org/MailNews:Logging on how to create such a log.

Comment 2

[mad.engineer]

I'll attach the imap debug log shortly. In the mean time I'm attaching some screen shots to help show this issue better.

Comment 3

[mad.engineer]

Attachment: This is what TB 3.0.1 shows originally

Comment 4

[mad.engineer]

Attachment: Same message when opened

Comment 5

[mad.engineer]

Attachment: Same message shows correct attachment size when viewing with other email client

Comment 6

[mad.engineer]

Attachment: TB Error message

Comment 7

[mad.engineer]

Attachment: imap log file with debug turned on

Comment 8

[mad.engineer]

I've attached the imap.log file. Had to zip it up due to the size limits. I followed the steps listed at: https://wiki.mozilla.org/MailNews:Logging to generate the file, specifically following:

set NSPR_LOG_MODULES=imap:5
set NSPR_LOG_FILE=c:\imap.log
"C:\Program Files\Mozilla Thunderbird\thunderbird.exe"

Comment 9

[David :Bienvenu]

I suspect this has to do with the mime structure of the message. Phil might have a better idea...

Comment 10

[WADA:World Anti-bad-Duping Agency]

Show "Order Received" column(UID if IMAP), and let us know the UID of mail.

Issue like Bug 246415 may be involved.
See also Bug 516211 for "offline use=off" and "Display Attachments Inline=off" case. In this case, message body is not displayed although fetch itslef is normally done.

As you set offline use=off, fetched data is saved in Disk Cache.
> Disk Cache directory of Tb3 on MS Win.
> C:\Documents and Settings\wada\Local Settings\Application Data\Thunderbird\Profiles\<random>.<prof_name>\Cache
> _CACHE_MAP_, _CACHE_001_, _CACHE_002_, _CACHE_003_ are control file.
Can you check cached file content?
1. Delete all files in disk cache directory
2. Restart Tb (if possible, with logging, NSPR_LOG_MODULES=timestamp,imap:5)
3. View the mail
4. Save created files in Disk Cache(not _CACHE_xxx_), and check content of it.
5. Open attached .ppt file
6. Save created files in Disk Cache(not _CACHE_xxx_), and check content of it.
7. View/Message Source
8. Save created files in Disk Cache(not _CACHE_xxx_), and check content of it.
9. Terminate Tb

As attachment size=0 is reported, I guess that data at step 3 is used (attachment part is not downloaded yet, because not-inline-displayable .ppt file).

Comment 11

[Phil Lacy]

Mad, can you upload the actual message. Save it to disk and with a text editor, sanitize it of personal info such as names and addresses.  Editors with find/replace work well for that. If the attachments are personal you probably can remove most of the body of the attachment within the editor. Maybe leave one line for place holder in the editor to corrupt the file for privacy. At least the headers can be seen. We can insert similiar types of files if need be.

Comment 12

[mad.engineer]

Since both of you requested different things, want to make sure that I understand the steps correctly first:

WADA: --> I do have the files that you have mentioned above in the Cache folder 
-----
under the Profile directory. 78 total with 4 control file starting with _CACHE_...

When you say: "Save created files in Disk Cache(not _CACHE_xxx_)", I'm not sure what you mean by that. Can you please clarify this step?. Secondly, after performing all the above steps, do you want me to upload the imap.log file again here?. 

The UID of the message in question as seen in the "Order Received" column is: 695304

Phil:--> When you say "upload the actual message", you mean open the message and then do a "Save As" and upload that file here?

Thanks

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)
> under the Profile directory. 78 total with 4 control file starting with _CACHE_...
> When you say: "Save created files in Disk Cache(not _CACHE_xxx_)", I'm not sure what you mean by that.
> Can you please clarify this step?.

After clear disk cache directry & restart of Tb, not so many files will be created, if you view single mail only.
Keep backup of all files in Disk Cache after each step, because some of them are deleted by next step. Keep back up of _CACHE_xxx too plese, because mail data was saved in it if mail is small.
See attached file to bug 516211, please.

> performing all the above steps, do you want me to upload the imap.log file again here?.

If different flow from already attached log, and analysis of both files in disk cache and imap log by developer is required, upload both, please, after remove sensitive data in files and log.

Comment 14

[WADA:World Anti-bad-Duping Agency]

Attachment: Log for UID fetch 695304 in attached IMAP log

(In reply to comment #12)
> The UID of the message in question as seen in the "Order Received" column is:
> 695304

Thanks. I extracted log lines for UID fetch 695304.

Comment 15

[WADA:World Anti-bad-Duping Agency]

"UID fetch 695304" and IDLE by Tb3 is as follows.
It looks that cached data between "17 UID fetch 695304 (BODY.PEEK[1])" and "18 IDLE" was used for opening attachment data after "19 UID fetch 695304 (BODY.PEEK[2])".

> 15 UID fetch 695304 (BODYSTRUCTURE)
> 16 UID fetch 695304 (BODY.PEEK[HEADER] BODY.PEEK[1.MIME] BODY.PEEK[2.MIME])   >    Content-Type: MULTIPART/MIXED; BOUNDARY="-559023410-2421732-1264465383=:12502"
>   BODY[1.MIME] {59}
>    Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
>   BODY[2.MIME] {283}
>    Content-Type: APPLICATION/msword; name=email_user_name.doc
> 17 UID fetch 695304 (BODY.PEEK[1])
> 18 IDLE
> DONE
> 19 UID fetch 695304 (BODY.PEEK[2])
> 20 IDLE
> DONE
> 21 UID fetch 695304 (BODY.PEEK[2])
> 22 IDLE

Comment 16

[Phil Lacy]

(In reply to comment #12)
> 
> Phil:--> When you say "upload the actual message", you mean open the message
> and then do a "Save As" and upload that file here?
> 

Yes

Comment 17

[Phil Lacy]

I created a similar message and could open the doc file attachment without problem. I have other similar bug's patches deployed.

My TB does not show attachment size. Is there a setting for that or an extension?

Comment 18

[mad.engineer]

I've installed the Attachment size add-on to show the size. But I've tried un-installing it and the still the problem exist.

Comment 19

[mad.engineer]

Attachment: This is the email after doing Save As

Comment 20

[mad.engineer]

I've uploaded the email as requested

Comment 21

[Phil Lacy]

I can open it with a notepad.  msword reports the document is not valid maybe because you deleted some of the raw file for privacy. I'll upload my test email and let you try it. (about 1hour)

Comment 22

[Phil Lacy]

Attachment: email sample

this has a valid doc file and is in the same email structure as the one giving you problems.
Download to disk and drag to your IMAP folder in TB.
Then select to open and double click or 'open' attachment to see if the problem is the same.

Comment 23

[mad.engineer]

When you say "drag to your IMAP folder in TB", you mean send a mail to myself with this file as an attachment and then try to save the attachment again?. Not quite sure about this.

Comment 24

[Phil Lacy]

In 3.0 you can now drag an email into a mail folder. So save the attachment to disk by right clicking this link and 'save link as' attachment 425510 [details] (not sure if I made a good link in this comment, if not just do the same for the attachment link in comment 22)

-You should save it to your hard disk and it should be an *.eml file. 

-Open TB and select a folder in your imap account. You can create a new folder in that account if you would rather, in order to keep it separate, 

-then drag the file and drop it in the message-list pane or right on top of the folder in the folder pane, either way the email will copy over to your IMAP account. It's a large doc file attachment in the message so give it time to upload. that depends on your upload connection speed.

-Then you can open it and try to open the attachment.

Comment 25

[mad.engineer]

OK, I followed you steps by saving the file that you attached to Disk, then dragged n dropped in a separate IMAP folder. Double-clicked the message to open it in TB. Saw the MS-Word attached file. Double-clicked the attachment and was able to open and see the contents of the attachment fine.

So, now I'm curious as to what is the root cause of this issue?. Thanks

Comment 26

[Phil Lacy]

One difference in the messages is your messages has these headers and I don't know for sure what they do in this case

Content-ID: <[e-mail]>
Content-Description: 

Maybe David or WADA can shed some light on it. If you can save your whole message as is, to disk, and remove those lines and drag it in to your temp folder to see if that makes a difference, that's just a trial-and-error type idea.

Comment 27

[WADA:World Anti-bad-Duping Agency]

As seen in IMAP log, attachment part is not downloaded by first fetch. So, size=0 at attachment pane is add-on's bug, or inconsistency between Tb3 and attachment about size of a part. It may affect following attachment handling. Attachment handling may see first fetched data(no data in part for attacment yet) instead of additinally downloaded part data for the attachment.

[e-mail], does your problem occur even after next?
 1. Start Tb with -safe-mode
 2. Rebuild Index
 3. Clear Disk Cache (Tools, Options, Advanced, Network&Disk Space)
 4. View the mail, and double click the attachment

Comment 28

[mad.engineer]

Tried the above steps and it's working now for some reason.

Comment 29

[WADA:World Anti-bad-Duping Agency]

Closing as INVALID, because add-on's bug.
[e-mail], please re-open, if you'll find Tb's bug instead of add-on's bug.