579682 - Attachments disappear when attempting to open and view mail (Tb generates association of .file_extension->multipart/appledouble in mimeTypes.rdf, then multipart/appledouble is applied to file of the .file_extension sent with application/octet-stream)

Status: RESOLVED FIXED

(MailNews Core :: Attachments, defect)

Description

[[e-mail]]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 ( .NET CLR 3.5.30729)
Build Identifier:  ver 4.42.0.0 [taken from .exe properties as I had unstalled the 3.1 version]

incoming mail with attachments [all problem ones are .pdf so far] have the attachment disappear when viewing mail

Reproducible: Sometimes

Steps to Reproduce:
1.Incoming mail with an attachment is shown in the inbox as having an attachment. I know it is true given the time to download and have tried sending test mails with attachments.
2.When mail is clicked on to view the attachment does not appear at the foot of the page. there is nothing to open.
3.When i click back to the inbox the paperclip representing the attachment has gone. All is apparently lost.
4. I have only noticed this happening to .pdf files.
5.When I go back to look at some .pdf attachments that came in before I upgraded to TB 3.1 also disappear into the ether under 3.1 whereas under 2.0.0.24 the version I had prior they had been secure and viewable on request many times.
Actual Results:  
attachments disappeared

Expected Results:  
it should have given me options to view the attachment

I had previously forwarded the problem mails + attachment to another party prior to upgrading to 3.1. I then thought I could recover the attached file by forwarding back to me. The same loss of attachment occurred when they arrived under ver 3.1. I then saved as the attachments from the remote sight which uses T/B 2.0.024 and resent them attached to a new e-mail. All worked OK then.
I uninstalled 3.1 and went back to v2.0.0.24 and again tried to forward the email that was loosing the attachments I had previously receive under 3.1 [which disappeared]. The problem did not reoccur [under 2.0.0.24]
My OS is Win XP pro

Comment 1

[rog_c]

Attachment: showing 'lost' attachments as well as unaffected ones in my Inbox

I would like to report having the same issues with PDFs as [e-mail] but would like to add the following observations:
1. Even though the paperclip attachment icon disappears, the actual PDF attachments are not lost.  I noticed this because the message file size does not drop and a quick look at the message source shows that the entire message is still there.  To confirm this, I forwarded one of the e-mails as-is to my Yahoo WebMail account and the e-mail arrived with the attachment perfectly intact.
2. When I first started having this problem, I simply closed Thunderbird (and my computer) and came back to it a few hours later.  Upon loading up Thunderbird and clicking on the messages that no longer had a paperclip, they magically started to reappear.
3. This seems to be happening more often to very small PDF attachments (ie. less than 10K) and less often with PDFs that are larger.  In fact, when the bug starts to occur, it can sometimes only affect small PDFs while leaving larger ones alone. (see attachment)

Comment 2

[[e-mail]]

Further discoveries here. I re installed version 2.0.0.24 with no more of the problems of lost attachments. My inbox has 16,000 message and my outbox 10,000. I compacted those folders and as I reopened the mails that I knew previously had the attachments [but did not show them under Ver 3 ]The attachment icons all of a sudden reappeared at the foot of the page and could be opened as normal. On going back to the inbox the paper clip that had previously gone as for Rog [e-mail] re appeared also. I'm not sure if the problem was corrupt folder or a bug in Ver 3 or both. Given the grief it caused I'm loath to go back to ver 3 just yet to try it.

Comment 3

[Tom O]

Just to aid the bug team, and as a follow up to my previous posts I am adding my same experience with attachemnts disappearing. 

After much aggravation and bewilderment, I reverted back to ver 2.0.0.24 and I am happy to report that I have NOT experienced the problem with the older version.

I will stay away from Ver 3.0 for the forseable future until it is well documented that the bug has been fixed. 

Long live version 2.0.0.24 (20100228)
TBird RULES!!!!

Comment 4

[David :Bienvenu]

Are you downloading your mail from an imap server or a pop3 server? Do you have an anti-virus program running?

Comment 5

[Tom O]

re you downloading your mail from an imap server or a pop3 server? Do you have
an anti-virus program running?

POP 3
I Uninstalled my anti virus (AVG) to see if that would help and it did not. Still had attachements that disappeared even with no AV installed

Again have had no problems with version 2.0.0.24

Comment 6

[David :Bienvenu]

thx for the info. uninstalled, or disabled? Several people have had this problem in the past few days, but I'd never heard any reports of it until quite recently, and it's a very strange problem.

Comment 7

[Tom O]

Disabled at first. problem still persisted. Then I completely uninstalled it! Prob still persisted
BTW I'm running Win XP Pro SP3 w/AMD athelon +1800 processor if that helps

Comment 8

[Gerwin]

Same problem here (Xp pro SP3, TB 3.1.2). 

Interestingly, out of a bunch of attachments, only the pdfs disappeared but e.g. jpgs remained visible. 

When a message with a single pdf attachment arrives, it is first marked correctly as having an attachments, but the attachment-sign disappears as soon as the message is clicked on.

Behaviour was reproducible every time, regardless of whether full-text search of TB or the virus-scanner (Avira) was enabled. Did not do any more tests as this was on a production machine.

Noticed that a test email with a pdf-attachments sent from the very same machine/installation of TB/account to itself, had this at the beginning of the attachment section in the source-view:

Content-Type: application/octet-stream;
name="EinladRefat.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
filename="EinladRefat.pdf"

However, "application/octet-stream;" did not have a corresponding entry in the Preferences/Attachment menu.

Downgrade to 2.0.0.24 solved the problem for now.

Comment 9

[David :Bienvenu]

Can you attach the mimetypes.rdf file from your user profile dir? See http://kb.mozillazine.org/Profile_folder_-_Thunderbird

Comment 10

[David :Bienvenu]

That was meant for Gerwin. Also, are you using imap or pop3? If the former, does copying the message to a local folder fix the issue? If not, could you try a simple test and copy the message to a small local folder (e.g., a new sub-folder of Local Folders) and then open the folder in a text editor and change the Content type to application/pdf and restart TB, and see if the attachment works in the local message?

Comment 11

[Tom O]

Sorry I can't be of any further technical help on this. I downgraded to v 2.24 and have had NONE of the problems that I experienced with v 3

Comment 12

[David :Bienvenu]

From a private e-mail, I've learned that this happens for pop3 accounts (so it's not an imap-specific issue), and there's more evidence that reverting to 2.0.24 fixes the issue. I've gone through the changes to the attachment handling code and I haven't found anything that looks too suspicious. But since it's libmime, it's very hard to tell.

Comment 13

[David :Bienvenu]

Ah, I tried using the mimetypes.rdf file from one of this bug's victims instead of my own, and the problem manifested itself, so it does seem to have to do with mimetypes.rdf (and potential corruption). Now that I can recreate this, I think there's a pretty good chance I can figure out this bug today.

Comment 14

[David :Bienvenu]

If I remove the following lines from the bad mimetypes.rdf, the bug goes away:

  <RDF:Description RDF:about="urn:mimetype:multipart/appledouble"
                   NC:fileExtensions="pdf"
                   NC:description="Adobe Acrobat Document"
                   NC:value="multipart/appledouble"
                   NC:editable="true">
    <NC:handlerProp RDF:resource="urn:mimetype:handler:multipart/appledouble"/>
  </RDF:Description>

I don't know how they got in mimetypes.rdf, especially on a windows machine, but I would suspect an Acrobat plugin (though I have the pdf plugin and don't see this issue or have that snipped in my own mimetypes.rdf). But what I do know is that these lines make us think the pdf part is an appledouble part, in the 3.x code, anyway.

Comment 15

[David :Bienvenu]

nsIMIMEService->GetTypeFromExtension(".pdf") is returning multipart/appledouble, which I assume is the fault of the strange stuff in mimetypes.rdf

Comment 16

[David :Bienvenu]

For those of you still seeing this, see http://getsatisfaction.com/mozilla_messaging/topics/new_thunderbird_eats_attachements#reply_3440583 for a possible way of fixing the issue.

Comment 17

[David :Bienvenu]

Attachment: possible fix

This fixes the issue for me. I haven't tried it on OS/X, which is the only place I can think of that it might be important. My suspicion is that some version of the Acrobat extension put this strange stuff in mimetypes.rdf. I strongly doubt that pdf files are apple/double anyway. Of course, the question is why did 2.0 not misbehave the same way. My suspicion here is that the mime file attachment content type code has been completely rewritten in the uri loader and is returning a different content type override given the same mimetypes.rdf. I'm going to try to build a 2.0 tree to verify this, but don't hold your breath :-)

Comment 18

[Tom O]

You guys are truly amazing and dedicated for staying on this. 
Im still sticking w/20 for the time being but High Fives all around for you work and dedication!!!

Comment 19

[WADA:World Anti-bad-Duping Agency]

Attachment: Mail folder file of three mails

mail-1 : mutipart/mixed,
         Content-Type: multipart/appledouble; name="...PDF".
After Repair folder, attachment icon is shown, because of multipart/mixed mail. When mail is clicked, attachment icon disappers and nothing is shown at attachment pane, because there is no sub-part of multipart/appledouble.

mail-02 : mutipart/mixed,
         Content-Type: multipart/appledouble; boundary="...".
         two text/plain sub parts.
           Content-Type: text/plain; name="ccc-01.txt"
           Content-Type: text/plain; name="ddd-01.XYZ"
Delete mimeTypes.rdf, and restart Tb. Disable all plugins.
  Note: At this step, attachments of mail-03 is shown at attachment pane.
With Display Attachments Inline=Off, ddd-01.XYZ is shown at attachment pane.
"Open" at context menu of ddd-01.XYZ of mail-02.
Choose Open with: Notepad, check "Do this automatically on file like this from now on", and OK.
=> content of ccc-01.txt and ddd-01.XYZ is shown.
=> association of .XYZ -> multipart/appledouble is generated in mimeTypes.rdf
At this step, atachment icon of mail-03 disappers, and nothing is show at ataachment pane of mail-03.

mail-03 : mutipart/mixed,
         Content-Type: application/octet-stream; name="....XYZ"
Once association of .XYZ->multipart/appledouble is generated in mimeTypes.rdf, this part won't be shown like multipart/appledouble part of mail-01.

Root cause of problem is creation of association of .XYZ->multipart/appledouble in mimeTypes.rdf by Tb. If association is created for .XYZ, it should be Content-Type for name="...XYZ"(text/plain if test mail-02) instead of content-type of container(==multipart/appledouble).

Note:
Once association of .XYZ->multipart/appledouble is generated in mimeTypes.rdf, .XYZ file will be sent with Content-Type: multipart/appledouble; name="...XYZ". It's already known issue, and is repeatedly reported.

Comment 20

[WADA:World Anti-bad-Duping Agency]

Attachment: mimeTypes.rdf with association of .XYZ->multipart/appledouble generated by Tb

Comment 21

[WADA:World Anti-bad-Duping Agency]

FYI.
Replacement of application/octet-stream by multipart/appledouble of mail-03 is result of quirks for application/octet-stream from Tb 3.0.
See Bug 576663 for issues when assciation for file extension is not defined in mimeTypes.rdf(and internal table of Tb).

Comment 22

[David :Bienvenu]

WADA, thx for looking at this.  I see the behavior you describe in trunk builds as well, so it's probably not just 3.0. One thing that's a bit surprising is the bad mimetypes.rdf I've seen, the description                    NC:description="Adobe Acrobat Document" doesn't seem auto-constructed by Thunderbird, unlike the case you were able to reproduce. 

Cc'ing dmose, who I think has been involved in mimetypes.rdf issues before.

Comment 23

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #22)
> the description NC:description="Adobe Acrobat Document" doesn't seem
> auto-constructed by Thunderbird, unlike the case you were able to reproduce. 

The string in desctiption attribute is generated by OS, not by Tb. On Win, string in descrition is string shown at "Type" column of Windows Explorer. If file extension is not registered to Windows registry, MS Win shows "<file-extension> File" in "Type" column, so NC:description="xyz File" is set in my test case.

Comment 24

[David :Bienvenu]

yes, I'll have to figure out who's adding that mimetype, but I think we'll still need this patch to deal with mimetypes.rdf files with this issue.

Comment 25

[WADA:World Anti-bad-Duping Agency]

"Creation of mimeTypes.rdf entry of multipart/appledouble" was already reported to bug 355142. As original report of that bug is for [actual type] == application/octet-stream case in next structure with Tb 1.5/Sm 2.0a1, phenomenon with Tb 3.0 or later may be slightly diffrent due to quirks on application/octet-stream.
>  multipart/appledouble
>   1: application/applefile
>   2: [actual type]

Comment 26

[David :Bienvenu]

thx, WADA. Well, that bug certainly needs a bit more attention than it's been getting.

Comment 27

[WADA:World Anti-bad-Duping Agency]

I guess history is next.

(1) multipart/appledouble handling is special since initial, because of special design of multipart/appledouble.

"file" is a set of first application/applefile part and second [actual type] part. So, multipart/appledouble is assigned as mime-type internally. Data related to 1st application/applefile part is hidden at anywhere, and if resource falk is not used(on non-Mac system), content of first application/applefile part is ignored. At attchment pane, name of second [actual type] part is shown.
  
(2) UI for mimeTypes.rdf registration/maintenace was changed.

Moziila's "Helper Applicatios" was removed. (Tb 1.5, Sm 2.0) Registration to mimeTypes.rdf is done only via "Open dialog". Logic relevant to "Do this automatically on file like this from now on" doesn't care for speciality of multipart/appledouble, then bug 355142 was born by design/implementation change. The bug merely produced next problem which never occurs frequently.
  If user generated entry or multipart/appledouble for an file extension(XYZ),
  .XYZ file is sent with Content-Type: multipart/appledouble; name="....XYZ",
  then recipient can't be aware of attachment or can't receive attachment. 

(3) Quirks for application/octet-stream was introduced by Tb 3.0.

Because the quirks replaces application/octet-stream by mimeTypes.rdf entry for file extension, application/octet-stream is replaced by multipart/appledouble if multipart/appledouble is defined in mimeTypes.rdf for file extension.
Because many mailers send with application/octet-stream instead of proper Content-Type:(it's reason why the quirks was implemented), bug 355142 and problems due to the bug has been exposed to many users. 

Changes like next are required?
(a) Register [actual type] of second part correctly (fix of bug 355142)
(b) Never generate mimeTypes.rdf entry for multipart/xxx
    (protection from future similar troubles)
(c) Ignore(or delete) already defined entry for multipart/xxx in mimeTypes.rdf
    (protection from already generated bad entries)

Comment 29

[David :Bienvenu]

(In reply to comment #27)
> Changes like next are required?
> (a) Register [actual type] of second part correctly (fix of bug 355142)
> (b) Never generate mimeTypes.rdf entry for multipart/xxx
>     (protection from future similar troubles)
> (c) Ignore(or delete) already defined entry for multipart/xxx in mimeTypes.rdf
>     (protection from already generated bad entries)

That sounds right, though other than (c), I'm not sure where this code all lives. I worry that it might be core code.

Comment 30

[David :Bienvenu]

Attachment: don't try to open apple double files

this makes us set the content type of apple double attachments to the content type of the actual attachment. This prevents us from passing appledouble as the content type to the helper app dialog. On non-mac platforms, this would be the right thing to do. I'm not sure about the mac, but I could just #ifndef XP_MACOSX that part of the patch.

Comment 31

[David :Bienvenu]

As a side note, I tried for a while to get mail.app to send appledouble mime parts, but could not do so. Anyone who is seeing this bug, it would be helpful to know which mail agent might have sent an appledouble part.

Comment 32

[Michael A. Pasek]

Attachment: Eudora 6.2.4 (Mac) "Attachments" settings

Eudora "classic" (at least the Mac version) allows selection of attachment encoding method; "AppleDouble" is one of the choices (see attached screen shot).

Comment 33

[WADA:World Anti-bad-Duping Agency]

According to next document, Entourage, Mailsmith, Eudora, Apple Mail, can control "send in appledouble or not".
> http://home.earthlink.net/~bobbau/email/more-tips/#attachments
> Attachment Sending Format (for Mac users)
"Uncheck of [ ] Send Windows Friendly Attachments" looks "send in appledouble" option of Apple Mail.

Comment 34

[David :Bienvenu]

(In reply to comment #33)

> "Uncheck of [ ] Send Windows Friendly Attachments" looks "send in appledouble"
> option of Apple Mail.

I tried that, of course, but it didn't make mail.app send an appledouble file - maybe it needs a file that has a resource fork that it can't describe with mail headers.

Comment 35

[WADA:World Anti-bad-Duping Agency]

Attached mail to bug 355142 is real mail data of multipart/appledouble, with resource falk(application/applefile) and data falk(jpeg, application/octet-stream in that case).

Comment 36

[WADA:World Anti-bad-Duping Agency]

This bug could't be reproduced with .JPG, and problem of "Content-Type: multipart/appledouble; name=aaa.JPG upon mail send" couldn't be reproduced, even though bug 355142 could be reproduced and .JPG->multipart/appledouble was generated in mimeTypes.rdf.
Internal definition of .JPG->image/jpeg looks used prior to mimeTypes.rdf entry use. File extension which Tb doesn't have internal definition seems required to test. Modification of mail source to .JPGX is sufficient for test on non-Mac.
Note:
This bug is for .pdf, duped Bug 589464 is for .qxp, .eps, .ps, .ai, and my test case is for non-defined .XYZ.

Comment 37

[David :Bienvenu]

Comment on attachment 474253 [details] [diff] [review]
possible fix

we're thinking about just taking this part for 3.1.4

Comment 38

[Mark Banner (:standard8)]

Comment on attachment 474253 [details] [diff] [review]
possible fix

Ok, I can't reproduce, but reading about this sounds like the sane thing to do at least initially. r=Standard8.

I'll land on comm-1.9.2 in the appropraite places if you can do the trunk landing.

Comment 39

[Mark Banner (:standard8)]

Checked into 1.9.2:
http://hg.mozilla.org/releases/comm-1.9.2/rev/da29bc7c880d
http://hg.mozilla.org/releases/comm-1.9.2/rev/6c8aebcfc88d

Comment 42

[[:Aureliano Buendía]]

I suppose that we can change the title because this behavior affects also *.doc attachments

Comment 43

[Mark Banner (:standard8)]

Comment on attachment 474914 [details] [diff] [review]
don't try to open apple double files

I think we should go for this as-is and land it on trunk. If we get issues reported then we can consider a Mac specific back out.

Comment 44

[David :Bienvenu]

fixed on trunk - http://hg.mozilla.org/comm-central/rev/870f6de6d857