606129 - Attachments ending in .key displayed inline even if MIME type is application/octet-stream

Status: UNCONFIRMED

(Thunderbird :: Mail Window Front End, defect)

Description

[Martin von Gagern]

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.11) Gecko/20101020 Gentoo Firefox/3.6.11
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.11) Gecko/20101020 Lightning/1.0b3pre Thunderbird/3.1.5

I just got a Apple Keynote file by mail, and was surprised that Thunderbird displayed the attachment inline. Looking at the message source, I found that the sending MUA correctly identified the content disposition as attachment and the content type as application/octet-stream. I built a small demo mail to reproduce the problem. It seems that the file extension makes the difference: changing it to something other than .key will change the rendering from inline to attachment.

Reproducible: Always

Steps to Reproduce:
Open attached demo email
Actual Results:  
Attachment displayed inline

Expected Results:  
Attachment displayed as an attachment only

Comment 1

[Martin von Gagern]

Attachment: Demo email

Comment 2

[Ludovic Hirlimann [:Usul]]

>the content type as application/octet-stream.

Is the default content type for anything unknown.

Comment 3

[Jonathan Protzenko [:protz]]

I can't reproduce using Thunderbird 3.1.x. The attachment is properly displayed as an attachment, regardless of whether I have Display > Attachments inline checked or not. Could you please check that the problem still appears in safe mode? http://kb.mozillazine.org/Safe_mode

Comment 4

[David :Bienvenu]

Jonathan, were you trying this on Linux? It works here for me as well, on Windows.

Comment 5

[Martin von Gagern]

(In reply to comment #3)
> Could you please check that the problem still appears in safe
> mode? http://kb.mozillazine.org/Safe_mode

Could reproduce the problem in safe mode, in the same setup as above.
Could also reproduce the issue with a newly created profile.
Could also reproduce with precompiled binary instead of from-source version.
Failed to reproduce the problem on another machine with OS X.

Comment 6

[Martin von Gagern]

Encountered this again today, in TB 12.0.1 with the same profile on the same machine. I did some additional investigation and came up with the following facts:
- *.key is mapped to "application/pgp-keys" in my /etc/mime.types
- application/pgp-keys is listed in app_and_image_types_which_are_really_text
- changing the file name to e.g. *.me results in the same behaviour.
  *.me mapts to "application/x-troff-me" which is in that list as well
- changing the name to e.g. *.rar hides it, so it isn't that EVERY known file type
  is displayed inline.

See http://mxr.mozilla.org/comm-central/source/mailnews/compose/src/nsMsgCompUtils.cpp#1456 for the app_and_image_types_which_are_really_text list and its context.

There are still a number of reasons why I find these discoveries quite confusing. Why should a list intended for encoding while SENDING a message have any impact on the way a received (or opened) message is DISPLAYED? Why should thunderbird query my /etc/mime.types using the extension from the file name, instead of using the mime type the message sender assigned to the attachment?

Comment 7

[Martin von Gagern]

OK, perhaps app_and_image_types_which_are_really_text is a red herring. According to strace, TB appears to query the shared MIME-info Database on my system, i.e. /usr/share/mime/ and friends. /usr/share/mime/subclasses in particular lists application/pgp-keys as a subclass of text/plain.

I tried to remove *.key from my /etc/mime.types, but without effect so far. Perhaps the association still lingers in some cache or other, although I did run update-mime-database on both the system and my local mime info directory.

Comment 8

[Ludovic Hirlimann [:Usul]]

How is .key defined in mimetypes.rdf in your Thunderbird profile ?

Comment 9

[Martin von Gagern]

(In reply to Ludovic Hirlimann [:Usul] from comment #8)
> How is .key defined in mimetypes.rdf in your Thunderbird profile ?

Apparently not at all: "grep -i key mimeTypes.rdf US/mimeTypes.rdf" yields no single match. But as I wrote in comment #5 that I could reproduce this with a new profile as well, I'm not particularly surprised by that result.

Comment 10

[Ludovic Hirlimann [:Usul]]

Squib, protz do we have some special case for .key ?

Comment 11

[Jim Porter (:squib)]

(In reply to Ludovic Hirlimann [:Usul] from comment #10)
> Squib, protz do we have some special case for .key ?

Not that I know of. I tried the attached email and things look ok for me...

Comment 12

[Martin von Gagern]

(In reply to Ludovic Hirlimann [:Usul] from comment #10)
> Squib, protz do we have some special case for .key ?

As outlined in comment #7, the local mime configuration system does probably play a major role here. So right now I see the bug like this:

1. TB treats the file as application/pgp-keys due to its file extension, despite the fact that the sender specified the mime type as application/octet-stream.

2. TB displays the attachment inline due to the fact that pgp-keys is a subclass of text/plain, despite the fact that the sender specified the content disposition as attachment.

On the whole, this looks like TB considers the local automatic mime inference mechanisms to yield information superiour to what the mail headers contain. Particularly for extensions used by multiple mime types, this is a problem.

Comment 13

[Jim Porter (:squib)]

(In reply to Martin von Gagern from comment #12)
> 1. TB treats the file as application/pgp-keys due to its file extension,
> despite the fact that the sender specified the mime type as
> application/octet-stream.

As far as I know, libmime (which determines if something should be shown inline) doesn't check file extensions at all. Besides, even if I change the message so that the key file has a mimetype of application/pgp-keys, everything works out fine for me.

> 2. TB displays the attachment inline due to the fact that pgp-keys is a
> subclass of text/plain, despite the fact that the sender specified the
> content disposition as attachment.

pgp-keys is not a subclass of text/plain. The only special thing we do with pgp-keys is when sending a message, we flag that mimetype as something that doesn't necessarily need base64-encoding.

> On the whole, this looks like TB considers the local automatic mime
> inference mechanisms to yield information superiour to what the mail headers
> contain.

For opening an attachment, perhaps.* Not for determining whether something should be shown inline or not.

* I don't think this is true either, but it's not my area, so I couldn't say.

Comment 14

[Martin von Gagern]

(In reply to Jim Porter (:squib) from comment #13)
> As far as I know, libmime (which determines if something should be shown
> inline) doesn't check file extensions at all.

Well, SOMEONE does care about the extension, as changing it makes this bug here disappear.

Trying to check that statement of yours, I found the following comment in the libmime sources:
/* There are some clients send out all attachments with a content-type
 of application/octet-stream.  So, if we have an octet-stream attachment,
 try to guess what type it really is based on the file extension.  I HATE
 that we have to do this...
*/
http://mxr.mozilla.org/comm-central/source/mailnews/mime/src/mimei.cpp#841

So it appears there isn't a special case for *.key, but a special case for application/octet-stream.

I strongly disagree with this guessing approach. All accross the web, application/octet-stream is treated as THE universal black box content type, to avoid any mistreatment of files, to ensure they get stored to disk rather than opened with the wrong app, and so on. Not being able to rely on such behaviour when sending mails to TB might lead developers to invent new mime types, like application/x-please-dont-open-but-really-only-save-this or whatever.

I also disagree for my personal use. If a message is broken in some way, I want to be shown a conservative representation and be able to report bugs against the senders MUA. Inlining binary data is not conservative, and hiding missing mime type information from the user masks broken software. So instead of bugging the sender about not providing information, I'm here bugging the receiver about wrongly guessing information it can hardly determine correctly.

I still don't see why an adjusted mime type should result in a changed content disposition as well.

> pgp-keys is not a subclass of text/plain.

It is on my system:

$ grep pgp-keys /usr/share/mime/subclasses
application/pgp-keys text/plain

Iirc my strace showed that TB does actually read this file, possibly through libmime. I guess the original source of this classification lies here:
http://cgit.freedesktop.org/xdg/shared-mime-info/commit/?id=fa74079b19c1f0ef

> > On the whole, this looks like TB considers the local automatic mime
> > inference mechanisms to yield information superiour to what the mail headers
> > contain.
> 
> For opening an attachment, perhaps.* Not for determining whether something
> should be shown inline or not.
> 
> * I don't think this is true either, but it's not my area, so I couldn't say.

Glad we at least agree as to the INTENDED behaviour. I was half afraid someone was going to say "well, bugger mail headers, extensions are all a windows guy can understand, so let those rule." (ref. bug #335171 comment #6)

If the source code comment I quoted above explains why the mime type and thus the associated opener application might be changed for octet-stream attachments, the inlining issue still remains a riddle.