441791 - Thunderbird produces wrong content type for PDF (*.pdf, application/x-unknown-application-pdf, application/x-download) if previous download affected mimeTypes.rdf

Status: RESOLVED FIXED

(MailNews Core :: Composition, defect)

Description

[Viktor Kabelac]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Build Identifier: Thunderbird 2.0.0.9 (Windows/20071031)

I've just received an email message with these (significant) headers and content. Basically, in the PDF attachment's content-type there is the file extension ("*.pdf") instead of the mime type "text/pdf". Outlooks can manage such a message, other more restrictive clients do not.

User-Agent: Thunderbird 2.0.0.9 (Windows/20071031)
MIME-Version: 1.0
X-Enigmail-Version: 0.95.6
Content-Type: multipart/mixed;
 boundary="------------080700070004030804090009"

--------------080700070004030804090009
Content-Type: multipart/alternative;
 boundary="------------030908040800050908070706"


--------------030908040800050908070706
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit

--------------030908040800050908070706
Content-Type: text/html; charset=ISO-8859-15
Content-Transfer-Encoding: 8bit
--------------030908040800050908070706--

--------------080700070004030804090009
Content-Type: *.pdf;
 name="Programm Workshop Iconparc GmbH mit Firma Schweitzer.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename*0="Programm Workshop Iconparc GmbH mit Firma Schweitzer.pdf"

Reproducible: Didn't try

Steps to Reproduce:
1.
2.
3.

Comment 1

[rsx11m]

I'm having trouble to reproduce this on current nightly trunk builds.
My first guess would be a corrupted entry in the mimeTypes.rdf file,
however when creating an (initially correct) entry for PDF files, then
modifying it to refer to "*.pdf" rather than "application/pdf" as type,
nevertheless it goes out with the correct MIME type.

> Outlooks can manage such a message, other more restrictive clients do not.

Outlook usually uses the file extension instead, not the MIME Content-Type header information, thus it would run into problems for other cases.

Could you ask the sending party to provide a copy of his/her mimeTypes.rdf
file and attach it to this report?

http://kb.mozillazine.org/Profile_folder_-_Thunderbird

Comment 2

[Viktor Kabelac]

Attachment: This is the sender's mimeTypes.rdf

As requested, this is the file from the sender. You seem to be right, there are a few places where the mask "*.pdf" appears, and at least one where I'd have expected a mime type ID.

Comment 3

[Phil Ringnalda (:philor)]

So the question is, how did they get there? Import, saving a received message with the bogus type, manually editing based on bad advice?

Comment 4

[rsx11m]

There are actually two entries, one for "application/pdf" (correct), the other one for the bogus "*.pdf" MIME content type. I didn't try that scenario while testing. To expand on comment #3, are we looking at a bug or wrong usage?

Comment 5

[Phil Ringnalda (:philor)]

Not sure if there's a one-true-bug for it, but the "incoming crap can confuse outgoing types since we'll save a second bogus/crap->.foo mapping when we already know about right/stuff->.foo and then what you get when you ask for the type for .foo is anyone's guess" is a long known and not just Thunderbird bug.

Comment 6

[rsx11m]

Well, in that case it could be checked if the entry to be added is an actual
MIME type. Here, "*.pdf" violates that rule (asterisk and no '/' in type), thus shouldn't be included in the first place. If somebody manually adds something
like this to mimeTypes.rdf, or by some extension, nothing could be done to prevent this of course...

Comment 7

[Viktor Kabelac]

#6: I'm pretty sure the user has not added that manually. If something has added it on its own (while I do not think he's got any fancy extension)...well, that's possible.
I'd ask a similar question - the adding is not done directly into the XML, there'll be an API, right? Would there be a reason to add a second record for the same type? Would it be possible to test the mime type to be added for plausibility or even check it against a list of known MIME types? Or...display a warning if adding a MIME type that's not in the list of known ones?

Comment 8

[rsx11m]

Ok, this looks like bug 142517 comment #23 with a similar test case in
attachment 177149 [details], that bug resolved as WFM though in the end.

To answer the 2nd question of comment #7, an entry should be added to mimeTypes.rdf when an unknown MIME content type is received and the user
selected an action for it, confirming "Do this automatically...".

Thus, the first entry would be created for "application/pdf", a second one possibly for "*.pdf" for the same handler app, if "*.pdf" is treated like
any other MIME type (assuming that box was checked for both). The ambiguity
occurs then when attaching a file with multiple matching rules.

Comment 11

[aleth [:aleth]]

This has been reopened due to a pdf-limited fix landing in m-c. Who knows about mimetypes and can check what still needs doing here?

Comment 12

[Kent James (:rkent)]

I suppose the answer to your question is that any of the core reviewers (myself, aceman, jorg, magnus) know something about mimetypes.

I'm not sure why this was unduped. Do we have any evidence that the fix to bug 306471 (which I am very happy to see, by the way) didn't also fix this bug?

Comment 13

[aleth [:aleth]]

(In reply to Kent James (:rkent) from comment #12)
> I'm not sure why this was unduped. Do we have any evidence that the fix to
> bug 306471 (which I am very happy to see, by the way) didn't also fix this
> bug?

There's a long and complicated discussion in the linked bugs, but see bug 373621 comment 93 for why this was unduped.

Comment 14

[Franta Hanzlik]

My friend has _serious_ problems with this bug, because PDF attachments are sent with mime Content-Type "application/x-download" (instead of correct "application/pdf") - and many sites blocks those mails (with errors as "You attempted to send a message that contained an executable file. Our company policy prohibits the sending of executable files via email. The message was not delivered." etc.).

This 10-year-old bug (and other similar) predispose FF/TB as a toy for a few enthusiasts, not as a reliable tool for serious work.
I personally already some time ago switch to Claws Mail and I'm very happy; my friend are not so lucky, because he has to work in Windows.

Comment 15

[aleth [:aleth]]

(In reply to Franta Hanzlik from comment #14)
> My friend has _serious_ problems with this bug, because PDF attachments are
> sent with mime Content-Type "application/x-download" (instead of correct
> "application/pdf") - and many sites blocks those mails (with errors as "You
> attempted to send a message that contained an executable file.

Recently, a fix for this landed for Firefox (comment 11). We should check it also works OK in current Thunderbird nightlies.

Comment 16

[Jorg K (CEST = GMT+2)]

So what's the question?
For a PDF attachment the mime type should be
Content-Type: application/pdf; not "application/x-download" (comment #14) or anything else.

The summary reads:
  Thunderbird *produces* multipart/mixed message with file extension instead of content type.
That is surely not the case, TB produces application/pdf. Did no one care to try? So this bug is invalid.

That said, if I change application/pdf in a TB produced message to application/x-download, I can still access the attachment in TB 52 Daily.

So there is no need for any action here, so the "toy" (thanks for the insult) actually works.

Comment 17

[aleth [:aleth]]

(In reply to Jorg K (GMT+2) from comment #16)
> So what's the question?
> For a PDF attachment the mime type should be
> Content-Type: application/pdf; not "application/x-download" (comment #14) or
> anything else.
> 
> The summary reads:
>   Thunderbird *produces* multipart/mixed message with file extension instead
> of content type.
> That is surely not the case, TB produces application/pdf. Did no one care to
> try? So this bug is invalid.

OK, thanks for checking! Not knowing compose so well, I wasn't sure I was doing the obvious thing.

Considering this was fixed for Firefox only in Fx 50, it's quite possible it's broken in 45 though.

Comment 18

[Jorg K (CEST = GMT+2)]

This has absolutely nothing to do with the FF fix. Of course the mime type is correct in TB 45 and any other version I've ever used, but I only started at 3.x.

Comment 19

[Viktor Kabelac]

Dear Jörg

> The summary reads:
>   Thunderbird *produces* multipart/mixed message with file extension instead
> of content type.
> That is surely not the case, TB produces application/pdf. Did no one care to
> try? So this bug is invalid.
How do you define invalid? Back in 2008 when I reported this bug, it was very valid. Thunderbird has sent me a message with invalid MIME type.
Is there anything you would argue with?
At the time of reporting, I didn't know there was a crucial text file just about any program can change and so effectively thwart Thunderbird's MIME type handling. In my eyes and in line with the basic rule "be tolerant in what you accept and strict in what you produce", this should not happen and in software I work on, I'd view that as a design bug. There should be no way one crooked incoming message peristently kills a mapping in the program.
So if that happens, I see it as a bug, albeit maybe not in the form as originally assumed.
And in that case this should end with a followup issue and not just "works for me", possibly reinforced with an "always had".

> That said, if I change application/pdf in a TB produced message to
> application/x-download, I can still access the attachment in TB 52 Daily.
That may be due to the "ripening time" of 8 years that was allowd for this bug. Some handling may have gotten more robust in the meantime, still not making the bug report for an 8 years old version invalid. "Aussitzen" does help sometimes, but should not become the desired course of action.
And, by the way, the Firefox fix also sounds to me like a fix specifically targeted at one symtom, instead of finding the root cause that leads to storing a wrong content type for a file extension. Somewhat of "if the result of 10*10 is 120, correct result to 100".

> So there is no need for any action here, so the "toy" (thanks for the
> insult) actually works.
And here, unfortunately, you completely drifted off the path of an IT professional, or at least as a customer would expect aprofessional to act.
In fact, I had the same thought: a program with developers having this "it's for free, so take it or get Office, I'm doing this here for you, so how dare you" stance unfortunately has one large issue if it wants to be taken seriously. Having a bug hang around for 8 years (!) and then cnceling it with "hey, works for me, I don't get what your issue is" is not a good sign for a potential professional user, let alone one who, in future, may be exoected to pay for something related to this program.
Moreover, the "toy" was aimed at TB, almost certainly not at you, so where you ever saw an insult there, I strongly suggest to check whether you are not the best supplier of those to yourself.

I wish the developer sommunity would act more professional, taking issues seriously and not argue that an issue doesn't exist just because it never occured to them. In my field, I just drill in and look for the cause, because a customer almos never reports bugs just for the fun of it. If they report it, they almost certainly experienced it and it's my task, something justifiably expectable of me as a SW developer, to go find how it happened.

Comment 20

[Jorg K (CEST = GMT+2)]

(In reply to Viktor Kabelac from comment #19)
> How do you define invalid? Back in 2008 when I reported this bug, it was
> very valid. Thunderbird has sent me a message with invalid MIME type.
I don't have the time to verify this now, but I take your word for it.
Fact is: It works now in all supported versions.

I believe it worked when I started using TB in 2010, so I'd like to dispute that this problem has been around for eight years. If you want to take this matter further, please bisect this yourself by downloading old versions of TB from
https://ftp.mozilla.org/pub/thunderbird/
In fact, as a responsible reporter you should have checked regularly on the status and closed the bug once you found it working.

> I wish the developer sommunity would act more professional, taking issues
> seriously and not argue that an issue doesn't exist just because it never
> occured to them.
Thanks for continuing the insults. Now you're questioning our professionalism.
Look, TB is maintained bu a small team of overworked, stressed and unpaid volunteers. We do the best we can to maintain a quality product for our 25-30 million users. Calling this product a "toy" in fact shows little respect. And we do take issues seriously, but we have to time to follow up on something that obviously works now. If you're not happy with the product, you have three choices:
1) Fix the problem yourself, it's open source.
2) Hire someone to do it.
3) Use something else.

Can we please stop the discussion. There is no bug any more and we don't have time to look into it any further. As I said, you can convince yourself in which version this began working.

Please read the description of the various bug states here:
https://bugzilla.mozilla.org/page.cgi?id=fields.html
To make you happier, I'll make it "works for me", OK?
If you want to bisect it and deliver me the changeset that fixed the bug, I can set it to fixed, too.

Comment 21

[Jorg K (CEST = GMT+2)]

Correction: we have NO time to follow up on something that obviously works now

Comment 22

[Viktor Kabelac]

You already made it clear that you have neither the time nor will to actually discuss anything. Despite the numerous users/reporters putting up valid points here and in the linked bugs, all you distill it to is feeling personally attacked/insulted. It may well be that you, as a software developer, cannot be expected to act as a front-end to users, but somehow the organisation doesn't provide no one in the middle, so you are stuck with it too.
So instead of becoming personal/insulted, you may try to try getting more in your users' shoes. Because it's only users who make a product great, not the overworkedness of its developers.
As for me, this just leaves a bitter aftertaste, and a feeling of an organisational fault.

Comment 23

[Jorg K (CEST = GMT+2)]

What would you like to discuss?
The behaviour reported in 2008 no longer exists. So the technical matter is indeed closed and Bugzilla is a bug tracker, not a forum for any other discussion.

I tried TB 24 from 2014, the oldest version I have installed, and it works there. As I said, if you can tell me in which version this was fixed, I'm happy to make the bug reflect this.

The bitter aftertaste is due to the reporter coming out of the woods after eight years during which the bug was fixed to start a discussion about unrelated issues. Again: If you report something, please check in each version that comes out whether the problem persists and if not, close your bug.

BMO is currently recording 1.3 million issues, many of which duplicates, so of course it will happen that an issue is fixed without all bugs reporting it being closed. That's where we have to rely on the reporters to follow up.

This thread is really taking time away from other real issues.

Comment 24

[Viktor Kabelac]

As a bug reporter, not a developer, but a user who went a bit farther beyond just sending somewhere a generic email full of rants, I took the time to report and document the issue and supplied further information when asked.
I certainly do not see it as my task to check in every version whether my issue maybe disappeared in the meantime. I would check if someone said "I think I fixed it, can you check again". But without any sign of change, how often would you wish bug reporters to review all of the bugs where no meaningful activity happens, whether they are still on?

I was sitting in the woods, waiting for activity to happen in the bug I reported. Now it happened, now i'm back (In the meantime, I was busy doing professional software development elsewhere).
You can hardly expect more from a mere user of your product.

Comment 25

[Jorg K (CEST = GMT+2)]

(In reply to Viktor Kabelac from comment #24)
> You can hardly expect more from a mere user of your product.
You're not a "mere" user. You opened a BMO account and reported a bug.
And surely you sent a PDF in the last few years and noticed that the problem has gone away.
But fair enough: You were waiting for activity in the bug. You've got it. We officially know now that the problem no longer persists.

Comment 26

[Kent James (:rkent)]

Jorg, are you aware of the long history around problems with PDF mimetype? Your comments "I believe it worked when I started using TB in 2010, so I'd like to dispute that this problem has been around for eight years." and "The summary reads: Thunderbird *produces* multipart/mixed message with file extension instead of content type. That is surely not the case, TB produces application/pdf. Did no one care to try? So this bug is invalid." implies that you are not. IIUC, in certain circumstances attachment downloads can cause changes to the mimetypes.rdf file, and that causes outgoing attachments to be sent with the wrong content type, particularly PDF (at least in my experience).

I fought with this issue in my world here: https://exquilla.zendesk.com/hc/en-us/community/posts/213130966--SOLVED-550-5-6-0-M2MCVT-StorageError-storage-error-in-content-conversion- and there are lots of previous related bugs.

That being said, I'm not really sure how to relate this current bug to the other bugs. Comment 2 shows a mimetypes.rdf that produced a problem, OK. Is this bug about understanding how that file got created, or is it about sending the correct mimetype even with the file with the misdirection? I'm also not sure that we can or should sort all this out unless someone has a more recent case where this is occurring.

Comment 27

[rsx11m]

The hope is that a solution for bug 503309 would prevent mimeTypes.rdf from picking up incorrect MIME types from incoming e-mails. So, either this is a special case of that issue, something fixed by another bug, or something obscure still sticking around (and no, I haven't re-tested this). It's not likely that bug 373621 fixed this, unless nsExternalHelperAppService::GetTypeFromExtension() is consulted during composition to determine the MIME type of an attachment AND overrides mimeTypes.rdf entries.

Comment 28

[Vincent Lefevre]

Well, the bug still exists in:

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101
        Thunderbird/45.2.0

which is not that old. The message was sent with:

Content-Type: application/x-unknown-application-pdf;
 name="Programme_previsionnel_2016.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="Programme_previsionnel_2016.pdf"

Comment 29

[Jorg K (CEST = GMT+2)]

(In reply to Kent James (:rkent) from comment #26)
> Jorg, are you aware of the long history around problems with PDF mimetype?
I'm not aware of any problems ;-) (ignorance is bliss). I simply attached a PDF to a message and checked that the mime type was correct.

So why don't the people who know more about the problem post STR here? Apparently I was all wrong and a very specific problem still persists (see comment #28) but no one has spelled it out clearly so far. And a clear bug summary would help as well.

In any case, this is not an actionable bug to me, so perhaps it should have been closed as "incomplete". I'll make it unconfirmed until someone delivers a reproducible case. So during all these eight years, no one bothered to do so.

Perhaps bug 373621 did indeed fix something here since it corrected an issue related to mimeTypes.rdf. We'll never know since we don't have STR. Note that bug 142517 also got fixed as WFM.

Comment 30

[Viktor Kabelac]

(In reply to Jorg K (GMT+2) from comment #29)
> I'm not aware of any problems ;-) (ignorance is bliss). I simply attached a
> PDF to a message and checked that the mime type was correct.
The bug history stated very early and just as clearly that the wrong MIME type is due to poisoned mimeTypes.rdf, so as far as my imagination goes, you could have just spared yourself a test with a healthy one.

Whatever you do with this very beardy bug, it'd be nice to, for a change, just address the underlying issue.

Comment 31

[Jorg K (CEST = GMT+2)]

I came in at comment #18 and didn't read the history since the summary didn't mention any of it. Clearly, I messed up completely. I'm happy to look into it further is someone else exposes the "underlying" issue with some clear STR.

Comment 32

[rsx11m]

Attachment: Minimal mimeTypes.rdf with *.pdf

This is still reproducible on Windows 7 with the 45.4.0 release and a 2-month old 50.0a1 trunk build (2016-07-27), but no longer with 52.0a1 (2016-10-12).

STR per popular request:
 - take a profile which has an account configured
 - use this minimal mimeTypes.rdf to replace the profile's one
 - verify that the Attachment tab in Options shows "(*.pdf: .pdf)"
 - create a message where you attach a ".pdf" file
 - send that message or save it as draft
 - inspect the message source

On current 45.4.0 release and 50.0a1 (prior to bug 373621):

> Content-Type: *.pdf;
>  name="something.pdf"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
>  filename="something.pdf"

On current 52.0a1:

> Content-Type: application/pdf;
>  name="something.pdf"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
>  filename="something.pdf"

So this specific case looks fixed.

(In reply to rsx11m from comment #27)
> It's not likely that bug 373621 fixed this, unless
> nsExternalHelperAppService::GetTypeFromExtension() is consulted during
> composition to determine the MIME type of an attachment AND overrides
> mimeTypes.rdf entries.

apparently then applies and should make that WFM for the next 52.x round of releases.

The question is if bug 306471, which landed end of August, also fixed bug 503309. However, the test example "*.pdf" wasn't picked up by any of the tested versions, thus the invalid-syntax case is caught already upfront (which leaves the question how the sender in this case was able to get such an entry into his or her mimeTypes.rdf in the first place).

Comment 33

[Jorg K (CEST = GMT+2)]

Thanks!
I see this is in better hands then mine. By the looks of it this got fixed recently (after eight years!), so I'm having a second go at closing this:

Fixed by bug 306471 or bug 373621.
Is someone wants to bisect this closer, go ahead.