Closed Bug 1418444 Opened 6 years ago Closed 5 years ago

TB is not able to save an attachment if it is application/octet-stream and the message is stored in a non-sync IMAP folder

Categories

(MailNews Core :: Networking: IMAP, defect)

defect
Not set
normal

Tracking

(thunderbird_esr6067+ fixed, thunderbird67 fixed, thunderbird68 fixed)

RESOLVED FIXED
Thunderbird 68.0
Tracking Status
thunderbird_esr60 67+ fixed
thunderbird67 --- fixed
thunderbird68 --- fixed

People

(Reporter: bflmpsvz, Assigned: gds)

References

Details

(Keywords: regression)

Attachments

(3 files, 2 obsolete files)

Attached image attachment_settings.png
User Agent: Mozilla/5.0 (Windows NT 5.1; rv:41.0) Gecko/20100101 Firefox/41.0
Build ID: 20151014143721

Steps to reproduce:

Open received mail with attached .png image, which has Content-Type: application/octet-stream;


Actual results:

If the .png file is smaller than about 20540 bytes, TB shows it and everything is O.K. If it is larger, TB does not show it, and when I try to save it, only 27 bytes is saved.

TB is 52.2.1 under Windows XP . I have tried upgrade and downdgrade TB. Versions from 2.0.0.22 to 45.8.0 are able to save the attachment, versions 52.1.1 and 52.4.0 are not able.


Expected results:

I found that:

1) The attachment is O.K., because I am able to open Message Source, copy the text and decode the attachment in external BASE64 decoder.

2) It really depends on png. If I rename the file and change by binary editor the first 3 bytes (PNG), TB does not detect that the file is png and allow me to save it without any problem.

Below is Message Source of mail with attached 2 png files, which TB is not able to show or save, but is able it (to show and to save too) if only one of them is attached. But together they exceed that mysterious limit 20540 bytes. When I try to save them, the same 27 bytes (not any part of png file) is saved. 
...
Date: Thu, 16 Nov 2017 13:04:04 +0100 (Central Europe Standard Time)
From: My NAME <mn@domain.cz>
To: My NAME <mn@domain.cz>
Subject: 202 bytes png and 20539 bytes png attached
Message-ID: <alpine.WNT.2.20.1711161302440.2160@mycomp>
User-Agent: Alpine 2.20 (WNT 67 2015-01-07)
X-X-Sender: NAME@mailbox.domain.cz
MIME-Version: 1.0
Content-Type: multipart/mixed; BOUNDARY="48812-9991-1510833804=:2160"

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--48812-9991-1510833804=:2160
Content-Type: text/plain; charset=US-ASCII; format=flowed

202 bytes png and 20539 bytes png attached

--48812-9991-1510833804=:2160
Content-Type: application/octet-stream; name=test_icon.png
Content-Transfer-Encoding: BASE64
Content-Description: 
Content-Disposition: attachment; filename=test_icon.png

iVBORw0KGgoAAAANSUhEUgAAADgAAAA4BAMAAABaqCYtAAAAFVBMVEX///8A
AIAAgIDAwMCAgIAAAACAAACXnUVWAAAAcElEQVR4Xu3PwQ3DIBBE0UF27nzk
BpYKsEgBsUIJcQPpv4ikAM/NR971a1Za3WmakjwVyHZYK+EqNYIwQ74QuBjj
oFzfha2txcXx0lIwsb/7wMX2HN39kpq07C5u7R/RNc5PP1xM+XGSZYE8dKdp
+gGxwgoM8ItJgwAAAABJRU5ErkJggg==

--48812-9991-1510833804=:2160
Content-Type: application/octet-stream; name=UAC.png
Content-Transfer-Encoding: BASE64
Content-Description: 
Content-Disposition: attachment; filename=UAC.png

iVBORw0KGgoAAAANSUhEUgAAAcYAAAFfCAIAAABbafv1AABQAklEQVR42u2d
CZwU1bn2O99NYkz05i7J/W5urkluEhNjrgvooLlR4hd3NIooLqg44o43bigK
yiIyC/smsoPsyDCsyg4iMCwjDMzIMjCswzAwwzArs4HK93Sd7upTa1d318xU
N8/zqx/0VJ86Sy3/fs97Tr3H9zVFURTlknw8BRRFUUQqRVEUkUpRFEWkUhRF
UU2L1HNanaUoiopD6VDWfEg9Zya1Wo2NjQ0URVFxJYDLiq1RENYXtUHaGBTq
.............
YBWrjMlrw4nVsriwLUVR3pTpsqGim28FUyc8dYpUo7kqg1WODSwIS1EUFRcS
DJUDmcthyyOCacRINbVYdeHEGyiKouJKxtUfIrVMY0KqDWHjeoEQiqKo6BbR
cROpYSFLURQVR4oRgO4jlaIo6oIVkUpRFEWkUhRFEakURVFEKkVRFBVe/x9R
t7TurN9i/QAAAABJRU5ErkJggg==

--48812-9991-1510833804=:2160--

Hexadecimal content of the "test_icon_wrong_saved.png"  file: 

00000   4E 18 AC 6E 87 72 A5 AA ED C2 29 65 6D E7 68 C2  N.¬n‡rASíÂ)emçhÂ
00010   79 68 69 D7 9D A2 77 5E 99 A9 DD                        yhi×t?w^™©Ý
Could you please attach a sample message (.eml file). Does it work if you change the content type to image/png?
Testing mail
When the content type is image/png, everything works fine.
I attach the .eml copy of the testing mail. (anonymized)
Now I found, that if I attach .jpg file with wrong type (application/octet-stream instead of image/jpg), the result is the same as with .png - TB is not able to open it even in an external viewer and when I try to save it, only those strange 27 bytes is saved. I have tried pdf, it is not affected (application/octet-stream instead of application/pdf).
Attachment #8929795 - Attachment mime type: message/rfc822 → text/plain
I can't reproduce this, neither in TB 52.4 nor in a (slightly outdated) developer version TB 58 (currently we're at TB 59). I can drag or save the two attachment from the test message to the desktop without a problem. I tried storing the test message in a local folder and an IMAP folder and both cases worked.

Do you have the problem when you run TB without add-ons, see Help menu? What happens on a new profile. Start |thunderbird -p| to create one.

Where is the message stored? Local folder or IMAP folder? If IMAP, synchronised for offline use?
I have checked it on two computers (both Windows XP), a few profiles, behaviour is the same.
Messages are on IMAP, not synchronised for offline (only headers are stored locally). I suppose it has no meaning, when I am able to save the .eml file.

Now I found that when I save the mail as .eml and then I open it using File-Open Saved Message, there is no problem - TB show images and I can save them, regardless of the wrong Content-Type.

What about the Windows XP? I send the same mail to my colleagues, who use TB under Windows 7, and they had no problems to see the images.
I can reproduce the problem with a non-sync IMAP folder. The messages work fine in a local folder or an IMAP folder synchronised for offline use.

Non-sync IMAP folders are always special and cause problems. TB tries to read the "body structure" of the message and any "inline" message parts, like images. application/octet-stream is not an inline message part, hence it's not downloaded. However, you should still be able to save it.

I'll have to look at it when I find the time. This worked in TB 45, so something that broke recently.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: regression
Summary: TB is not able even save an attachment if it is application/octet-stream → TB is not able to save an attachment if it is application/octet-stream and the message is stored in a non-sync folder
Component: Untriaged → Networking: IMAP
Product: Thunderbird → MailNews Core
Version: 52 Branch → 52
Summary: TB is not able to save an attachment if it is application/octet-stream and the message is stored in a non-sync folder → TB is not able to save an attachment if it is application/octet-stream and the message is stored in a non-sync IMAP folder
Maybe you have already noticed, but another workaround for this is the turn off inline attachments.

With inline attachments enabled, I see place holders only for the jpeg/png attachment and, as reported, the links at the bottom won't display the files in the viewer application.

When the message is opened, the "entire message" url is requested and the html and plain text body are fetched. Then each of the attachment "part style" urls are requested and, for each of these, the html and plain text body is fetched instead and stored in the attachment's cache.

When an attachment link is accessed at the bottom, the cache entry is read and contains the message body instead of the expected png or jpeg so the viewer chokes. Saving also doesn't work.

The problem, as you pointed out, seem to be the content type application/octet-stream is disabled for inline download in nsIMAPBodyShell.cpp. However, somewhere else (that I haven't been able to find) .png, jpeg and .jpg files are always assumed to be inline displayable in tb so, even though octet-stream prevents their download, they still need to be fetched for inline use.

A fix is to just allow application/octet-stream to to be fetched for inline just like application/x-pkcs7 in function ShouldFetchInline() in nsIMAPBodyShell.cpp. (This doesn't seem to force non-inline displayable types such as pdf to always be downloaded when assigned the content type application/octet-stream.)

P/S: Another workaround is to turn of browser.cache.memory.enable in config editor.
Attached patch mimei.cpp.patch (obsolete) — Splinter Review
The mime headers for the problem message looks like this:

Content-Type: application/octet-stream; name=test_icon.png
Content-Transfer-Encoding: BASE64
Content-Description: 
Content-Disposition: attachment; filename=test_icon.png

From what I have found, when content type is application/octet-stream and content disposition is "attachment", the content is not typically displayed inline but only as a link so the user can save it or open it in an external viewer. (google: content-type application/octet-stream)

Only if the content type is something like image/png and the disposition is "inline" should the attachment actually appear inline. However, tb currently displays inline if the content type is something that tb knows can be displayed, e.g., image/png, image/jpeg and the file type matches even if disposition is just "attachment" and not "inline".

When tb sees content-type application/octet-stream and content-disposition "attachment" it looks at the file type and if it can be inlined, it changes the type *internally* to, e.g., image/png, image/jpg etc. However, as pointed out in comment 10, the mime part is never actually requested for application/octet-stream, resulting in corrupted cache. But instead of changing nsIMAPBodyShell.cpp, a change in mimei.cpp might be made so the inline part requests for anything originally with content-type application/octet-stream never occur, and also will never appear inline even if attachment inline is set in tb. 

This shows a possible fix without changing nsIMAPBodyShell.cpp. With this change, the the reporter's attachment links open or download OK but the attachments *never* display inline due to the octet-stream content type. Not sure if this is OK or not but it least it answered by question in comment 10:

> However, somewhere else (that I haven't been able to
> find) .png, jpeg and .jpg files are always assumed to be inline displayable
> in tb so, even though octet-stream prevents their download, they still need
> to be fetched for inline use.
I think I prefer a fix where nsIMAPBodyShell.cpp is changed to fetch inline-able content.
Note that I will be landing *a lot* of white-space changes in IMAP, so please don't start a new patch now.

With no changes, the email attached above (attachment 8929795 [details]) displays and saves OK with only memory cache and not inline attachment (just links). But with attachments set to display inline, nothing displays inline (except broken picture icons) and attachment only save as 27 byte files (actually base64 encoding of something like "Attachment will be downloaded on demand").

With the same email in local folders or in imap folder with full offline storage there is no problem with inline or not-inline.

The patch I proposed above (attachment 8982764 [details] [diff] [review]) will make the attachments always appear as links and never inline regardless of the inline/not-inline tb setting. They can be saved and opened in an external viewer only with that patch. (The patch also requires a rebase since there are conflicts due to recent changes in the same area.)

With the changes to nsImapBodyStructure.cpp that I also proposed, all the problems should be solved. So I will attach a patch for that next.

Attached patch 1418444-review-changes.patch (obsolete) — Splinter Review

This seems to solve the problem. See comment 10 for some justification for the fix. Also, I have tested quite a bit with this change applied and haven't seen a problem.

Assignee: nobody → gds
Attachment #8982764 - Attachment is obsolete: true
Attachment #9052794 - Flags: review?(jorgk)

This works nicely. I removed the unneeded parenthesis and the "n" compare of 12 on the "octet-stream". "n" compare of 7 is used to catch x-pkcs7-{signature|mime|crl}.

The patch fixes the "broken images" and saving the attachments, an overall win.

Attachment #9052794 - Attachment is obsolete: true
Attachment #9052794 - Flags: review?(jorgk)
Attachment #9052936 - Flags: review+

Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/1aba8edbbfa2
Allow inline display and saving of Content-Type: application/octet-stream. r=jorgk

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 68.0
Attachment #9052936 - Flags: approval-comm-beta+
Attachment #9052936 - Flags: approval-comm-esr60?
Attachment #9052936 - Flags: approval-comm-esr60? → approval-comm-esr60+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: