1673093 - Download large attachment fails, saved as 27 or 45 bytes, when offline store (mbox/maildir) not used and message/attachment too large to fit in cache RAM/memory and mail.server.default.mime_parts_on_demand=false. workaround: mime_parts_on_demand=true

Status: RESOLVED DUPLICATE of bug 1805186

(MailNews Core :: Networking: IMAP, defect)

Description

[Michael Zapf]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0

Steps to reproduce:

Trying to download a file attached to an e-mail via IMAP; the file is about 18 MiB large (unencoded), 24.4 MiB (base64-encoded). A sample file to attach can be fetched from http://www.mizapf.eu/files/testmail4.zip . This is a file with random content (/dev/urandom), stored in a zip container.

Create a mail message and attach this file to the mail message. Send this message to a mail server. Connect to the server via IMAP with Thunderbird 78, open the message and try to save the attachment.

Actual results:

"Downloading messages" appears in the status line, but without a progress bar; the download seems to have terminated immediately. In the target directory, a new file has appeared with a size of exactly 45 bytes. The contents of the downloaded file are always the same, independent of the actual contents of the file:

00000000: 0e27 ac7a b35a 721a e272 1b5e 9e29 e16a .'.z.Zr..r.^.).j
00000010: 5b70 8ab7 5ab9 f027 7eb6 a07a 4a26 a657 [p..Z..'~..zJ&.W
00000020: adb6 17ab ba7b 5eae 07a5 69d7 a7 .....{^...i..

The effect occurs

• in Thunderbird 78, not in Thunderbird 68 (same server, same message, same attachment) and
• if at least one attached file has this size; then all attached files are saved as the same 45 bytes and
• if the attached file is a ZIP file or a PDF file (this is where I noticed it first, other types may be affected as well). If the file is a application/octet-stream, the download works correctly. It suffices to change the extension of the file to "bin" prior to attaching to make the download work.

If the whole message is copied to a local folder, the attached ZIP file can be downloaded correctly in TB78.

This effect occurred for two different IMAP servers: a) Dovecot, b) IMAP server of Oracle Beehive

Expected results:

As with TB68, the attachment(s) should be correctly saved to disk, independent of the size.

Comment 1

[Magnus Melin [:mkmelin]]

Please attach an imap log - https://wiki.mozilla.org/MailNews:Logging

Comment 2

[Michael Zapf]

The log file can be found at http://www.mizapf.eu/files/imap.log.zip

I clipped away the start of the file in order to prevent disclosure of people and topics from my IMAP mailbox.

Comment 3

[Wayne Mery (:wsmwk)]

see log

Comment 4

[gene smith]

I attached the testmail.bin (after unzipping testmail4.zip) and it works OK with trunk. Does it have to be named something other than .bin to fail? Or should I attach the zip file to duplicate. Sorry, late here and not sure what reporter means I should do to duplicate the exact problem.

Also, in log files looks like a "fetch by parts" is occurring. I thought that was turned off by default for for PGP in 78. i'll look closer tomorrow when I'm awake.

Comment 5

[Michael Zapf]

No, don't unzip. The point is that this file, attached as zipped, cannot be saved, but when you attach a file with other media type, it seems to work. You don't even have to unzip the file; renaming is sufficient.

Comment 6

[gene smith]

Michael, I wrote a test email in plain text and attached your testmail4.zip as-is. Then saved it to Drafts. On open of Drafts folder the full attachment is uploaded to cache (I have no offline store mbox file for Drafts). I can then save the file and unzip it OK. So I don't see a problem. I can also "view source" on the messages and it looks like this (not the full file):

FCC: imap://[e-mail]/Sent
X-Identity-Key: id9
X-Account-Key: account43
To: Gene Smith <[e-mail]>
From: gds <[e-mail]>
Subject: test zip attch
Message-ID: <[e-mail]>
Date: Thu, 29 Oct 2020 13:15:07 -0400
X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0;
 attachmentreminder=0; deliveryformat=4
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101
 Thunderbird/83.0a1
Content-Type: multipart/mixed;
 boundary="------------C5F6135E85D8646D03F72F22"
Content-Language: en-US
MIME-Version: 1.0

--------------C5F6135E85D8646D03F72F22
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

test

--------------C5F6135E85D8646D03F72F22
Content-Type: application/zip;
 name="testmail4.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="testmail4.zip"

UEsDBAoAAAAAAJ0CV1EBh+5NACgdAQAoHQEMABwAdGVzdG1haWwuYmluVVQJAAPKBZJfywWSX3V4
CwABBOgDAAAEZAAAAL2so+Cq9GH7TZGs90dQGpEiutRa9xUXl3p+Jp7jZR/N2foIyVHZx4HzT4jX
m6ewmzT7QZYFt5ufrwUS3zylvVB2NghKfuP3mBQAuFXQvogNzNhSinnNB7spwmTVawhcVAFGUHvG
:
:
AAAAAACdAldRAYfuTQAoHQEAKB0BDAAYAAAAAAAAAAAApIEAAAAAdGVzdG1haWwuYmluVVQFAAPK
BZJfdXgLAAEE6AMAAARkAAAAUEsFBgAAAAABAAEAUgAAAEYoHQEAAA==

--------------C5F6135E85D8646D03F72F22--

Do you have offline store? I haven't yet tried this on dovecot but I will now.

Comment 7

[gene smith]

Ok, I'm seeing it with Dovecot. But I'm seeing a 27 byte file like this: bug 1418444. I'll try to see what's going on.

Comment 8

[gene smith]

I haven't gotten to the root of the problem but, for me, a work-around is to set this back to true: mail.server.default.mime_parts_on_demand. At release 78 this was changed to default to false to help out with encrypted emails, I think so you calculated the signature over the complete email and not just the main body of the email.

Reporter Michael, please let me know if changing mail.server.default.mime_parts_on_demand to true helps or not.
The 27 bytes I see saved are this string in base64: "This body part will be downloaded on demand." I haven't determined what your 45 bytes are.

Note: there's another pref that sounds similar called mail.imap.mime_parts_on_demand but it's only relevant for "legacy" profiles AFAIK.
Note also: setting mail.server.default.mime_parts_on_demand to false typically has a good effect as in this: bug 1576584.

Comment 9

[Michael Zapf]

This seems to fix it indeed.

I set the flag to true, then (without restarting) I was able to download the sample zip file. When I immediately switched to another mail account it failed again (45 byte file), but after restarting Thunderbird it worked for all accounts. I reset the flag to false and got the 45B file again, and set to true again to get the full file.

Also, I noticed that Thunderbird seems to have problems shutting down when the download yields the 45B file. The process is still in the table after I closed the main window. When the download succeeds, the process is terminated as expected.

Comment 10

[gene smith]

Ok, that's good to know.
I'm still wondering if you are using offline store (store messages locally to mbox/maildir file) when fetching the message? Also, are you displaying messages with attachments inline (this is under "View" menu.
Finally, I may need you to record the log again with an additional logging parameter: IMAPCache:5
I'll let you know later if I need the log. If so, it will be with the "on demand" set back to false.
Thanks!

Comment 11

[gene smith]

I don't think I need another log since I can pretty much duplicate the problems here. Still need to know the parameter I asked about.
These are the problems I see:

1. The size of the message is slightly larger than the default cache memory entry size (about 25M)
2. Increasing the cache entry memory entry size from 25M to 30M doesn't help. You have to increase the total cache memory from 200M to 201M to fix it. This seems wrong but I don't know much about these "necko" parameters.
3. Even with cache allocation too small, should still work and allow you to save the attachment. Just every time msg accessed, it has to be re-downloaded.
4. Renaming attachment from .zip to .bin fixes the problem. This is because of the fix for bug 1418444 for octet-stream. bin==>octet-stream.
5. With this change, named testmail4.zip attachment works too:

--- a/mailnews/imap/src/nsIMAPBodyShell.cpp
+++ b/mailnews/imap/src/nsIMAPBodyShell.cpp
@@ -651,17 +651,18 @@ bool nsIMAPBodypartLeaf::ShouldFetchInli
     return false;  // we can leave it on the server
   }
 #endif  // XP_MACOSX
 
   // Fetch type APPLICATION now if the subtype is a signature or if it's an
   // octet-stream. Otherwise, fetch on demand.
   if (!PL_strcasecmp(m_bodyType, "APPLICATION") &&
       PL_strncasecmp(m_bodySubType, "x-pkcs7", 7) &&
-      PL_strcasecmp(m_bodySubType, "octet-stream"))
+      PL_strcasecmp(m_bodySubType, "octet-stream") &&
+      PL_strcasecmp(m_bodySubType, "zip"))
     return false;  // we can leave it on the server
   if (!PL_strcasecmp(m_bodyType, "AUDIO")) return false;
   // Here's where we can add some more intelligence -- let's leave out
   // any other parts that we know we can't display inline.
   return true;  // we're downloading it inline
 }

Maybe many others like "pdf" would have to be added here since I think it was reported that pdf's failed too. Probably not good solution.

6. Not sure why only fails on dovecot but OK on other server. Seem like on dovecot, unmodified tb fetches the wrong part when trying to download the zip attachment. Need to look closer compared to microsoft and other servers.

Comment 12

[Michael Zapf]

I keep turned off the setting "Keep messages for this account on this computer" (pulling all messages sounds wrong to me when I think about IMAP). Also, to prevent downloading, I usually deactivate the Junk filter.

Comment 13

[gene smith]

Ok, just wanted to be sure since I only see the problem when I don't have offline store.

I see two solutions to the issue that don't require any tb code changes:

1. What you have already done, set mail.server.default.mime_parts_on_demand to true. This will fix the problem by reverting the imap fetch behavior back to 68 and earlier. The disadvantage is that it can mess up signature calculations for encrypted messages that are now supported in 78. However, to always have a full/atomic message as required by new PGP/encryption, it may be necessary to use online store for that anyhow, but I don't know the details on this.

2. Another problem is that the message size, when fetched from dovecot is just slightly above the configured cache entry size, 25M. Another server, e.g., microsoft's, returns lines that are 4 bytes longer (72 vs. 76). Therefore the message length is slightly lower (fewer CR/LF's) than the entry size and works with default setting for 78. The default settings for memory cache are these:

browser.cache.memory.capacity 200000
browser.cache.memory.max_entry_size 25000

The values are in Kb. Increasing max_entry_size to 26000 doesn't work. You have to increase the capacity so that capacity/8 is greater than or equal to 26000 to have a maximum entry size of 26000*1024 bytes. I.e., set this as follows:

browser.cache.memory.capacity 208000
browser.cache.memory.max_entry_size 26000

Another way is to just set the capacity to the desired value and set the entry size to -1:

browser.cache.memory.capacity 256000
browser.cache.memory.max_entry_size -1

This will allow a message size up to (256000/8)*1024 or 32M. The -1 just means there is no explicit limit on the entry size. However, there is always an entry limit of no more than 1/8th of the capacity.

Note: All documentation I could find implies that the 1/8th limit applies only to disk cache which tb doesn't use. The documentation states that for memory cache which is used by tb that an entry can be up to 90% of the capacity setting. This does not seem correct based on my testing but the 1/8th limit seems to actually apply also to memory cache as used by tb for message storage when offline store (mbox/maildir) are not enabled.
See https://searchfox.org/comm-central/source/mozilla/modules/libpref/init/StaticPrefList.yaml#820
Edit: Permalink not working, now line 909: https://searchfox.org/comm-central/source/mozilla/modules/libpref/init/StaticPrefList.yaml#909
or search for 1/8 in file.
Edit again: See https://searchfox.org/comm-central/rev/5f59e3d23a6d1ab84c43ea0b89fea0138297a1e2/mozilla/modules/libpref/init/StaticPrefList.yaml#1001 <---- these "permalinks aren't very permanent, maybe have to use a moz central link:
https://searchfox.org/mozilla-central/rev/eddb810ffd5499f0984123fe4bfea6064ebad3c8/modules/libpref/init/StaticPrefList.yaml#1013

Comment 14

[gene smith]

Although there are non-obvious workarounds for this, the basic problem here is that if you have no offline store and your message size exceeds the allocated memory cache or you have cache.memory disabled and you have an attachment that is inherently not inline (like a zip or pdf) and you have fetch parts on demand turned off (the default for >=78), the part number for the attachment, e.g., [2], is never fetched when it is save-as'd. Also, the full message, [], is not re-fetched. Only the message body, [1], is re-fetched.

However, when the attachment is opened the full message is re-fetched and the attachment appears correctly in the application.

So the question is why is the imap access method different between "open" and "save-as" when accessing an attachment?

Comment 15

[gene smith]

Attachment: GetImapPartToFetch.diff

I don't know why the access method is different. When doing "save-as", which fails, the imap response is streamed to a listener. When you do an open into an application, the imap response is fed to a "docshell" object. The decision point is here:
https://searchfox.org/comm-central/rev/cb614692f0c96a4ae1b35f417be81edb39e0dc16/mailnews/imap/src/nsImapService.cpp#595

In both cases it appears that mime-part fetch on demand (MPOD) is forced back to true here:
https://searchfox.org/comm-central/rev/cb614692f0c96a4ae1b35f417be81edb39e0dc16/mailnews/imap/src/nsImapService.cpp#2483
because "part=..." appears in the URI. (Having MPOD configured as false is required to duplicate this bug.) With docshell (open to an app) the full message is re-fetched and the attachment is extracted OK (by libmime code, I think). But when a stream listener is in effect (save-as) only part 1 (the message body) is fetched so the saved information is corrupt because the attachment is never fetched at all.

The call to OpenCacheEntry() in nsImapProtocol.cpp "kicks-off" the imap access using the URI passed to it. The URI contains the proper "part=2" substring needed to request a fetch of the attachment. However, when nsImapUrl::GetImapPartToFetch(char** result) is called, it does not detect the "part=" string in the URI -- instead this function only looks for a "section=" substring and ignores the part= substring. Since there is no section= substring in the URI, the result is null so the attachment part is never fetched.

Looking at many URIs, I rarely, if ever, see "section=" when a multi-part message is accessed. But I always see "part=". So I don't know why only "section=" is looked for and "part=" is ignored in GetImapPartToFetch(). However, technically only the "section" is defined for email URI/URLs in https://tools.ietf.org/html/rfc5092; "part=" is not mentioned. In the tb libmime code, section is referred to as imappart while part is referred to as libmimepart. The imappart/section is required when fetching a message part using, e.g., [2]. The libmimepart/part always seems to have "1." in front so section=2 corresponds to part=1.2.

The attached diff adds checking "part=" for when "section=" is not found. It leaves off the leading "X." for the returned part string. With this change the reporter's example email works properly with current default tb settings and part [2] is fetched, as it should, when saving and opening the attachment.

However, I'm not sure at all if this is a correct fix and why it was done like this to begin with. I can find no documentation on what libmimepart/part and imappart/section actually mean, how they are related or why they differ some. All I could find is a 20 year old reference basically wondering the same thing and possibly addressing a similar issue: bug 80347 comment 8. If anyone reading this can explain this I would appreciate it!
Edit P/s: There is also some discussion of GetImapPartToFetch() starting here bug 417480 comment 44. However "Part" is only used as a delimiter for the returned section. So it is looked at but never returned; only section (imappart) is returned and will be null unless the bodystructure for the message is fetched when the message is first opened. This will only occur when "fetch mime parts on demand" is enabled.

Comment 16

[Magnus Melin [:mkmelin]]

Comment on attachment 9186166 [details] [diff] [review]
GetImapPartToFetch.diff

Review of attachment 9186166 [details] [diff] [review]:
-----------------------------------------------------------------

Unfortunately, I don't really have any clue either.

Comment 20

[gene smith]

This bug (i.e., huge attachment saves as only a very short file) only occurs when offline store (mbox/maildir) is not used and the message or an attachment is too large to fit in the allocated cache RAM memory space and TB mail.server.default.mime_parts_on_demand is set to false which is the default for TB 78 and later.

The patch I attached above only works for a very simple case. With more complex message structures it won't work and I have been unable to come up with a solution that I feel confident won't cause later regressions. The main problem is that the imap URI doesn't contain a "section=" but only has "part=". Part= value can't be used directly in the imap FETCH while section= value, when present, is used directly to access and download an attachment. To obtain the section= value the imap bodystructure must be fetched when the message is first accessed. This bug causes bodystructure to be fetched only when the attachment is saved, so section= never appears in the URI.

The simplest work-around is to just set mail.server.default.mime_parts_on_demand back to true. Of course this may break end-to-end encryption if it is being used which is the reason mime_parts_on_demand was changed to false in version 78. So it might just be done temporarily to save the rare attachment that is very large, or it can be set permanently true only for a specific server account that doesn't use encryption, e.g., by creating and setting mail.server.server3.mime_parts_on_demand to true.

Another work-around is to increase the amount of cache memory allocated as described in comment 13 above to accommodate the largest possible attachment. If allocated RAM cache memory is made large enough, it is not necessary to set mime_part_on_demand to true.

Comment 21

[Mike Looijmans]

I hit this bug today when I received a 20MB+ mail through IMAP. All attachments became 27 bytes when saved.

Since the bug is related to the IMAP handling, the workaround I used was pretty simple: Copy the message to a "Local folders" folder, and then save the attachments from there. This worked perfectly and didn't require any changes to settings.

Comment 22

[gene smith]

Copy the message to a "Local folders" folder, and then save the attachments from there. This worked perfectly and didn't require any changes to settings.

Yes, that should work too since the copy to local fetches the whole message from the server and writes it to the local folder on disk.

Comment 23

[cm-schl]

Just had the same problem and found this bug report.
I would like to add, that only setting mail.server.default.mime_parts_on_demand to true solves most cases of problems but there is a special case where this isn't enough:

If a user doesn't activates to download all mails for offline use AND receives a mail with a large .eml in it (sender forwards a mail as attachment) AND the user has disabled the function "View -> Display Attachments Inline" then the problem appears again after setting mail.server.default.mime_parts_on_demand to true.

The solution is to additionally set
browser.cache.memory.capacity to 256000 ( = 32 MB)
browser.cache.memory.max_entry_size to -1

like Gene already said.

Comment 24

[Rob T.]

I'm seeing this (or a very similar) problem trying to download a 24.5MB PPTX attachment with Thunderbird 91.2.1 with a new fairly default profile. One non-default choice is the setting for the account is set to "Synchronize the most recent 30 days" rather than the default choice of Synchronize all messages locally regardless of age. The option "Don't Download messages larger than" is not checked, which I believe is the default.

I've noticed on recent versions of Thunderbird, that this kicks off a loop where Thunderbird starts using 100% of one CPU.

I don't see this problem with a similarly sized (24.8MB) .mp4 file.

Comment 25

[Thomas D. (:thomas8)]

Resolved duplicate per Gene's Bug 1805186 Comment 12, where a fix for this is being tried.