570914 - IMAP attachments are sometimes corrupt (if base64 encoded part, data of "This body part will be downloaded on demand." is base64 decoded upon save and 27 bytes file is created. offline-use=on, auto-sync is paused by new mail click)

Reporter

Description

•

15 years ago

User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.0 ( .NET CLR 3.5.30729) Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 I am working with thunderbird 3.04 with IMAP server. Sometimes, when trying to view or save attachments, they are saved as 27 bytes files instead of the original file. I have noticed that this might happen when the message was selected, and I selected another message before the original message was downloaded completely. When returning back to the original message that has the attachments, all attachments were saved as this 27 bytes file. Restarting thunderbird has no effect over this. It seems that the cache of the message gets corrupt and does not get fixed. This bug means that the cache does not make sure that the attachment was downloaded correctly before it is being saved to the cache. Reproducible: Sometimes Steps to Reproduce: I did not manage to reproduce it at all times 1. Have a slow connection to your IMAP server, and the inbox folder set to be synchronized for offline usage. 2. Select a message with attachments 3. Select another message and let it load 4. Reselect the original message with attachments and try to open or save the attachments. Actual Results: This will result in a 27 bytes file and not the attachment. Expected Results: Should have saved the original attachment.

mozilla

Reporter

Comment 1

•

15 years ago

Attached file This is the file that is generated, it is the same for all attachments with this problem. — Details

WADA:World Anti-bad-Duping Agency

Comment 2

•

15 years ago

> Restarting thunderbird has no effect over this. > It seems that the cache of the message gets corrupt and does not get fixed. It sounds that downloaded data is partal data but re-download is not invoked. Same problem as Bug 92111 or Bug 390795? Do you use MS Exchange Server? (Q1) What is shown by next? File/Work Offline, View the mail, View Source, Save in .eml, Save attachment. Go back Work Online mode, do same things. (Q2) What happens when the mail is copied to other IMAP folder? Create two new IMAP folders, copy the mail and several big mails. At a folder: Do procedure you stated in comment #0. Is problem reproduced? At another folder : View the mail, wait for normal download. File/Work Offline, save mail as .eml file. Is complete mail data saved in .eml? (Q3) Offline-use=on? Or off? (Folder Properties, Synchronization) What is View/Message Body As setting? What is View/Display Attachment Inline choice? What kind of mail? HTML mail? text mail? Simple multipart/mixed mail? Or nested with multipart/alternative, multipart/related? What kind of attachment? Data which shown in inline such as image? Or data which is always shown as attachment? (Q4) Get IMAP log for download of the mail in newly created IMAP folder. Is something wrong seen in IMAP level flow? > https://wiki.mozilla.org/MailNews:Logging > SET NSPR_LOG_MODULES=timestamp,imap:5 > http://tools.ietf.org/html/rfc3501 for IMAP command/response If you need to open log data to public, please remove sensitive data from log file. In your case, mail data itself is irrelevant except for the 27 bytes part. Delete irrelevant lines if you attach file to this bug.

mozilla

Reporter

Comment 3

•

15 years ago

Hi, Thanks for the quick response. I am not working with exchange, I am working with UW IMAP 2007e on a Linux server under SSL. I did notice that if I do "View Source", there is "This body part will be downloaded on demand." in the MIME part of the attachment. Q1. The "View Source" was grayed (disabled) when I went to offline mode. Q2. a. The problem was not reproduced. However, I think it might be related to that when I open Thunderbird for the first time, and it starts downloading the Inbox folder, and I move around the messages before it completes the load of the entire folder. b. The message is shown fine Q3. a. Offline use is on b. Message Body As - Original HTML c. Display attachment inline is checked d. It's an HTML mail e. Content-Type: multipart/mixed; f. No, the attachment is not inline, it only happened on regular attachments. HOWEVER, on a different occasion, when I have inline images, the image was only partly downloaded, and could never be downloaded completely. Every time I went back to that E-Mail I saw the partial image. Q4. Since I couldn't reproduce it in Q2, I didn't get the problem in the log. I will leave the log on incase I get this problem again. As I said, this problem is not easily reproduced... It seems more like a thunderbird "synchronization" problem - the attachments are marked as synched even though they are not. Thanks

Charles

Comment 5

•

15 years ago

This is *precisely* what happens to mine - the attachments, if saved to the local filesystem, are exactly 27 bytes in size, so, even though I opened mine the day before, I'll mark it as a duplicate of this one since the description here sounds more precise. So, even though I canot formally 'CONFIRM' this bug, I am unofficially confirming it. One additional comment: I updated our most problematic user yesterday afternoon to 3.1rc1, and she said she didn't encounter any more problems after I finished redownloading all of her headers (20,000+ messages in both Inbox and Sent folders). She said this was significant, but I'll wait a little longer before declaring that 3.1 already fixes this problem. But, if you can easily/quickly reproduce the problem, you might try upgrading to 3.1rc1 and see if it fixes it for you too. I'll report back in a day or two whether or not it stays 'fixed'.

Charles

Comment 6

•

Comment 17

•

•

15 years ago

Confirming per attached data and test result with crafted mail.

Status: UNCONFIRMED → NEW

Ever confirmed: true

Summary: When working with IMAP attachments are sometimes corrupt → When working with IMAP attachments are sometimes corrupt (if base64 encoded part, data of "This body part will be downloaded on demand." is base64 decoded upon save)

mozilla

Reporter

Comment 29

•

15 years ago

Hi, When trying to check for your Q5, suddenly the attachments were fine! It seems that TB this time understood that the attachments were not downloaded, and downloaded them! So I guess we can narrow the problem even further - this problem is might not be the state of the attachment, but that in certain cases TB might ignore this state and save the attachment anyway. A6 - All files were saved with 27 bytes. A7 - Yes, in your first comment you instructed me to do a view source, so I checked and that's when I saw this... Great catch!!! You found the cause of the mysterious 27 bytes...

WADA:World Anti-bad-Duping Agency

Comment 30

•

15 years ago

Bug 534835 is quirks for incorrect(shorter) rfc822.size by MS Exchange or some servers. I thought this kind of quirks works in any not-fully-downloaded-yet data cases. But it seems wrong. In this bug's case, mismatch of length probably doesn't happen. I guess some flags are wrongly set in special occasion or some special combinations of flags are mis-interpreted. "Working Offline" and some actions during offline mode looks to have corrected error prawn status fortunately in your case.

WADA:World Anti-bad-Duping Agency

Updated

•

15 years ago

Component: General → Networking: IMAP

•

15 years ago

Quick update, happens with 3.1 as well. The "size" column showd 1k and not the actual size.

Charles

Comment 40

•

•

Comment 63

•

15 years ago

(In reply to comment #60) > If you will meet problem of "27 bytes attachment(s)" in forwarded mail to your > Gmail account, check message source or saved .EML file content of original mail > at your account who forwarded the mail to your Gmail account, please. I tried but maybe too late, the whole content was fetched (viewing source). I'll try on the next auto-forwarded message.

u331193

Comment 64

•

15 years ago

FYI, I've been to tools > filters and edited my filter. I've several options (combo box) : Apply filter : ( ) When receiving mail ( ) during manual execution (X) When receiving mail or manual execution ( ) checking mail (after ordering) ( ) checking mail (after ordering) or manual execution. My translation is probably an average one. So, I was onto "When receiving mail or manual execution", I've switched to the last option. I'll tell you if it's better. I clicked on "Execute" on the window containing the filter list, my mail have been forwarded with full attachments this time. I'm now certain the forward set up as a filter is launched too early, so that thunderbird has no time to fetch the content of the attached parts before forwarding the mail. Hope this helps...

WADA:World Anti-bad-Duping Agency

Comment 65

•

15 years ago

(In reply to comment #63) > I tried but maybe too late, the whole content was fetched (viewing source). It is an evidence that your case is not this bug. If this bug, view source of original should show partial mail data(data in pdf attachment part is string of "This body part will be downloaded on demand."). > I'm now certain the forward set up as a filter is launched too early, > so that thunderbird has no time to fetch the content of the attached > parts before forwarding the mail. I believe problem is in forward by filter, but I don't think timing related issue. I guess problem like next, mismatch between "forward of Tb" and current spec/implementation around "Disk/Memory Cache". Upon forward, mail is not downloaded to offline-store yet by auto-sync. So, fetch is similar to offline-use=off folder case, and downloaded in Disk Cache. And, as multiart/mixed mail with pdf part(not shown in inline), "multipart on demand" is applied and "fetch body[1]", fetch body[2.1]" etc. is issued instead of "fetch body[]"(request of whole mail data). At this step, pdf part of mail data in Disk Cache is "This body part will be downloaded on demand.". This data in Disk Cache is marked "partial" because whole mail data is not held. So, current Tb always issues "fetch body[1]" upon next mail display by mail click if offline-use folder, and issues "fetch body[M.N]" if pdf part is requested by user. I guess forward used this partial data as whole mail data. Anyway, open separate bug for your forward case, please, with crisp problem descrition, with attaching forwared mail to Gmail(file saved as .EML file). No need of original mail(whole mail data) for initial analysis. lapsus63@gmail.com, if possible, check with next setting before bug open, please. (via Config Editor, restart Tb after change to avoid unwanted issue) mail.imap.mime_parts_on_demand : true -> false mail.server.default.mime_parts_on_demand : true -> false By this setting, Tb always issues "fetch body[]" even if offline-use=off. So "download of mail data for forward" will download whole mail data too. If you checked with this setting change, report it in the separate bug instead of this bug, please.

Ian Neal

Comment 66

•

14 years ago

I'm getting this issue too, the source says "This body part will be downloaded on demand." but it never is, just 27 bytes (and yes my size column says 1k, though does say the correct size after a summary rebuild until I click on the message and then it reverts back to 1K). If I set both of the mime_parts_on_demand prefs to false and then rebuild the summary then the misbehaving message is fixed. The backend server in my case is Netscape Messaging Server.

Wayne Mery (:wsmwk)

Updated

•

14 years ago

•

8 years ago

str

The patch looks ok on the surface, but I cannot really reproduce the original problem. There are a lot of comments in the bug, and it was not clear to me the exact required conditions. Could someone give me a summary of what it takes to reproduce this?

fvroman

Assignee

Comment 85

•

8 years ago

str

The bug is easy to reproduce: * Set a filter rules which auto forward incoming email * Receive an email with an attachment * Check the forwarded email in the destination box (may be yours) => the attachment does not have the right size (it is always 27bytes) => Whatever the original attachment, the content is always the same "Thisbodypartwillbedownloadedondemand" (convert the binary content to base64) In fact, when auto forwarding occurs the attachments are not in cache and there is no request to download them before sending the email.

fvroman

Assignee

•

8 years ago

And thank you for the contribution! I can see that you have more patches lined up.

Assignee: nobody → fvroman

fvroman

Assignee

Comment 90

Beta (TB 54): https://hg.mozilla.org/releases/comm-beta/rev/213a3cb0acba2b2c1263da90a7ede8541faa6765

status-thunderbird54: affected → fixed

Jorg K (CEST = GMT+2)

Updated

•

8 years ago

Attachment #8859644 - Flags: approval-comm-esr52? → approval-comm-esr52+

Jorg K (CEST = GMT+2)

Comment 95

•

8 years ago

TB 52 ESR: https://hg.mozilla.org/releases/comm-esr52/rev/62cd4d71b086a8a2887dfe78859267c42aa9b6ff

status-thunderbird_esr52: affected → fixed

tracking-thunderbird_esr52: --- → 54+

Stefan

•

7 years ago

Thanks, I reverted the change from bug 1494764, so this issue will be fixed in TB 60.5.1.

gene smith

Comment 100

•

7 years ago

(In reply to Stefan from comment #96)

It's back.
Thunderbird 60.5.0, Windows 10 latest updates.
Same behaviour.

Can you tell me exactly what problem you saw? Was it a missing or 27 byte attachment after using a filters to forward a just received email like this description from above:

(In reply to fvroman from comment #85)

The bug is easy to reproduce:

Set a filter rules which auto forward incoming email

Receive an email with an attachment

Check the forwarded email in the destination box (may be yours)
=> the attachment does not have the right size (it is always 27bytes)
=> Whatever the original attachment, the content is always the same
"Thisbodypartwillbedownloadedondemand" (convert the binary content to base64)

In fact, when auto forwarding occurs the attachments are not in cache and
there is no request to download them before sending the email.

I need to determine why the code I added that Jorg backed out caused a problem.

Flags: needinfo?(stefan.jurco)

gene smith

Comment 101

•

7 years ago

I don't see a difference with the original 60.5 code or with reverted code. But uncertain on the proper steps to reproduce this bug or whatever was seen in comment 96.

It seems to depend on the bodystructure of the message, but the only problem I see, both ways, is if the folder is not sync'd for offline and attachments are displayed inline. When I place a message with 3 jpeg attachments in the folder, I only see 3 "broken picture" icons inline and the files don't open when double clicked. No problem if not-inline is selected with either code method (of course, nothing inline but the 3 files do open OK in a graphics app).

Otherwise, doing the STR in comment 85 and comment 0 I don't see a problem with or without the revert. (I even slowed down the network to 56kbps.)

I did have to comment out the !aNew assert in nsImapProtocol.cpp since it kept crashing me when I clicked on folders that were in process of autosync. This inspires me to get back into bug 1454542.

Stefan

Comment 102

•

7 years ago

I did some more digging and found out that "Thisbodypartwillbedownloadedondemand" (or other equivalents) are minor issues for our users - might be just server overloaded or combination with other problems described below.

But more often attachments are corrupted in other ways and that happens for almost any attachment.

The root of all problems happens to be that Thunderbird appends closing bracket at the end of almost every email (18 out of 20 checked - for others see at the end)

example:

IDY4IDAgUgovUm9vdCA2OSAwIFIKPj4NCnN0YXJ0eHJlZg0KMzMwMTUKJSVFT0bTThUjU9HT
YwHjXvoEAPz0ITTXbnfXXvbwUAIUPrzRUjU9HTYwHjXvp65ycD09
--------------E07AB7934E6C50D0A2500C03--)

downloading this original attachment results in larger file - last line is base64 decoded for binary attachment or simply appended for text attachment.
Forwarding such attachments (either manually or via filter) results in truncated files not larger files.
If there are no attachments, only html+text email, last line is added to displayed message.
If the email is "single part" html message the closing bracket is added to the content.
If the email is "single part" text message the content is unchanged.

Everything looks good if there are multiple part endings at the end of email:

cYmBUMZQK3NJT9+njL/qXmNVntPILwe/j8tp6Ez6LM7zX3T0OZ4XO25rkrdXjc80/P8A3PvM
9OI31zM8dM89f//Z
--------------CCF187D232F5C880B26AF3F3--

--------------B6A90147D30F03DAA977F510--)

•

3 years ago

I can forward some examples.

Yes, examples are always helpful. You can attach to this bug or zip and send me the problem .eml file.

Also, how are these parameters set for you right now?

mail.imap.mime_parts_on_demand
mail.imap.mime_parts_on_demand_threshold
mail.server.default.mime_parts_on_demand

Thanks!

gene smith

Comment 110

•

3 years ago

Opening EML file permit saving attachments correctly.

Maybe you are saying the email has to be newly received for the problem to occur. If that's what you mean, then forwarding the examples to me is what is needed.

bugzilla.24456

Comment 111

•

3 years ago

(In reply to gene smith from comment #109)

Yes, examples are always helpful. You can attach to this bug or zip and send me the problem .eml file.

I did send you an eml.

Also, how are these parameters set for you right now?

Linux and MS-Win configs both have defaults parameters :

mail.imap.mime_parts_on_demand	true	
mail.imap.mime_parts_on_demand_threshold	500000
mail.server.default.mime_parts_on_demand	false

Thanks!

gene smith

Comment 112

•

3 years ago

Thanks for the sample and I can duplicate the problem. TB is trying to download the whole message and store it in memory cache. Currently TB allocates 25Mbytes for the maximum size message. Your sample message is slightly larger. The workaround is to increase the memory cache setting to accommodate bigger messages:

This will double the max size message that can be stored in memory cache from 25M to 50M:

a) change browser.cache.memory.capacity from default 200000 to 400000
b) change browser.cache.memory.max_entry_size from default 25000 to 50000

You really only need a bit more so b) could be set to 27000 and still work. However, you also need to set a) to 8 times b), so a) would need to be set to 27000 * 8 = 216000. This will reserve less of your RAM memory than if you use the settings to double the max message size.

What you are seeing is also reported in bug 1673093, specifically at bug 1673093 comment 13 and down. I still need to address this issue since it gets reported a lot by users who don't use offline storage and have huge attachments.

Comment 113

•

3 years ago

•

Edited

Another simpler fix for the sample message is to just set mail.server.default.mime_parts_on_demand to true. This will fetch the message by "parts" instead getting the whole thing at once. However, this just barely fixes the problem because the larger PDF part is just below 25M in size. If it were any larger it again wouldn't fit in the default 25M memory slot and the problem would recur.

Note: mail.server.default.mime_parts_on_demand now defaults to false to accommodate the recently added encryption/signature features which won't work without having the whole message fetched and stored in memory.

Edit: Another workaround is here: bug 1673093 comment 21
If you are unable to open or save the attachment, copy the message to a Local Folder and you should be able to open it from there. Actually, this should work if you copy the message to any folder having (disk based) offline store, like a POP3 folder or another imap folder having offline store.

bugzilla.24456

Comment 114

•

3 years ago

(In reply to gene smith from comment #112)

Thanks for the sample and I can duplicate the problem. TB is trying to download the whole message and store it in memory cache. Currently TB allocates 25Mbytes for the maximum size message. Your sample message is slightly larger. The workaround is to increase the memory cache setting to accommodate bigger messages:

As you mentioned it, I put 27000/216000 as parameters and could open attachments without problem for this mail, but for others (larger...), I couldn't !
So, I think the only way to workaround this behaviour is to adapt the maximum mail size allowed for (sending and ) reception at the (Postfix for me) server side to the browser.cache.memory.max_entry_size size (from default 25000 to 50000). Which is a pity, while not possible. This Postfix server was limited to 100M maximum per email... I understand (and approve) that big mail can't make sense nowadays but people still use that sometime.
Other workarounds like saving mail as eml + open it, or saving mail in a local folder (I usually remove local folders for my clients...) are mainly for experienced users. Normal people NEVER do or will do that. And as you tell it in comment 113, mime_parts_on_demand is not usable with encryption/signatures.

Could it be possible to check mail size before trying to open/save attached files ? and auto-magicaly grow browser.cache.memory.max_entry_size as needed ?, and decrease it afterwards ?
Are there bad behaviour I don't see except using more memory for Thunderbird ? Is this memory part allocated from start ?

If there is a maximum at 50M for mail, I consider a real problem here.

Thanks.

gene smith

Updated

•

3 years ago

This is the file that is generated, it is the same for all attachments with this problem. 15 years ago mozilla 27 bytes, application/x-sdlc		Details
Fix the issue when auto forwarding (using filters) mails with attachements 8 years ago fvroman 2.76 KB, patch	jorgk-bmo : review+ jorgk-bmo : approval-comm-beta+ jorgk-bmo : approval-comm-esr52+	Details \| Diff \| Splinter Review