570914 - IMAP attachments are sometimes corrupt (if base64 encoded part, data of "This body part will be downloaded on demand." is base64 decoded upon save and 27 bytes file is created. offline-use=on, auto-sync is paused by new mail click)

Status: RESOLVED FIXED

Description

[mozilla]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.0 ( .NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4

I am working with thunderbird 3.04 with IMAP server. Sometimes, when trying to view or save attachments, they are saved as 27 bytes files instead of the original file. I have noticed that this might happen when the message was selected, and I selected another message before the original message was downloaded completely. When returning back to the original message that has the attachments, all attachments were saved as this 27 bytes file. Restarting thunderbird has no effect over this. It seems that the cache of the message gets corrupt and does not get fixed.

This bug means that the cache does not make sure that the attachment was downloaded correctly before it is being saved to the cache.

Reproducible: Sometimes

Steps to Reproduce:
I did not manage to reproduce it at all times
1. Have a slow connection to your IMAP server, and the inbox folder set to be synchronized for offline usage.
2. Select a message with attachments
3. Select another message and let it load
4. Reselect the original message with attachments and try to open or save the attachments.
Actual Results:  
This will result in a 27 bytes file and not the attachment.

Expected Results:  
Should have saved the original attachment.

Comment 1

[mozilla]

Created attachment 450066 [details]
This is the file that is generated, it is the same for all attachments with this problem.

Comment 2

[WADA:World Anti-bad-Duping Agency]

> Restarting thunderbird has no effect over this.
> It seems that the cache of the message gets corrupt and does not get fixed.

It sounds that downloaded data is partal data but re-download is not invoked. Same problem as Bug 92111 or Bug 390795? Do you use MS Exchange Server?

(Q1) What is shown by next?
     File/Work Offline, View the mail, View Source, Save in .eml, Save
     attachment. Go back Work Online mode, do same things.
(Q2) What happens when the mail is copied to other IMAP folder?
     Create two new IMAP folders, copy the mail and several big mails.
     At a folder: Do procedure you stated in comment #0.
                  Is problem reproduced?
     At another folder : View the mail, wait for normal download.
                         File/Work Offline, save mail as .eml file.
                         Is complete mail data saved in .eml?
(Q3) Offline-use=on? Or off? (Folder Properties, Synchronization) 
     What is View/Message Body As setting?
     What is View/Display Attachment Inline choice?
     What kind of mail? HTML mail? text mail?
     Simple multipart/mixed mail? Or nested with multipart/alternative, 
     multipart/related?
     What kind of attachment? Data which shown in inline such as image?
     Or data which is always shown as attachment? 
(Q4) Get IMAP log for download of the mail in newly created IMAP folder.
     Is something wrong seen in IMAP level flow?
> https://wiki.mozilla.org/MailNews:Logging
> SET NSPR_LOG_MODULES=timestamp,imap:5
> http://tools.ietf.org/html/rfc3501 for IMAP command/response
If you need to open log data to public, please remove sensitive data from log file. In your case, mail data itself is irrelevant except for the 27 bytes part. Delete irrelevant lines if you attach file to this bug.

Comment 3

[mozilla]

Hi,

Thanks for the quick response.

I am not working with exchange, I am working with UW IMAP 2007e on a Linux server under SSL.

I did notice that if I do "View Source", there is "This body part will be downloaded on demand." in the MIME part of the attachment.

Q1. The "View Source" was grayed (disabled) when I went to offline mode.
Q2. 
 a. The problem was not reproduced. However, I think it might be related to that when I open Thunderbird for the first time, and it starts downloading the Inbox folder, and I move around the messages before it completes the load of the entire folder.
 b. The message is shown fine
Q3. 
 a. Offline use is on
 b. Message Body As - Original HTML
 c. Display attachment inline is checked
 d. It's an HTML mail
 e.  Content-Type: multipart/mixed; 
 f. No, the attachment is not inline, it only happened on regular attachments. HOWEVER, on a different occasion, when I have inline images, the image was only partly downloaded, and could never be downloaded completely. Every time I went back to that E-Mail I saw the partial image.
Q4. Since I couldn't reproduce it in Q2, I didn't get the problem in the log. I will leave the log on incase I get this problem again. As I said, this problem is not easily reproduced... It seems more like a thunderbird "synchronization" problem - the attachments are marked as synched even though they are not.

Thanks

Comment 5

[Charles]

This is *precisely* what happens to mine - the attachments, if saved to the local filesystem, are exactly 27 bytes in size, so, even though I opened mine the day before, I'll mark it as a duplicate of this one since the description here sounds more precise.

So, even though I canot formally 'CONFIRM' this bug, I am unofficially confirming it.

One additional comment: I updated our most problematic user yesterday afternoon to 3.1rc1, and she said she didn't encounter any more problems after I finished redownloading all of her headers (20,000+ messages in both Inbox and Sent folders). She said this was significant, but I'll wait a little longer before declaring that 3.1 already fixes this problem.

But, if you can easily/quickly reproduce the problem, you might try upgrading to 3.1rc1 and see if it fixes it for you too.

I'll report back in a day or two whether or not it stays 'fixed'.

Comment 6

[Charles]

(In reply to comment #0)
> Actual Results:  
> This will result in a 27 bytes file and not the attachment.
> 
> Expected Results:  
> Should have saved the original attachment.

I always add the 'Size' column in the message list pane... and when a message has this problem, the size is incorrectly displayed as 1K, rather than the correct size...

Can you confirm the same on yours?

Comment 7

[mozilla]

I also have the "Size" column on, but I never checked it before.. :)
Currently I have only one message in my inbox with this problem, but the size shown is correct. I can't be sure for other cases because I didn't check it.

Comment 10

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> I also have the "Size" column on, but I never checked it before.. :)
> Currently I have only one message in my inbox with this problem, but the size
> shown is correct. I can't be sure for other cases because I didn't check it.

[e-mail](bug opener), please see next in bug 570804 comment #0, 
> and the size for the message will be incorrectly reported as 1K.
and please see Bug 543076 and Bug 543076 which I closed as DUP of Bug 543076, which looks externally similar phenomenon to your "attachment looks corrupted".
In testing for Bug 565852, I coudn't see incorrect Size column value like 1KB for large mail.
I guess some comment poster to your bug is consused.

Comment 11

[Charles]

(In reply to comment #10)
> and please see Bug 543076 and Bug 543076 which I closed as DUP of Bug 543076,

How is a bug closed as a duplicate of itself?

> which looks externally similar phenomenon to your "attachment looks
> corrupted".
> In testing for Bug 565852, I coudn't see incorrect Size column value like 1KB
> for large mail.
> I guess some comment poster to your bug is consused.

Like I said, it doesn't *always* revert to 1K, but the times I have personally seen it happen - especially when it is happening multiple times when I'm rebuilding the Index for a folder - it does spontaneously change the size displayed from the real size to 1K.

Now that I think of it - I *think* that when the size *didn't* change it was a PDF attachment, and when it did, it was a .doc... but I can't be certain. I'll try to watch for this...

Also, maybe these are very similar bugs, but for one reason or another - say a slight environment difference (extension or other) - the symptoms differ slightly.

Comment 12

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #11)
> Like I said, it doesn't *always* revert to 1K, (snip)
> it does spontaneously change the size displayed from the real size to 1K.

Charles Marcus, your problem is better to be analyzed separately. Please re-open your bug, with setting dependency to this bug. I can't think your case is absolutely same problem as this bug, but I can't think your case is absolutely different problem from this bug.

Comment 13

[Charles]

I think your suggestion is premature - I don't think my problem is different, but I'm curious to see if the bug reporter notices the 1K size in the size column...

I think the fact that the resulting 'bad' attachments are exactly 27 bytes in size in both of our cases is pretty strong evidence that it is the same, or at least a strongly related bug.

Also, maybe you missed that I'm fairly sure that 3.1rc1 doesn't exhibit this problem - if this is true, then, as much as I'd like to - and as much as I hate not knowing what caused a problem - I really can't spend any more time trying to troubleshoot a problem that is already fixed.

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #13)
> I think the fact that the resulting 'bad' attachments are exactly 27 bytes in
> size in both of our cases is pretty strong evidence that it is the same, or at
> least a strongly related bug.

Interesting.
Bug opener says "*all* attachments were saved as this 27 bytes file." It may be an important factor of this bug and your bug.
But, "funny 1KB size" is involved in your case although such funny phenmenon is not seen in this bug's case.
Charles Marcus, let's analyze your case in your bug, with caring for incident of "exactly 27 bytes" always.

Comment 15

[Charles]

(In reply to comment #14)
> Bug opener says "*all* attachments were saved as this 27 bytes file." It
> may be an important factor of this bug and your bug.

I agree...

> But, "funny 1KB size" is involved in your case although such funny phenmenon
> is not seen in this bug's case.

He didn't say he didn't see a 1K size, he said he hadn't *checked* it - ie, I read his reply to mean he hadn't noticed one way or another... and I thouight I had suggested to him to check it next time it happens, but I see I didn't... :(

[e-mail] > can you confirm/deny this next time it happens?

> Charles Marcus, let's analyze your case in your bug, with caring for incident
> of "exactly 27 bytes" always.

Maybe you misunderstood - in *all* cases, the attachments, if/when saved out to the local filesystem, are 27 bytes in size. '1K' is what is displayed in the size *column* in the message list pane - apparently TB cannot display anything smaller than that?

Comment 16

[Ness]

If I may jump in here...

My problem is not with 27b or 1K or similar.
Sometimes, but rarely, I can see that attachment is not of the size it should be, but mostly attachments ARE in exact size as I can see and open it directly on server, via web mail.

The most usual error message I'm getting is related to impossibility to read some location in C:/Doc&Sett/User/App data/Temp, but without any additional message which could lead to problem solution...

If press "Help" or "Troubleshooter" button, all I can see is info that my AV soft maybe blocking this attachment.
OK, I was tried with uninstalling my AV, including registry clean up to be sure that all AV files are erased, but - no results. Still couldn't open attachments.

Comment 17

[mozilla]

(In reply to comment #15)

> He didn't say he didn't see a 1K size, he said he hadn't *checked* it - ie, I
> read his reply to mean he hadn't noticed one way or another... and I thouight I
> had suggested to him to check it next time it happens, but I see I didn't... :(
> 
> [e-mail] > can you confirm/deny this next time it happens?

Sure. You are right, I can't confirm/deny that perhaps sometimes it actually showed 1kb since I never really looked at this column.

However, it ALWAYS created the 27 bytes files, which I understand it is the same case you have.

The best clue on this is the "This body part will be downloaded on demand." in the message source. This suggests that the body was not saved to the synchronization store, but was marked as downloaded.

Comment 18

[mozilla]

(In reply to comment #16)

> The most usual error message I'm getting is related to impossibility to read
> some location in C:/Doc&Sett/User/App data/Temp, but without any additional
> message which could lead to problem solution...

Ness, did you try to save the file instead of opening it? Did it create a 27 byte file?

Comment 19

[Ness]

Sure. All types of files, but as they react when opened directly from TB, the same was if they are saved, with the same following message.

I've tried with checking option "Leave message on server", so I can download it again and again, but always the same. Doesn't matter am I tried to open it or save it as first action.

Comment 20

[Charles]

(In reply to comment #16)
> If I may jump in here...
> 
> My problem is not with 27b or 1K or similar.
> Sometimes, but rarely, I can see that attachment is not of the size it should
> be, but mostly attachments ARE in exact size as I can see and open it directly
> on server, via web mail.
> 
> The most usual error message I'm getting is related to impossibility to read
> some location in C:/Doc&Sett/User/App data/Temp, but without any additional
> message which could lead to problem solution...
> 
> If press "Help" or "Troubleshooter" button, all I can see is info that my AV
> soft maybe blocking this attachment.
> OK, I was tried with uninstalling my AV, including registry clean up to be sure
> that all AV files are erased, but - no results. Still couldn't open
> attachments.

So maybe your problem is different then... <shrug>

Comment 21

[Charles]

(In reply to comment #17)
> (In reply to comment #15)
> 
>> He didn't say he didn't see a 1K size, he said he hadn't *checked* it -
>> ie, I read his reply to mean he hadn't noticed one way or another... and
>> I thouight I had suggested to him to check it next time it happens, but I
>> see I didn't... :(
>> 
>> [e-mail] > can you confirm/deny this next time it happens?
>
> Sure. You are right, I can't confirm/deny that perhaps sometimes it actually
> showed 1kb since I never really looked at this column.

Ok, good, we'll wait and see if you can confirm this later...

Also, when you do, please note what type of document it is... I have a vague feeling that PDF's did *not* show as 1K, but .doc and .xls files did...

> However, it ALWAYS created the 27 bytes files, which I understand it is the
> same case you have.

Yes, exactly.

> The best clue on this is the "This body part will be downloaded on demand." in
> the message source. This suggests that the body was not saved to the
> synchronization store, but was marked as downloaded.

Yes, my users tell me they see the same thing, and I have visually confirmed it when they come get me before they fix it (by moving the message to the Temp folder I have created for them to work around this problem until it is fixed).

Comment 22

[Charles]

(In reply to comment #19)
> Sure. All types of files, but as they react when opened directly from TB, the
> same was if they are saved, with the same following message.

Ness, you didn't answer the question... did you:

1. Save the file to your local filesystem (Desktop or wherever)?

2. If so, was the resulting file size exactly 27 bytes?

Comment 23

[Ness]

I did replied Marcus...

Doesn't matter what is the size of attached file in TB, after saving to computer's root, any directory, the size remain the same, but file still cannot be opened and I'm getting the same messages as when I've tried to open it from TB.

Or, in case that file (e.g. picture) can be opened in TB, but it's not visible completely but just half of it, identical situation happen if I save it to some location at computer.

Comment 24

[Ness]

That's the main reason why I was posted bug with title "TB DESTROYING attachments...".

Seems like attachments are somehow "crashed" during entering in TB's Inbox from server...

Comment 25

[WADA:World Anti-bad-Duping Agency]

IMAP log will be required for diagnosis. Mixing of log data for your case and Charles Marcus's case in this bug makes problem analysis confusing. So I've re-opened your bugs. 
Ness and Charles Marcus, watch your bugs each other please. And, please write about "what is same as this bug" and "what is different from this bug" in your bugs, with analyzing and watching this bug. And, please attach IMAP log, data in offline-store etc. in your case to your bug instead of this bug.

Comment 26

[WADA:World Anti-bad-Duping Agency]

Returning to original report of comment #0 by bug opener of this bug and comment #3 which is answers to questions in comment #2.

(In reply to comment #3 by bug opner)
> Q1. The "View Source" was grayed (disabled) when I went to offline mode.
> Q2. 
>  a. The problem was not reproduced.
>  b. The message is shown fine

"multipart/mixed with non-inline attachements" is same test case as bug 565852(it's for problem with offline-use=off). If download into offline-store file is not completed when offline-use=on(e.g. close Tb while big mail is being downloaded), text of bug 565852 #12 is shown, and View/Message Source is grayed out.
As "View Source" was grayed in your case, status of mail looks "download is not completed yet".

(Q5) What was shown at message pane for the mail while working offline?

If attachment part is not downloaded yet into offline-store file, it should be status of "download is not completed" and it should be true because "View source is grayed out" while working offline. So, fetch for the attachment part should be requested when save of the attachment is requested.
When you clicked new mail in IMAP folder of "offlie-use=on", if download of whole mail data by auto-sync is not invoked yet, mail is downloaded to DisK Cache to show mail body(same as folder of offline-use=off), so, expected  phenomenon is bug 565852 like one. But phenomeno is one written in comment #0.
As you can't reproduce problem with the mail in new small IMAP folder, it may be contenstion between "Downloading to Disk Cache because mail is not aoto-sync'ed yet" and "Downloading to offline-store file for auto-sync". However, it's hard to know what happened because you can't reproduce problem by simple test scenario.
We look better to focus on "*ALL* attachments were saved as this 27 bytes file" first, which is common among three bugs.

(Q6) Did you "save all" type "Save attachment"? If you do "Save" for each attachment, does problem occur? Any of saved file is 27 bytes?

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #17)
> However, it ALWAYS created the 27 bytes files, which I understand it is the
> same case you have.
> The best clue on this is the "This body part will be downloaded on demand." in
> the message source. This suggests that the body was not saved to the
> synchronization store, but was marked as downloaded.

(Q7) Where could you see the "This body part will be..."? By View/Message Source?

I suspected such data because "ALWAYS 27 bytes", but I can't guess why data becomes data you atached to this bug. But I forgot that it was PDF attachment part - base64 encode usually...
I could see same data by following crafted mail data.
> --------------060104010306070103050204
> Content-Type: application/pdf;
>  name="This body part will be downloaded on demand.PDF"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
>  filename="This body part will be downloaded on demand.PDF"
>
> This body part will be downloaded on demand.
> --------------060104010306070103050204--

Comment 28

[WADA:World Anti-bad-Duping Agency]

Confirming per attached data and test result with crafted mail.

Comment 29

[mozilla]

Hi,

When trying to check for your Q5, suddenly the attachments were fine! It seems that TB this time understood that the attachments were not downloaded, and downloaded them!
So I guess we can narrow the problem even further - this problem is might not be the state of the attachment, but that in certain cases TB might ignore this state and save the attachment anyway.

A6 - All files were saved with 27 bytes.
A7 - Yes, in your first comment you instructed me to do a view source, so I checked and that's when I saw this... 

Great catch!!! You found the cause of the mysterious 27 bytes...

Comment 30

[WADA:World Anti-bad-Duping Agency]

Bug 534835 is quirks for incorrect(shorter) rfc822.size by MS Exchange or some servers. I thought this kind of quirks works in any not-fully-downloaded-yet data cases. But it seems wrong. In this bug's case, mismatch of length probably doesn't happen. I guess some flags are wrongly set in special occasion or some special combinations of flags are mis-interpreted. "Working Offline" and some actions during offline mode looks to have corrected error prawn status fortunately in your case.

Comment 31

[Charles]

(In reply to comment #23)
> I did replied Marcus...

You replied, but didn't answer the question - and you still didn't answer the question.

But apparently WADA has made some progress...

I'll wait and see if anything comes out of this that I can test...

Thanks for sticking with us on this WADA... I have literally had people screaming to go back to Thunderbird 2, because it never had this problem...

Comment 32

[Charles]

(In reply to comment #25)
> Ness and Charles Marcus, watch your bugs each other please. And, please write
> about "what is same as this bug" and "what is different from this bug" in your
> bugs, with analyzing and watching this bug. And, please attach IMAP log, data
> in offline-store etc. in your case to your bug instead of this bug.

I'll do my best, but as I have said, I'm sorry to say I rarely see the problem on my PC, and our users don't always tell me when they experience the problem. The one user who has been experiencing it the worst (dozens of times per day she says) won't let me uninstall 3.1rc1 and go back to 3.0.4 because she says it is now fixed and she is too busy to help me troubleshoot a problem that has already been fixed (she's a Sales Droid)...

Since it *is* apparently fixed in 3.1rc1, I find it hard to argue with her...

Comment 33

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #32)

Charles Marcus, as you are only person who could observe "it spontaneously changes to 1KB", and could check Tb3.1rc1 with your some users, and can get feed back from your users(no response is also a feed back of "problem doesn't happen so frequently than before"). I think "it spontaneously changes to 1KB" is a hint to what kind of someting wrong happens. I want to analyze the "spontaneously changes to 1KB" in your bug, and to watch whether Tb3.rc1 is effective workaround or not. And, if IMAP log will be required, log is better to be analyzed in separated bug.

Comment 34

[Charles]

(In reply to comment #33)
> (In reply to comment #32)
> 
> Charles Marcus, as you are only person who could observe "it spontaneously
> changes to 1KB",

So far - maybe no one else ever noticed it...

> and could check Tb3.1rc1 with your some users, and can get
> feed back from your users(no response is also a feed back of "problem doesn't
> happen so frequently than before").

No, around here, no response is just as likely to mean "if I tell Charles, he'll just ask me a bunch of questions and want to troubleshoot the problem, so I just won't tell him and will simply move the message to a different folder like he showed me to work around the problem."

> I think "it spontaneously changes to 1KB" is a hint to what kind of
> something wrong happens. I want to analyze the "spontaneously changes
> to 1KB" in your bug, and to watch whether Tb3.rc1 is effective workaround
> or not.

I asked her again today, and she said everything is still fine, and she is adamant that it is now fixed, as she says it would have happened many dozens of times by now.

> And, if IMAP log will be required, log is better to be analyzed
> in separated bug.

Meaning my bug, yes, I agree...

Still waiting to hear from mozilla@flowsoft if can confirm the size changing to 1K, which (I guess) would mean we could close my bug again...

Comment 35

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> I have noticed that this might happen when the message was selected,
> and I selected another message before the original message was
> downloaded completely.
> When returning back to the original message that has the
> attachments, all attachments were saved as this 27 bytes file.
> Restarting thunderbird has no effect over this.
> It seems that the cache of the message gets corrupt and does not get fixed.

> This bug means that the cache does not make sure that the attachment was
> downloaded correctly before it is being saved to the cache.

(In reply to comment #3)
> Q2. 
>  a. The problem was not reproduced. However, I think it might be related to
> that when I open Thunderbird for the first time, and it starts downloading the
> Inbox folder, and I move around the messages before it completes the load of
> the entire folder.

> It seems more like a thunderbird "synchronization" problem
> - the attachments are marked as synched even though they are not.

Returning back to origial phenomenon of comment #0 reported by bug opener of this bug.

Partial jpeg issue sounds same phenomenon to one I reported to bug 572974 with offline-use folder, with big mail(mail of 12MB and 20MB is used in test), with mail.server.default.mime_parts_on_demand=false to force download of whole mail data. As you use slow connection and your setting is "Display attachment inline is checked", similar situation to that bug can happen if mail of jpeg atachments. That bug is "downloded to Disk Cache" case and this bug is "download to offline-store file" case. But I guess similar problem can occur. 

If mail is viewed before download by auto-sync, I guess that "download on demand" is applied. If it's right, similar situation can happen just after an attachment part is written as "this part will be downloaded..." for PDF attachments, because PDF attachment part is not downloaded by "download on demand".

Comment 36

[WADA:World Anti-bad-Duping Agency]

Correction. offline-use=off is used in test for bug 572974.

Comment 37

[WADA:World Anti-bad-Duping Agency]

If auto-sync is running on IMAP folder when you click a new mail in the IMAP folder, auto-sync is halted and mail data is downloaded to Disk Cache with "Download on demand". If similar problem to bug 572974 haapens on the mail data in Disk Cache, "27 bytes file" and "This part will be downloaded..." by View Source can occur.
[e-mail](bug opener), see bug 572974 comment #2 for it, please.

Comment 38

[WADA:World Anti-bad-Duping Agency]

xre bug 572974 for ease of search and tracking.

Comment 39

[mozilla]

Quick update, happens with 3.1 as well. The "size" column showd 1k and not the actual size.

Comment 40

[Charles]

I have updated every person that has been complaining, and have repeatedly asked if they have had any other occurrences, and none have.

One thing I did do as a part of the upgrade was delete everyone's entire local IMAP stores and let TB re-download everything from scratch...

Did you do this? Or at least 'Repair' all of the folders? Maybe it is possible for certain messages to not fix themselves, but once they are fixed they stay fixed?

Comment 41

[mozilla]

(In reply to comment #40)
> I have updated every person that has been complaining, and have repeatedly
> asked if they have had any other occurrences, and none have.
> 
> One thing I did do as a part of the upgrade was delete everyone's entire local
> IMAP stores and let TB re-download everything from scratch...
> 
> Did you do this? Or at least 'Repair' all of the folders? Maybe it is possible
> for certain messages to not fix themselves, but once they are fixed they stay
> fixed?

Hi,

It happened on a new message that was received from the server by TB 3.1, so repair has nothing to do with it I believe...

Comment 42

[Charles]

As I said... I have had ZERO problems like this any more, so you just migbht want to at least *try* the same things I did:

DELETE ALL LOCAL IMAP STORE/CACHES

Then, once you've got your folders reconfigured the way you like, see if it happens again...

Until you do this, you simply cannot be sure that it isn't some weird corruption in the local imap store.

Comment 43

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #39)
> Quick update, happens with 3.1 as well. The "size" column showd 1k and not the
> actual size.

I could observe "changing to size=1KB" by similar steps to bug 572974 utilizing bug 565852.
- Offline-use=off, mail.server.server2.mime_parts_on_demand=true
- Clear Disk Cache, view each mail
- View a multipart mail with PDF atachments, and view a non-multipart mail
  => bug 565852 occurs.
- View the multipart mail with PDF atachments
  => Fetch for first part (mail body) is issued.
     Subject at message header box is changed.
     Mail body text is shown at message pane.
     Downloading is shown, progress bar is shown.
- Pull off of LAN cable, and plug in Lan cable again
  => size at thread pane is changed to 1KB.

mail.server.server2.mime_parts_on_demand=true version of bug 572974.

Comment 44

[WADA:World Anti-bad-Duping Agency]

Note:
In my above test with Tb 3.1, view source showed whole mail data although size column is still 1KB(real mail data size=12MB). So my test result is not absolutely same as yours. I couldn't always see problem by same test. Timing is probably relevant.

Comment 45

[WADA:World Anti-bad-Duping Agency]

Simplest recovery procedure:
  Move the troubled mail to other mail folder, then move back.

Comment 46

[WADA:World Anti-bad-Duping Agency]

As you use offline-use=on, following may be a workaround, or a way to reduce frequency of your problem.
> browser.cache.disk.enable=false & browser.cache.memory.enable=false
See bug 565852 comment #15.

Comment 47

[Charles]

(In reply to comment #46)
> As you use offline-use=on, following may be a workaround, or a way to reduce
> frequency of your problem.
> > browser.cache.disk.enable=false & browser.cache.memory.enable=false
> See bug 565852 comment #15.

What are the disadvantages of this? Does this cause major performance impact?

Comment 48

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #47)
> What are the disadvantages of this? Does this cause major performance impact?

When auto-sync is enabled and offline-use=on folder, Cache(disk cache/memory cache) is ordinally not used. Even if cache is used due to pause of auto-sync by clicking of a new mail which is not auto-sync'ed yet, first download is executed with download-on-demand. So, there is no performance issue usually.
Only problem is re-download of whole big mail occurs when mail of big/many inline-displayable attacments is clicked again with "Display Attachments Inline", before mail is download by re-scheduled auto-sync.

As for "performance impact" when auto-sync=enabled/offline-use=on, the workaround will usually improve performance of "re-download by clicking of mail again before download by re-scheduled auto-sync" as far as that only small part is re-downloaded by download-on-demand, because the workaround resolve nearly all problems of bug 565852.

Comment 49

[WADA:World Anti-bad-Duping Agency]

Note:
Above is for mail/news only. RSS feeds, HTTP(start page, remote image) etc. may be affected.

Comment 50

[Charles]

(In reply to comment #48)
> As for "performance impact" when auto-sync=enabled/offline-use=on,

Actually, I have auto-sync=*disabled*/offline-use=on, so is that scenario covered by the workaround?

This and the related bug 565852 is very confusing...

First, is Tools > Options > Advanced > Network & Disk Space >  Disk Space: Use up to: set to 0 the same as setting browser.cache.disk.enable=false?

Second, is there a GUI equivalent of browser.cache.memory.enable=false?

I have seen this problem only a very few times since upgrading everyone to 3.1, so it is difficult to tell if changing a setting 'fixes' it or not...

Comment 51

[Charles]

(In reply to comment #50)
> (In reply to comment #48)
>> As for "performance impact" when auto-sync=enabled/offline-use=on,

> Actually, I have auto-sync=*disabled*/offline-use=on, so is that scenario
> covered by the workaround?
> 
> This and the related bug 565852 is very confusing...
> 
> First, is Tools > Options > Advanced > Network & Disk Space >  Disk Space: Use
> up to: set to 0 the same as setting browser.cache.disk.enable=false?
> 
> Second, is there a GUI equivalent of browser.cache.memory.enable=false?

A response would be helpful. It is difficult to contribute to resolving this very troublesome bug without reasonably timely answers to questions like this.

Comment 52

[David :Bienvenu]

I don't really understand the question, but you can toggle the browser.cache.memory.enable pref in the config editor.

Comment 53

[Charles]

There are 3 questions actually, but I'm confused as to why you don't understand the question (presumably you meant the last one based on your reply)...

Is there or is there not a GUI based option (somewhere in Tools > Options or Tools > Account Settings) that is the equivalent of setting browser.cache.memory.enable=false?

As for the other questions...

1. Does the workaround as described in bug 565852 comment #15

(boils down to setting both:
browser.cache.disk.enable=false
browser.cache.memory.enable=false)

also apply if I have auto-sync=disabled/offline use=on (because the workaround ?

2. Is setting the GUI option

Tools > Options > Advanced > Network & Disk Space

  Disk space

  Use up to: [ 0 ]

the same as setting browser.cache.disk.enable=false?

It seems to slowly be getting worse again - or maybe people are just getting frustrated enough to complain...

This bug really, really, really needs to get squashed, and squashed permanently. TB2 *never* had this problem, ever, in the 9+ years we've been using TB (40 - 75 PCs during this time), and people are getting frustrated, and starting to curse Thunderbird, and this has never happened either.

Asgain, all of these related bugs that WADA has been trying to wrangle are very confusing. Someone really, REALLY needs to take ownership of the overall issue, take the bull by the horns and figure out what is going on. This is a very serious problem, and both myself and our users have lost a *lot* of confidence in TB because of this.

Comment 54

[David :Bienvenu]

No, there's no gui option for setting the memory cache size, that's why you have to use the config editor; if there was a gui option, then you would not have to use the config editor, if that makes any sense.

disabling the browser disk cache is most likely the same as setting the size to 0, though that's a question for the network/necko folks, not me.


sorry, can't parse this:
also apply if I have auto-sync=disabled/offline use=on (because the workaround
?

but all these settings are orthogonal/independent - disk cache size, memory cache size, autosync, offline flag for folders, etc.

Comment 55

[Charles]

(In reply to comment #54)
> No, there's no gui option for setting the memory cache size, that's why you
> have to use the config editor; if there was a gui option, then you would not
> have to use the config editor, if that makes any sense.

This is why I hate 'workarounds' that require setting things in about:config - you end up totally forgetting what you did and didn't change, and when - it makes troubleshooting this stuff a freaking nightmare.

But yes, it does make sense - I'm just trying to wrap my head around these different/confusing options.

> sorry, can't parse this:
> also apply if I have auto-sync=disabled/offline use=on (because the workaround
> ?

Yeah, sorry, I altered that text before posting and forgot to complete that part, although it isn't that hard to figure out - the workaround I referenced above that comment was for a situation where the user had auto-sync=enabled, but I don't.

> but all these settings are orthogonal/independent - disk cache size, memory
> cache size, autosync, offline flag for folders, etc.

Maybe, but setting those two settings (did you even read the referenced bug comment?) is what it took to 'fix' the problem.

Sorry if my frustration is showing through again, but I don't know HOW to troubleshoot this problem to resolution, or I would, even if it took me a week or a month.

I'm just trying to make it clear to you guys how serious of an issue this is.

I'd feel much better if whatever developer is in charge of this particular part of the code would at least *say* something to let us know he is 'on it'. As it stands, it seems like there is no movement on any of these bugs for a while now, and I really cannot allow our users to continue using Thunderbird if these bugs are going to hang around for years (like many other bigs - HTML compose bug anyone?) because no one will simply step up and take ownership of them and FIX them.

Comment 56

[David :Bienvenu]

We can't reproduce this problem; if we had steps to reproduce it, then we would have a chance of fixing it...

Comment 57

[Charles]

Apparently WADA has been able to do so - if I'm reading all of his comments in this and the related bugs discussed here.

And your last comment was 'it would be a while before you could look at it' (bug 565852 comment #14)... nothing after that indicating you had tried bu tcouldn't reproduce it and asking for more info...

That is what I mean - someone needs to 'take ownership' of this, and really start digging. WADA has made a lot of effort in these bugs showing steps to reproduce the problem(s), so I really don't think 'we can't reproduce it' is a valid response at this point - unless you're saying you have exhaustively attempted to painstakingly follow WADAs instructions for reproducing it and have failed?

If necessary, I'll be happy to install a special debug-enabled version on one of my users systems that is having the problem and make sure they know to let me know every time they encounter the issue, and will then provide whatever debug output I can.

Comment 58

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #56)
> We can't reproduce this problem; if we had steps to reproduce it, then we would have a chance of fixing it...

I could observe all of next in bug 572974, by pulling off of LAN cable while downloading.
(a) Size at thread pane is spontaneously changed to partial data size.  
(b) Partial data in Disk Cache is used as downloaded mail.
(c) Even though error occurred, re-download is not initiated.
(d) In Work Offline Mode, View/Message Source is grayed out.
    (this looks different from original comment #0)
As I wrote in comment #43, I could see (a) of this bug(changed to 1KB) once but only once. I couldn't check "save as" when I wrote comment #43, because of  bug 572065.

David, can you check bug 572974 first which is far easier to reproduce than this bug?
I guess root cause is cause of bug 565852. "disabling of memory/disk cache" is workaround of bug 565852.

Comment 59

[lapsus63]

Hi,

I come from #548507 but I've been redirected here (by WADA, nice to see you again :)), where the topic seems to be more adapted. I'd like to inform you that I've configured an auto-forwarding in my thunderbird to a gmail account.
Each time I receive an attachment (pdf, odt, xlsx for example), I can get it from Thunderbird where the mail has been received, but in my gmail account I always receive a 27 bytes attachment too.
I have an IMAP account too.

Here is my build ID : Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12)
Gecko/20101027 Lightning/1.0b2 Thunderbird/3.1.6

Thanks for all.

Comment 60

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #59)
> I'd like to inform you that I've configured an auto-forwarding in my thunderbird to a gmail account.
> Each time I receive an attachment (pdf, odt, xlsx for example), I can get it
> from Thunderbird where the mail has been received, but in my gmail account I
> always receive a 27 bytes attachment too.

Very interesting.
You look able to reproduce this bug's phenomenon very frequently by your particular setting, your auto-forwarding. "Fetch by forward mail of message filter" seems to produce similar condition to "new mail click just after mail arrival and just after auto-sync tries to fetch whole mail data of the new mail to offline-store".
I'll try it with my Gmail IMAP account, or my Yahoo! IMAP account(see bug 493064 and bug 583602 comment #11 for US Yahoo! case, please), or my ordinal IMAP account of my ordinal ISP.

> but in my gmail account I always receive a 27 bytes attachment too.

If you will meet problem of "27 bytes attachment(s)" in forwarded mail to your Gmail account, check message source or saved .EML file content of original mail at your account who forwarded the mail to your Gmail account, please.

Comment 61

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #59)
> Each time I receive an attachment (pdf, odt, xlsx for example),
> I can get it from Thunderbird where the mail has been received,
> but in my gmail account I always receive a 27 bytes attachment too.

Attachments are normally saved if mail in folder of account who received original mail?
Problem on forward mail to Gmail only?
If this bug(offline-use=on) or bug 572974(ofline-use=off), 27 bytes problem should occur on original mail too.

Your case sounds phenomenon of "forward by filter uses partial mail data in Disk Cache". (partial in this context: multipart on demand is effective and data of PDF part is not fetched yet, because it's not shown in inline).
It may be phenomenon of "forward by filter doesn't invoke fetch of part which is attached to forward mail". It may be a result of change from "forward as attachment always" to "forward as inline, if user's setting is inline forward ".

[e-mail], open separate bug for your forward case, please, with attching both original mail and formard mail to bug(never paste, please), if 27 bytes problem doesn't occur on original mail.

Comment 62

[WADA:World Anti-bad-Duping Agency]

This bug is offline-use=on case, so Bug 531033 may cause this bug. Setting dependency for ease of tracking.

Comment 63

[lapsus63]

(In reply to comment #60)
> If you will meet problem of "27 bytes attachment(s)" in forwarded mail to your
> Gmail account, check message source or saved .EML file content of original mail
> at your account who forwarded the mail to your Gmail account, please.

I tried but maybe too late, the whole content was fetched (viewing source). I'll try on the next auto-forwarded message.

Comment 64

[lapsus63]

FYI, I've been to tools > filters and edited my filter.
I've several options (combo box) :
Apply filter :
( ) When receiving mail
( ) during manual execution
(X) When receiving mail or manual execution
( ) checking mail (after ordering)
( ) checking mail (after ordering) or manual execution.

My translation is probably an average one.
So, I was onto "When receiving mail or manual execution", I've switched to the last option. I'll tell you if it's better.
I clicked on "Execute" on the window containing the filter list, my mail have been forwarded with full attachments this time.

I'm now certain the forward set up as a filter is launched too early, so that thunderbird has no time to fetch the content of the attached parts before forwarding the mail.

Hope this helps...

Comment 65

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #63)
> I tried but maybe too late, the whole content was fetched (viewing source).

It is an evidence that your case is not this bug. If this bug, view source of original should show partial mail data(data in pdf attachment part is string of "This body part will be downloaded on demand.").

> I'm now certain the forward set up as a filter is launched too early,
> so that thunderbird has no time to fetch the content of the attached
> parts before forwarding the mail.

I believe problem is in forward by filter, but I don't think timing related issue. I guess problem like next, mismatch between "forward of Tb" and current spec/implementation around "Disk/Memory Cache".
  Upon forward, mail is not downloaded to offline-store yet by auto-sync.
  So, fetch is similar to offline-use=off folder case, and downloaded in Disk
  Cache. And, as multiart/mixed mail with pdf part(not shown in inline),
  "multipart on demand" is applied and "fetch body[1]", fetch body[2.1]" etc.
  is issued instead of "fetch body[]"(request of whole mail data).
  At this step, pdf part of mail data in Disk Cache is "This body part will be
  downloaded on demand.".
  This data in Disk Cache is marked "partial" because whole mail data is not
  held. So, current Tb always issues "fetch body[1]" upon next mail display by
  mail click if offline-use folder, and issues "fetch body[M.N]" if pdf part
  is requested by user.
  I guess forward used this partial data as whole mail data.

Anyway, open separate bug for your forward case, please, with crisp problem descrition, with attaching forwared mail to Gmail(file saved as .EML file). No need of original mail(whole mail data) for initial analysis.
[e-mail], if possible, check with next setting before bug open, please. (via Config Editor, restart Tb after change to avoid unwanted issue)
  mail.imap.mime_parts_on_demand           : true -> false
  mail.server.default.mime_parts_on_demand : true -> false
By this setting, Tb always issues "fetch body[]" even if offline-use=off. So "download of mail data for forward" will download whole mail data too.
If you checked with this setting change, report it in the separate bug instead of this bug, please.

Comment 66

[Ian Neal]

I'm getting this issue too, the source says "This body part will be downloaded on demand." but it never is, just 27 bytes (and yes my size column says 1k, though does say the correct size after a summary rebuild until I click on the message and then it reverts back to 1K).
If I set both of the mime_parts_on_demand prefs to false and then rebuild the summary then the misbehaving message is fixed. The backend server in my case is Netscape Messaging Server.

Comment 68

[Sergey Kochkarev]

I also have the same problems with IMAP and Thunderbird 6.0.2. My mail recepients receive emails with corrupted attachments when filter forwards incoming mail to them.

When I manually forward these emails, no attachments are corrupted (PDF files).

My mail server is hmailserver. PDF is associated with Foxit Reader.

Is it true that POP3 users do not have these problems?

Comment 69

[Charles]

(In reply to Sergey Kochkarev from comment #68)
> I also have the same problems with IMAP and Thunderbird 6.0.2.

Ummmm... 6?? Thunderbird is currently at version 15...

Your efforts to participate with the bug hunting process is appreciated, but you should always try to use the most recent version unless specifically asked to use a different one. Reporting behavior of an ancient version is not very helpful.

Please upgrade and try again.

Thanks

Comment 70

[Sergey Kochkarev]

The bug is open and critical. Can you give me a guarantee that it was fixed in 15?

Comment 71

[Charles]

(In reply to Sergey Kochkarev from comment #70)
> The bug is open and critical. Can you give me a guarantee that it was fixed
> in 15?

You're not serious?

1. There are no guarantees in life, and

2. Reporting bugs against an ancient piece of software is a waste of your and everyone else's time.

Comment 72

[Rob Janssen]

We have recently updated from SeaMonkey 2.10.1 to 2.13.1 in our company (we don't track each and every update), and from what I hear from the helpdesk this issue has raised its ugly head again.
We have seen the "27 byte file when saving attachment" issue before, and it other results (error messages from Word or PDF viewer when opening attachments directly from mail).
We use UW IMAPD and I have already set a locked preference to mitigate the issue:
lockPref("mail.imap.mime_parts_on_demand_threshold", 500000);

However, it still happens for larger messages and I think its frequency has increased with the latest update (when that is not a coincidence).

It looks like the problem is worsened by my efforts to avoid caching of messages on the local filesystem (which is a critical security and privacy issue):

lockPref("mail.server.default.offline_download", false);
lockPref("mail.server.default.autosync_offline_stores", false);
lockPref("offline.autoDetect", false);
lockPref("offline.download.download_messages", 0);
lockPref("offline.send.unsent_messages", 0);
lockPref("offline.startup_state", 2);

The mail program should be able to operate with an IMAP server and no local caching of messages, even when that would be so much more efficient.

Comment 73

[Zack]

17.0.8 on Windows 7 32bit, still having this same problem. Filters work fine for emails that don't have attachments. Any time there's an attachment, the email minus the attachment is sent. 

My wife constantly complains about it, so I'm just relaying some of her frustration.

Most of the attachments that are run through the filter are 74KB PDFs. But I've seen the same issue with other file types and sizes.

My extensions are Copy Folder 1.3, Remove Duuplicate Messages 0.1.13 and Test Pilot for Thunderbird 1.3.9. Plugins include Foxit Reader, Google Update, Java, Picasa, Shockwave Flash, VLC Web Plugin. 

I've tried in safe-mode and get the same result.

Comment 74

[Kevin]

WADA -

Please note, I had same error reported by lapsus63. Your comment 65 resolved error 100%.

  mail.imap.mime_parts_on_demand           : true -> false
  mail.server.default.mime_parts_on_demand : true -> false

Why is this not default setting in next version of TB update???

I see many posts on other blogs about same error, and nobody knows how to resolve, such an easy fix.

Thank you.

Comment 75

[Charles]

(In reply to Kevin from comment #74)
> WADA -
> 
> Please note, I had same error reported by lapsus63. Your comment 65 resolved
> error 100%.
> 
>   mail.imap.mime_parts_on_demand           : true -> false
>   mail.server.default.mime_parts_on_demand : true -> false
> 
> Why is this not default setting in next version of TB update???

What version are you running?

I used to have this problem with older versions, but haven't had this pref set for some time now...

I don't think there is a need to have it be the default on latest versions.

Comment 76

[Christian Schmidt-Guetter]

Applying this configuration switch (*.mime_parts_on_demand ==> false) does solve some of my Thunderbird issues.

Many thanks.

Comment 77

[Rob Janssen]

Of course this change causes a severe performance hit, especially when you have a slow
link to your IMAP server and receive large attachments.
There should be a solution without setting those values.
I think when the maintainers cannot locate the issue, they should install a workaround
so that whenever the attachment contains just the "will be downloaded on demand" text
(for whatever reason), it will be forced to be actually downloaded when that is required.

Comment 78

[Zack]

Just tried it. Worked for me as well. I happened to get an email that had an attachment and a filter, today. Email on the other end was "broken" as she says. 

Made the *.mime_parts_on_demand ==> false change and reran the filter on just that message, worked fine.

Thunderbird 17.0.8 Windows 7 32bit

Comment 79

[franjb68]

I agree with Rob Janssen (Comment 77): download of the attachments should be forced when forwarding.

Thunderbird 31.1.2 Windows XP SP3 still has the problem.

Comment 80

[Piotr Szymkowski]

I have same problem.

Set up filter to forward messages with attachments.

HTML+TXT+Attachment message is being forwarded and only HTML is send with 27 bytes attachment file

Comment 81

[Firefix]

Do the changes reported in comment 65 of this bug report also fix the problem retrospectively? I still seem to have a problem with attachments of emails to which filters have already been applied. Is there a way to fix that retrospectively?

Comment 82

[Adrien]

#81 - Those emails are already forwarded. It's temporally impossible to 'fix' them. The only thing you can do is change the setting and re-forward those emails.

Comment 83

[fvroman]

Created attachment 8859644 [details] [diff] [review]
Fix the issue when auto forwarding (using filters) mails with attachements

Comment 84

[Kent James (:rkent)]

The patch looks ok on the surface, but I cannot really reproduce the original problem. There are a lot of comments in the bug, and it was not clear to me the exact required conditions. Could someone give me a summary of what it takes to reproduce this?

Comment 85

[fvroman]

The bug is easy to reproduce:
* Set a filter rules which auto forward incoming email
* Receive an email with an attachment 
* Check the forwarded email in the destination box (may be yours)
=> the attachment does not have the right size (it is always 27bytes)
=> Whatever the original attachment, the content is always the same "Thisbodypartwillbedownloadedondemand" (convert the binary content to base64)

In fact, when auto forwarding occurs the attachments are not in cache and there is no request to download them before sending the email.

Comment 86

[fvroman]

(In reply to fvroman from comment #85)
> The bug is easy to reproduce:
> * Set a filter rules which auto forward incoming email
> * Receive an email with an attachment 
> * Check the forwarded email in the destination box (may be yours)
> => the attachment does not have the right size (it is always 27bytes)
> => Whatever the original attachment, the content is always the same
> "Thisbodypartwillbedownloadedondemand" (convert the binary content to base64)

=> sorry I have made a mistake it is not "convert the binary content to base64" but do a base64 decoding (base64 -d)

> 
> In fact, when auto forwarding occurs the attachments are not in cache and
> there is no request to download them before sending the email.

Comment 87

[Kent James (:rkent)]

Comment on attachment 8859644 [details] [diff] [review]
Fix the issue when auto forwarding (using filters) mails with attachements

Jorg, I'll be away for a few days, and this is in the compose code anyway. Could you look at this?

Comment 88

[Jorg K (GMT+1)]

Comment on attachment 8859644 [details] [diff] [review]
Fix the issue when auto forwarding (using filters) mails with attachements

I could reproduce the problem and the patch fixes it. The patch looks good to me.

Please note that we do not use tab characters, spaces only. I corrected that. I'll land the patch later today.

Comment 89

[Jorg K (GMT+1)]

And thank you for the contribution! I can see that you have more patches lined up.

Comment 90

[fvroman]

Sorry for the formatting my editor configuration was not correctly set.(In reply to Jorg K (GMT+2) from comment #89)
> And thank you for the contribution! I can see that you have more patches
> lined up.

Sorry for the formatting my editor configuration was not correctly set.

Comment 91

[Jorg K (GMT+1)]

(In reply to Jorg K (GMT+2) from comment #88)
> I'll land the patch later today.
Sorry, I haven't forgotten this. We typically land twice a day, but there was an irregularity yesterday and this morning someone "stole" my slot.

Comment 92

[Jorg K (GMT+1)]

https://hg.mozilla.org/comm-central/rev/819f6281ca932a8aa27efae50aa7546f692f5cf7

Comment 93

[Jorg K (GMT+1)]

Comment on attachment 8859644 [details] [diff] [review]
Fix the issue when auto forwarding (using filters) mails with attachements

Good candidate for uplift to fix ancient bug.

Comment 94

[Jorg K (GMT+1)]

Beta (TB 54):
https://hg.mozilla.org/releases/comm-beta/rev/213a3cb0acba2b2c1263da90a7ede8541faa6765

Comment 95

[Jorg K (GMT+1)]

TB 52 ESR:
https://hg.mozilla.org/releases/comm-esr52/rev/62cd4d71b086a8a2887dfe78859267c42aa9b6ff