531033 - Imap (GMail) INBOX: corrupted (mixed-up) messages due to synch. while reading (offlin-use=on)

Status: RESOLVED FIXED

(MailNews Core :: Database, defect)

Description

[Thomas Schyra]

User-Agent:       Opera/9.80 (Windows NT 6.1; U; de) Presto/2.2.15 Version/10.10
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.4pre) Gecko/20090915 Lightning/1.0pre Thunderbird/3.0b4

Messages are displayed in a crossover way: part of one message is mixed up with text from another if you read some while TB is still downloading/ synchronizing the Inbox with your IMAP-account (GMAIL in this case).
The Format is broken too: Even if html, the mix is displayed as text-only, all headers within are embedded.
Serversides the mails are still ok and can be downloaded to other clients without any problem.

Reproducible: Couldn't Reproduce

Steps to Reproduce:
1. Sychronize Inbox (few messages)
2. Read one of the new messages
3. Toggle the new messages (while synch is still going on!)

Actual Results:  
You have to have the right timing somehow, it's hard to reproduce but happens frequently. If it happens you see the behaviour filed in 'Details'

Expected Results:  
Show mails correctly, one by another.

This nug apperas on two different Clients (PC & Laptop), both Win7 x86_64

Comment 1

[Ludovic Hirlimann [:Usul]]

can you rebuild the index for your inboxes ?

Comment 2

[Mark Banner (:standard8)]

I'd really suggest you'd upgrade to TB 3 Release Candidate - there's been various issues in beta 4 with gmail.

http://www.mozillamessaging.com/en-US/about/press/archive/2009-11-24-01

Comment 3

[Thomas Schyra]

Oh, thx, dadn't get the Info that RC out now. I'll try that first. Don't know how to rebuild the Index?!

Comment 4

[Ludovic Hirlimann [:Usul]]

Right click on the folder , select properties. click rebuild index.

Comment 5

[Thomas Schyra]

Ok, I'll try that the next time this error occures. At least I moved the corrupted messages to another folder where they showed up the right way.

Comment 6

[Thomas Schyra]

Well. it happened again. now with Thunderbird RC1.

By the way: Index rebuild works fine and solves the issue.

Comment 7

[Thomas Schyra]

Ok guys, today I received a message with some files attached to it. the message was displayed properly - without any attachments. those where shown after rebuild only.
Another Message showed up with all header information inside and the plain text twice: on top and at the end. Something goes terribly wrong here...

Comment 8

[Matt Wilks]

I've had this same thing happen a couple of times now with the release version of 3.0.  It seems to happen after compacting a folder, but as Thomas said it is hard to reproduce.  The trick of rebuilding the index seems to have solved the problem, but I would definitely confirm that there is a bug here.

Comment 9

[Jonathan Protzenko [:protz]]

I'm setting this as new since
i) I've been experiencing this issue as well (and my bug report got DUP'd to another one that finally was closed because of incomplete information (see 553547))
ii) Some people seem to be experiencing it as well (see http://getsatisfaction.com/mozilla_messaging/topics/mail_arrives_mutilated?topic_tools=open)

Comment 12

[Wayne Mery (:wsmwk)]

(via Bug 543399 comment 8) if this is the same issue as Bug 543399, then a workaround is to set disk cache size to zero.

Comment 13

[Wayne Mery (:wsmwk)]

This is effectively/has the appearance of dataloass. protz, thanks for the gfsn link

Comment 14

[WADA:World Anti-bad-Duping Agency]

As seen in bug 565852, click of other mail while a mail is being downloaded causes interruption of the downloading followed by logout/re-login at a cached connection. It happens even after fix of bug 565852 if user clicked other mail while a mail is being downloaded, because there is no way to request of abort of fetch command to server. 
Bug 572974 is for a kind of downloaded mail data corruption which could be observed during test for bug 565852. Bug 572974 is a result of intentional test for problem re-creation such as Bug 570914 which occurred in real environment, and same phenomenon as Bug 570914 could be re-produced in Bug 572974.

Bug 501851 is another downloaded mail data corruption which can be reproduced easily/consistently.

Bug 587528 is report of different corruption pattern from above bugs and this bug, which can't be reproduced.

In any case, cause of garbled display of mail is mismatch of "offset data and mail length data in .msf" and "mail data held in offline-store file", or "wrong offset data and wrong mail length data in .msf". "Wrong mail status(downloaded normally or not) in .msf" is also relevant.

To analyze this kind of problem, at least next data is required.
(1) Shown mail data. Mail data saved as .eml.
(2) Back up of xxx.msf, and backup of xxx(offline-store file).
(3) Position of mail data of (1) in file of xxx(offline-store file).
I saw some reports for garbled display, "merged mail data which starts from wrong data", but I couldn't see data for diagnosis even data like (1) in such reports.

If you will meet problem again, keep data for diagnosis before recovery by rebuild-index/repair-folder, please.

Comment 16

[David :Bienvenu]

Attachment: don't use the disk cache if we're going to store the message offline - checked in.

I suspect this will help, at least with the case where it's the disk cache that's corrupted (as opposed to the offline store), which seems to be roughly half of the cases. We shouldn't be using the disk cache if we're going to store the message offline anyway.

Comment 17

[WADA:World Anti-bad-Duping Agency]

I read comment #0 and comment #7 by bug opener again.
  comment #0 : sounds same as Bug 587528 (simplest text/html mail)
  comment #7 : sounds same as Bug 604620 (simplest text/html mail)

Thomas Schyra(bug opener), is your comment #0/comment #7 same as Bug 587528/Bug 604620?

Comment 18

[WADA:World Anti-bad-Duping Agency]

Correction. Bug 604620 is example of simplest text/plain mail case.

Comment 19

[David :Bienvenu]

Comment on attachment 486703 [details] [diff] [review]
don't use the disk cache if we're going to store the message offline - checked in.

I don't know of a way to write a unit test for this - I need to run the message fetch with a real docshell, because a js docshell can't implement LoadUri, because it's not scriptable, and creating a real docshell in xpcshell tests tends to result in a docshell that prevent content from loading.

Comment 20

[David :Bienvenu]

I've got a few more improvements in the works for this. One is to verify that the cache entry corresponds to the imap message size, and if not, throw away the cache entry. This could help with some incorrect messages in the disk cache, though I may need to tweak the fix to allow for a little bit of wiggle room because I think in some cases we may add a trailing CRLF to messages.

I've also reproduced a case where if we abort a message download right near the end, with the line count tolerance to deal with different line endings, we'll think we have the complete message in the offline store but we don't. I know from the log that AbortMessage is getting called, but we're still setting the message as available in the offline store. I need to figure that out. It may be related to the fact that sometimes we get double copies of messages in the offline store.

Comment 21

[Mark Banner (:standard8)]

Comment on attachment 486703 [details] [diff] [review]
don't use the disk cache if we're going to store the message offline - checked in.

I believe I've tested this and made sure that entries no longer get written to the cache. It would seem strange to do that if we're storing offline as well.

Comment 22

[David :Bienvenu]

Comment on attachment 486703 [details] [diff] [review]
don't use the disk cache if we're going to store the message offline - checked in.

patch to not use disk cache landed on trunk - http://hg.mozilla.org/comm-central/rev/ee4eb19171c4

Comment 23

[David :Bienvenu]

I figured out why some imap message sizes weren't matching the cache entry size - one of my imap servers doesn't get the message size quite right, but we already have code to deal with that because Exchange does that as well; I just need to hook that up to the code that compares the message size vs. the cache entry size.

Comment 24

[David :Bienvenu]

Attachment: don't use disk cache entries that are the wrong size

This makes us throw away disk (and mem) cache entries that don't match the message size. This means we'll fall back to fetching the message from the imap server. I added some code to handle imap servers that don't return accurate rfc822.sizes. With the other patch in this bug not to use the disk cache if we're storing a message for offline use, this will only affect people who have turned off offline storage.

imap servers that don't always correctly return CRLF terminated lines will be affected by this patch because we correct the lines, which throws off the size a bit. We could adjust for this, but if this fixes the disk cache corruptions we're seeing, then I wouldn't block on that (though gmail is the main offender, though it's only some messages that have the issue - in particular, I think it's messages that come from gmail webmail!). Oh, and I need to make sure linux line endings don't throw this off, which I'll do in a minute.

Unit tests are still a no go for this w/o mozmill imap fake server support because we need a real docshell to invoke the imap code that stores things in the disk cache. I might bang my head on that particular wall a bit more if I'm feeling masochistic.

Finally, I've requested a 3.1.7pre try server build with this patch that I'm going to link to in the GS topic.

Comment 25

[David :Bienvenu]

grumble, mac/linux line endings definitely throw off the count by the number of lines in the message. So I need to figure out an adjustment for that.

Comment 26

[David :Bienvenu]

Attachment: fix mac/linux line ending issue

this fixes the mac issue - I test this code by turning off offline storage for a folder, repairing the folder to delete the offline store, and then setting a breakpoint on the line below that sets shouldUseCacheEntry to false

            if (messageSize != entrySize)
              shouldUseCacheEntry = PR_FALSE;

then, read a message, go to a different message, then back to the original message and make sure we don't hit the breakpoint. In general, I've been running with that breakpoint just to make sure I don't hit it.

Comment 27

[David :Bienvenu]

Attachment: verify the start of the cache entry as well as the length

This patch builds on the top of the previous cache length patch, and writes the first 50 bytes of the message as meta data to the cache entry, and then verifies that when we read back from the cache entry that the first 50 bytes match the meta data. I'm generating an other try server build for the GS folks to try with this patch. The original cache length patch apparently didn't help.

Comment 28

[David :Bienvenu]

I've finally seen disk cache corruption first-hand though I can't tell what caused the corruption. The disk cache entry length did match the message length, so I can see why that patch didn't help. But, the fact that the length matches really indicates to me that the cache itself is corrupted, because I can't imagine that we're writing exactly the write number of bytes, but the wrong data, with partial contents of an other message. Unfortunately, I wasn't running with the patched version so I can't tell for sure if the meta data fix would have helped.

One user sent me his corrupted offline store (separate issue) and I'm pretty sure that it was corrupted because an offline store compact happened while we were storing messages in the offline store. There are a couple code paths that don't grab the folder semaphore before writing to the offline store, and I believe that's the problem there.

Comment 29

[David :Bienvenu]

er, writing the right number of bytes.

Comment 30

[David :Bienvenu]

Attachment: fix interaction of offline download and compact of offline store

this makes all aspects of writing to the offline store respect the folder semaphore. The main case we're trying to fix is compact of the offline store causing corruption when messages are added to the offline store while the compaction is going on. Compact was already grabbing the semaphore, as was autosync, but storing a message in the offline store because the user clicked on it was not requiring or grabbing the semaphore, nor was copying the offline store copies of messages when messages are copied to other folders. This patch also adds a little batching to offline move/deletes, and does a bit of a sanity check on the size of messages in the offline store.

I haven't quite figured out a scenario where this lack of semaphore protection during compaction actually causes a problem, but I'll think about it.

Comment 31

[David :Bienvenu]

Attachment: oops, forgot the folder compactor part of the fix.

Comment 32

[Ludovic Hirlimann [:Usul]]

(In reply to comment #31)
> Created attachment 492016 [details] [diff] [review]
> oops, forgot the folder compactor part of the fix.

David is this a wip ?

Comment 33

[David :Bienvenu]

(In reply to comment #32)
> David is this a wip ?

see comment #30 - I haven't been able to recreate the problem, or come up with a scenario where corruption would occur without or with the patch, so I don't know if it's a wip or not. I haven't forgot to ask for reviews, if that's what you're asking.

Comment 34

[David :Bienvenu]

Some additional details about the one corrupted offline store I've been able to analyze:

The folder was very recently compacted. The messages in compacted offline stores start with "From "CRLF and have no x-mozilla-status headers; messages that have been added after the last compaction will have "From <date>"CRLF and x-mozilla-status headers (this isn't intentional, but it's turning out to be helpful in diagnosing the corruption).

There were a couple "From <date>"CRLF and x-mozilla-status lines interspersed in the existing offline messages, at the very end of the offline store, which tends to support the theory that offline download was happening during compaction, though it's not conclusive.  Because compaction copies all the offline messages to a temp file, and then copies the temp file back over the original file, it's not as if you get two urls writing to the same file. But compaction does force the db closed, which probably invalidates the offline header the imap folder uses when writing to the offline store. That may cause some problem, though I haven't convinced myself of that yet.

Comment 35

[David :Bienvenu]

Attachment: verify offline store msg starts with a header

This adds a check that an offline message really starts with a header line (i.e., we find ':' before CR or LF), which will detect common kinds of corruption.

I'll try to come up with unit tests for some aspects of this patch, but it's a bit difficult given the async nature of the offline store compaction code...

Comment 36

[David :Bienvenu]

Attachment: better verification of the cache entry

This adds a check that the cache entry starts with a message header, which will catch common corruptions.

Comment 37

[David :Bienvenu]

Attachment: fix comment

this should detect most cases of disk cache corruption. If we detect a corrupt disk cache entry, we just fall back to the offline store, or the imap server, so the risk is fairly slight.

Comment 38

[David :Bienvenu]

Attachment: cache and offline store verification patch - checked in.

This is a minimal 3.1 patch that just makes sure cache entries and offline store messages start with an actual header. 

Note that this patch exposed a real train wreck in the 3.1 version of MsgFindCharInSet, which was causing a lot of test failures, when running with the others parts of this patch.

I'll request a try-server version of this patch, and request feedback from GS, though feedback from U.S. users might be delayed by the holidays.

Comment 39

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #38)
> cache and offline store verification patch

Can you add Activity Manager message or Error Console log when detects error?

Comment 40

[David :Bienvenu]

(In reply to comment #39)
> Can you add Activity Manager message or Error Console log when detects error?

I don't think it's appropriate for the Activity Manager. I'm not sure about the error console. The problem with adding errors that are handled completely without the user knowing to the error console is that the user gets confused and thinks that the error is a symptom of whatever problem they are seeing. I'd be more inclined to add the info to the imap protocol log.

Comment 41

[David :Bienvenu]

feedback from one of my GS users is extremely encouraging - clicking on a message that he thought was corrupted resulted in a second or two delay while we fetched the message from the server, and the message displayed correctly, both in the try-server build, and then in 3.1.6 (because this patch makes us throw away corrupted messages, and recache them correctly.

Comment 42

[Mark Banner (:standard8)]

Comment on attachment 492804 [details] [diff] [review]
cache and offline store verification patch - checked in.

>diff --git a/mailnews/base/util/nsMsgUtils.cpp b/mailnews/base/util/nsMsgUtils.cpp

> MsgFindCharInSet(const nsCString &aString,
>                  const char* aChars, PRUint32 aOffset)
> {
>-#ifdef MOZILLA_INTENAL_API
>-  return FindCharInSet(aChars, aOffset)
>+#ifdef MOZILLA_INTERNAL_API
>+  return aString.FindCharInSet(aChars, aOffset);
> #else
>   PRInt32 len = strlen(aChars);
>   PRInt32 index = -1;
>-  for (int i = aOffset; i < len; i++) {
>-    index = aString.FindChar(aChars[i]);
>+  for (int i = 0; i < len; i++) {
>+    index = aString.FindChar(aChars[i], aOffset);

I'm not sure if we use the nsString version anywhere, but could you fix that as well please?

I'm going to give this patch a bit of a run in a few mins but otherwise it looks ok.

For 3.1 should we be taking this patch and the one to not use the disk cache if we're using the offline store (attachment 486703 [details] [diff] [review])? I think I'd be happy taking that one as well.

Comment 43

[Mark Banner (:standard8)]

Comment on attachment 492804 [details] [diff] [review]
cache and offline store verification patch - checked in.

Ok, I've played around with this for a bit and it seems to do what is intended.

Comment 44

[David :Bienvenu]

fixed for 3.1.7 (nsString version was used, so I left it in).

changeset:   5851:7c041f93df7d
tag:         tip
user:        David Bienvenu <[e-mail]>
date:        Thu Nov 25 14:18:23 2010 -0800
summary:     fix for bug 531033, corruption of imap offline store and disk cache
 (or, recovery from, rather) r/a=standard

Comment 45

[David :Bienvenu]

Attachment: 3.1 version of patch not to use the disk cache when offline storage requested - checked in

Comment 46

[Mark Banner (:standard8)]

Comment on attachment 493317 [details] [diff] [review]
3.1 version of patch not to use the disk cache when offline storage requested - checked in

Checked in: http://hg.mozilla.org/releases/comm-1.9.2/rev/bf2a9c8424b6

Comment 47

[David :Bienvenu]

Attachment: offline store semaphore protection with unit test

this makes several operations that write to the offline store make sure the offline store isn't locked (downloading a message to offline store, copying message body when doing an offline copy) and adds a unit test that verifies we don't get a message with an offline body when doing those operations while compaction of offline store is going on (which locks the offline store).

Comment 48

[David :Bienvenu]

I've landed the cache sanity checking patch on the trunk http://hg.mozilla.org/comm-central/rev/f0c903a93fc6 just to make life on the trunk better...

Comment 49

[Mark Banner (:standard8)]

Comment on attachment 493458 [details] [diff] [review]
offline store semaphore protection with unit test

Here's a couple of comments from a look at this patch:

Full review board: http://reviews.visophyte.org/r/493458/

on file: mailnews/imap/src/nsImapMailFolder.cpp line 1359
>   return WeAreOffline() ? NS_OK : Expunge(this, aMsgWindow);

Is this going to leave us in a state where we think we are expunging but we're
not, if we try and compact whilst offline?

(the line above has m_expunging = PR_TRUE).


on file: mailnews/imap/src/nsImapMailFolder.cpp line 4468
>     ReleaseSemaphore(static_cast<nsIMsgImapMailFolder*>(this));

Shouldn't this be nsIMsgFolder, as per the change in AcquireSemaphone in the
line above?

Comment 50

[Mark Banner (:standard8)]

I'm having trouble getting the attachments applying so that I can test them.

I think attachment 492755 [details] [diff] [review] is a later iteration of attachment 490299 [details] [diff] [review], though neither of them applies.

Attachment 493458 [details] [diff] applies, but doesn't compile, I guess it may need the first ones?

Comment 51

[Mark Banner (:standard8)]

Comment on attachment 490299 [details] [diff] [review]
fix mac/linux line ending issue

Cancelling reviews whilst the bitrot issues are resolved.

Comment 52

[David :Bienvenu]

yeah, sorry, I landed some separately reviewed patches so that life on the trunk could improve while the review was going on. I'll attach a de-bittrotted patch ASAP in the hopes that this can land before the holidays.

Comment 53

[David :Bienvenu]

Attachment: de-bitrotted patch, fix with unit test

Comment 54

[David :Bienvenu]

Attachment: fix merge issue w/ last patch, semaphore fix with unit test

Comment 55

[David :Bienvenu]

Attachment: remove batching perf changes

Comment 56

[David :Bienvenu]

Attachment: disable autosync while test is running

we need to disable autosync while running the test because if it kicks in, downloadAllForOffline fails, appropriately but unexpectedly for the purposes of the test.

Comment 57

[David :Bienvenu]

Attachment: oops, update wrong patch in mq

Comment 58

[Mark Banner (:standard8)]

Comment on attachment 499312 [details] [diff] [review]
oops, update wrong patch in mq

Ok, that looks fine now. Thanks.

Comment 59

[David :Bienvenu]

fix checked in - there may be other causes of offline store corruption (or disk cache corruptions), but we should cover them in new bugs, with reproducible steps...

Comment 60

[Mark Banner (:standard8)]

I've just had to disable the test on windows due to permanent orange there:

http://hg.mozilla.org/comm-central/rev/14586887520b

TEST-UNEXPECTED-FAIL | e:\buildbot\comm-central-trunk-win32-opt-unittest-xpcshell\build\xpcshell\tests\mailnews\imap\test\unit\test_offlineStoreLocking.js | test failed (with xpcshell return code: 0), see following log:
>>>>>>>
TEST-INFO | (xpcshell/head.js) | test 1 pending
Directory request for: SysD that we (mailDirService.js) are not handling, leaving it to another handler.
Directory request for: MailD that we (mailDirService.js) are not handling, leaving it to another handler.
Directory request for: MFCaF that we (mailDirService.js) are not handling, leaving it to another handler.
Directory request for: DefRt that we (mailDirService.js) are not handling, leaving it to another handler.
Directory request for: IMapMD that we (mailDirService.js) are not handling, leaving it to another handler.
TEST-INFO | (xpcshell/head.js) | test 2 pending
Doing test 1
Downloading for offline use
TEST-INFO | (xpcshell/head.js) | test 2 finished
TEST-INFO | (xpcshell/head.js) | running event loop
AUTH PLAIN line -AHVzZXIAcGFzc3dvcmQ=-
authorize-id: --, username: -user-, password: -password-
TEST-PASS | e:/buildbot/comm-central-trunk-win32-opt-unittest-xpcshell/build/xpcshell/tests/mailnews/imap/test/unit/test_offlineStoreLocking.js | [null : 360] 0 == 0
Doing test 2
TEST-PASS | e:/buildbot/comm-central-trunk-win32-opt-unittest-xpcshell/build/xpcshell/tests/mailnews/imap/test/unit/test_offlineStoreLocking.js | [null : 345] 0 == 0
Doing test 3
Directory request for: cachePDir that we (mailDirService.js) are not handling, leaving it to another handler.
Directory request for: cachePDir that we (mailDirService.js) are not handling, leaving it to another handler.
TEST-PASS | e:/buildbot/comm-central-trunk-win32-opt-unittest-xpcshell/build/xpcshell/tests/mailnews/imap/test/unit/test_offlineStoreLocking.js | [null : 360] 0 == 0
TEST-UNEXPECTED-FAIL | e:/buildbot/comm-central-trunk-win32-opt-unittest-xpcshell/build/xpcshell/tests/mailnews/imap/test/unit/test_offlineStoreLocking.js | 128 == false - See following stack:
JS frame :: e:\buildbot\comm-central-trunk-win32-opt-unittest-xpcshell\build\xpcshell\head.js :: do_throw :: line 439
JS frame :: e:\buildbot\comm-central-trunk-win32-opt-unittest-xpcshell\build\xpcshell\head.js :: do_check_eq :: line 491
JS frame :: e:\buildbot\comm-central-trunk-win32-opt-unittest-xpcshell\build\xpcshell\head.js :: do_check_false :: line 510
JS frame :: e:/buildbot/comm-central-trunk-win32-opt-unittest-xpcshell/build/xpcshell/tests/mailnews/imap/test/unit/test_offlineStoreLocking.js :: <TOP_LEVEL> :: line 105
TEST-INFO | (xpcshell/head.js) | exiting test
before 1376256, after 475136, break 00000000
Doing test 4
<<<<<<<

Comment 61

[Wayne Mery (:wsmwk)]

http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_mixes_headers_from_one_email_with_content_from_another has 57 interested parties. whew!

Comment 62

[mikedoo3]

(In reply to comment #61)
> http://getsatisfaction.com/mozilla_messaging/topics/
> thunderbird_3_mixes_headers_from_one_email_with_content_from_another has 57
> interested parties. whew!

I am still having this issue.  Was it supposedly fixed?  I am having the problem in Thunderbird 3.1.10

Comment 63

[David :Bienvenu]

We fixed many instances of this problem but if the problem is corruption of the disk cache, there are some corruptions that are difficult to detect and fix.  Since disk cache code is core code, there may be fixes in the upcoming releases that I don't know about.

Comment 64

[mikedoo3]

(In reply to comment #63)
> We fixed many instances of this problem but if the problem is corruption of
> the disk cache, there are some corruptions that are difficult to detect and
> fix.  Since disk cache code is core code, there may be fixes in the upcoming
> releases that I don't know about.

Is there a bug filed for the issue at 
http://getsatisfaction.com/mozilla_messaging/topics/thunderbird_3_mixes_headers_from_one_email_with_content_from_another
I can't find it.  I can pretty much duplicate this at will.  The headers and message content get messed up when copying/moving emails from INBOX to Local folders or IMAP folders (though I have only been able to duplicate it with Local folders).  I support a corporate install of Thunderbird, I have a whole bunch of people reporting this to me.  It can't be that rare.

Comment 65

[Jonathan Protzenko [:protz]]

I to me, I used to experience this issue, and I haven't seen it pop up in a while (I'm running Miramar nightlies), so you might want to try out the latest preview version of Thunderbird to see if it helps. http://www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/

Comment 66

[Thomas Schyra]

(In reply to comment #65)
Me neither, haven't seen this bug for a long time. I have to admit i switched my OS from MS to Ubuntu. But few days ago i got a mail which had a corrupted Subject. I considered it to be spam so i did not pay much attention to it. I even don't know if this issue is related because the rest of this mail looked proper.

My TB-Version: 3.1.10

Comment 67

[David :Bienvenu]

I don't know that anyone has reported an issue with copying imap messages to a local folder. Are you copying multiple messages at a time? The interesting detail there is that operation doesn't use the disk or memory or offline cache, so it fetches directly from the server and writes to the local folder.

Comment 68

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to comment #64)
I am using Mac Miramar nightly 3.3a May 19, 2011 and gmail and copied 9 messages from gmail IMAP to a local folder on my Mac and it worked fine. I'll try on Windows 7 and see if I can get it to fail.

> (In reply to comment #63)
> > We fixed many instances of this problem but if the problem is corruption of
> > the disk cache, there are some corruptions that are difficult to detect and
> > fix.  Since disk cache code is core code, there may be fixes in the upcoming
> > releases that I don't know about.
> 
> Is there a bug filed for the issue at 
> http://getsatisfaction.com/mozilla_messaging/topics/
> thunderbird_3_mixes_headers_from_one_email_with_content_from_another
> I can't find it.  I can pretty much duplicate this at will.  The headers and
> message content get messed up when copying/moving emails from INBOX to Local
> folders or IMAP folders (though I have only been able to duplicate it with
> Local folders).  I support a corporate install of Thunderbird, I have a
> whole bunch of people reporting this to me.  It can't be that rare.

Comment 69

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

(In reply to comment #68)
Just copied 2573 messages from gmail IMAP to a local folder on my Windows 7 Machine and it worked fine. Using Windows 7 Miramar nightly 3.3a May 19, 2011

> (In reply to comment #64)
> I am using Mac Miramar nightly 3.3a May 19, 2011 and gmail and copied 9
> messages from gmail IMAP to a local folder on my Mac and it worked fine.
> I'll try on Windows 7 and see if I can get it to fail.
> 
> > (In reply to comment #63)
> > > We fixed many instances of this problem but if the problem is corruption of
> > > the disk cache, there are some corruptions that are difficult to detect and
> > > fix.  Since disk cache code is core code, there may be fixes in the upcoming
> > > releases that I don't know about.
> > 
> > Is there a bug filed for the issue at 
> > http://getsatisfaction.com/mozilla_messaging/topics/
> > thunderbird_3_mixes_headers_from_one_email_with_content_from_another
> > I can't find it.  I can pretty much duplicate this at will.  The headers and
> > message content get messed up when copying/moving emails from INBOX to Local
> > folders or IMAP folders (though I have only been able to duplicate it with
> > Local folders).  I support a corporate install of Thunderbird, I have a
> > whole bunch of people reporting this to me.  It can't be that rare.

Comment 70

[David :Bienvenu]

yeah, copying from imap to local works in general; Perhaps there's something about certain environments (e.g., a virus checker) that interferes with our ability to write out the local messages.

Comment 71

[mikedoo3]

I can absolutely confirm a local folder corruption problem in TB 3.1.10.  I have not been able to duplicate it in the latest Miramar build though.  Basically I drag and drop individual emails to a local folder as fast as I can and eventually some corruption happens.  One of the emails in the local folder will have subject from one email but body of another email.  Repairing the folder fixes it but data is lost.  

So I guess the question is, do you care about this if I can't duplicate it in the latest Miramar?  I don't want to spend any more time trying to get more details if we don't care.  As I said, I pushed out thunderbird 3.1 to a corporate environment (upgrade from thunderbird 2), and I have users complaining of serious issues with corruption in local folders, doubling of messages in imap folders, all sorts of bad things that will be very difficult to debug.  We have some users on Exchange IMAP servers, some on Unix IMAP servers, not all reports have been users on one type of server or one particular server.

Comment 72

[Wayne Mery (:wsmwk)]

(In reply to comment #71)
> I can absolutely confirm a local folder corruption problem in TB 3.1.10. 

you should file a new bug report.