815012 - Mail messages corrupt after running filters

Status: RESOLVED FIXED

(MailNews Core :: Filters, defect)

Description

[Brian Vanderburg]

User Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/17.0 Firefox/17.0
Build ID: 20121119183901

Steps to reproduce:

System details:

Debian Linux 6.0 x32
Thunderbird 17
Enigmail 1.4.6
ImportExportTools 2.7.2.2
Lightning 1.9
Mail Merge 3.4.1
Mail Summaries 2.1.1
Restartless Restart 8
Thunderbird Conversations 2.5.2

Account layout:

aim.com
-->Inbox
-->(etc)
Local Folders
-->Inbox
-->Storage
-->-->Software
-->-->(etc)

I have two filters set up.  My first filter runs on my email accounts and moves email messages to my Local Folders inbox.  My second runs on Local Folders and moves email to sorted folders based on criteria such as subject or from.  (I use two filters since the filters only applies to one selected account and not to all accounts, so I can move from multiple accounts to the main account and then have them all filtered, instead of duplicating the sorting filters for multiple accounts)

On connected and after the password prompt, messages are automatically moved to my Local Folders inbox.  After that, I do a Run Filters on Folder on the Local Folders inbox, which sorts messages.


Actual results:

The messages are moved to the sorted folders.  However, certain messages from the mailing lists (Gimp, Inkscape), seem corrupt.  Viewing these messages after being moved to the Software folder shows sometimes no body or parts of the HTML.  Sometimes it even seems to show stuff from the wrong emails such as some an Amazon email confirming an app download instead of the mailing list email.

The following seems to fix it:

One way:  Repair folder on the Software and Inbox for Local Folders.  Then run the filters on the Inbox again.  Messages are then moved correctly.

Another way:  Before running the filters on the Inbox for Local Folders (after they have been automatically moved), do a repair on the folder.  Messages are then moved correctly when running filters.

(also, manually moving the messages doesn't seem to cause the issue, only running the filter on Local Folders Inbox to sort the messages)


Expected results:

The messages should have moved properly.

Comment 1

[:aceman]

Do you have any folder bigger than 2GB?
Can you attach such a corrupted message (export as eml)?

Comment 2

[Brian Vanderburg]

None of my folders are over 2GB.  They sometimes add up to several hundred messages though.  I'll post an exported message shortly.

Comment 3

[Brian Vanderburg]

Attachment: Corrupt Message 1

Comment 4

[Brian Vanderburg]

Attachment: Corrupt Message 1 fixed

Comment 5

[Brian Vanderburg]

Attachment: Corrupt Message 2

Comment 6

[Brian Vanderburg]

Attachment: Corrupt Message 3

Comment 7

[Brian Vanderburg]

I have attached several corrupt messages as well as the fixed form of one message after doing repair on both local folders and rerunning the filters.  I am also creating a new profile to see if the same happens there.

Comment 8

[Brian Vanderburg]

This problem does not seem to exist in the new profile.  I've installed all the same extensions, exported my email messages and imported them into the new profile, etc.  Could it have been some messed up files in my original profile directory?  Marking as resolved, invalid for now.

Comment 9

[Brian Vanderburg]

The first couple times I checked the mail in the new profile, everything was fine.  After that, the second run of the filters (the first moves from accounts to local folder on receiving them, the second from there to sub folders) still is causing problems after all.

Comment 10

[Brian Vanderburg]

Since I am still having this issue, even after disabling all extensions, creating a new profile, etc, I'll switch back to 16.0.2 as this problem wasn't happening until after I installed 17.

Comment 11

[Kent James (:rkent)]

I was able to duplicate this using a TB 18 debug build. When I run manual move filters on the Local Folders inbox, I get this for each message:

###!!! ASSERTION: whoops, something wrong with previous copy: 'mCopyState->m_leftOver == 0', file c:/tb/aurora/src/mailn
ews/local/src/nsLocalMailFolder.cpp, line 2231
###!!! ASSERTION: Fatal ... bad message format
: 'strncmp(start, "From ", 5) == 0', file c:/tb/aurora/src/mailnews/local/src/nsLocalMailFolder.cpp, line 2106

and the messages do not read correctly when opened in the message pane. The underlying mbox file looks OK, so it is the message offsets in the database that are getting messed up.

The initial moves of the messages from an IMAP folder to the Local Folders inbox also showed these messages:

###!!! ASSERTION: didn't finish prev msg: 'm_streamOutstandingFolder != aFolder', file c:/tb/aurora/src/mailnews/local/s
rc/nsMsgBrkMBoxStore.cpp, line 612

Repair of the target Local Folder where the messages were moved to resulted in all of the moved messages disappearing, so I guess there was something wrong with the target mbox file after all.

I'll need to move all of this to my comm-central development profile, and test on some different profiles to make sure that I can reproduce it reliably. Bit if this is real, it is a serious issue.

Comment 12

[:aceman]

Have you narrowed this down to any account types? Does it affect all accounts or "only" those storing messages in local files (POP, RSS, Local folders) ?

This seems to crop up in more reports.

Comment 13

[Kent James (:rkent)]

I've got some sort of problems that are starting to look reproducible now. I think these issues are this bug, but we'll see.

I'm starting to focus on the initial copy of the IMAP messages to the Local Folder pseudo-inbox. If I select two messages to move, then move them to a Local Folder, I reliably get this assertion (now using a comm-central debug build):

###!!! ASSERTION: didn't finish prev msg: 'm_streamOutstandingFolder != aFolder', file c:/tb/1-central/src/mailnews/local/src/nsMsgBrkMBoxStore.cpp, line 612

This assertion is really a debug check put in to see if nsMsgBrkMBoxStore::FinishNewMessage ever gets called. Apparently it does not on the first message, so the start of the second message generates that assertion.

It gets called once eventually, from this stack:

 	xul.dll!nsMsgLocalMailFolder::EndCopy(bool aCopySucceeded=true)  Line 2198	C++
 	xul.dll!nsCopyMessageStreamListener::EndCopy(nsISupports * url=0x06dcc728, tag_nsresult aStatus=NS_OK)  Line 121 + 0x21 bytes	C++
>	xul.dll!nsCopyMessageStreamListener::OnStopRequest(nsIRequest * request=0x034f5350, nsISupports * ctxt=0x06dcc728, tag_nsresult aStatus=NS_OK)  Line 151	C++
 	xul.dll!nsStreamListenerTee::OnStopRequest(nsIRequest * request=0x034f5350, nsISupports * context=0x06dcc728, tag_nsresult status=NS_OK)  Line 49 + 0x28 bytes	C++
 	xul.dll!`anonymous namespace'::SyncRunnable3<nsIStreamListener,nsIRequest *,nsISupports *,enum tag_nsresult>::Run()  Line 177 + 0x2a bytes	C++

which is coming somewhere from the IMAP thread.

What is unclear is that sometimes this results in corrupted message stores (both IMAP offline store and the local mbox), but sometimes it does not. My current suspicion is that the failed filter occurs when the message store was already corrupted by this prior move, but I do not have a clear path to prove that yet.

Comment 14

[Kent James (:rkent)]

I'm not convinced that the issues in comment 13 are related to the problems noted in comment 0 (though they could cause serious issues for maildir, so I posted a possible fix on bug 791966). FinishNewMessage is essentially a noop for mbox.

So at the point I am having a hard time reproducing this again.

Comment 15

[thierry.mallard]

Using two separate profiles, Thunberbird 17 x86_64 (Ubuntu), got again some mail corruption when working on locally moved e-mails. On the console, the following messages appears (don't know if that's of any help) 



Entity: line 5: parser error : xmlParseCharRef: invalid xmlChar value 8
<rdf:Description rdf:about='156a8bc2-3d54-11e2-0000-7368bcda775&#8;`��&#10;2012-
                                                                   ^
Entity: line 5: parser error : Input is not proper UTF-8, indicate encoding !
Bytes: 0xC9 0xDE 0x26 0x23
<rdf:Description rdf:about='156a8bc2-3d54-11e2-0000-7368bcda775&#8;`��&#10;2012-
                                                                    ^
Entity: line 5: parser error : xmlParseCharRef: invalid xmlChar value 16
8bc2-3d54-11e2-0000-7368bcda775&#8;`��&#10;2012-11-30T15:15:26+01:00&apos; &#16;
                                                                               ^
Entity: line 5: parser error : xmlParseCharRef: invalid xmlChar value 2
2-0000-7368bcda775&#8;`��&#10;2012-11-30T15:15:26+01:00&apos; &#16;���&#10;,&#2;
                                                                               ^
Entity: line 5: parser error : xmlParseCharRef: invalid xmlChar value 16
-7368bcda775&#8;`��&#10;2012-11-30T15:15:26+01:00&apos; &#16;���&#10;,&#2;@&#16;
                                                                               ^
Entity: line 5: parser error : xmlParseCharRef: invalid xmlChar value 4
8bcda775&#8;`��&#10;2012-11-30T15:15:26+01:00&apos; &#16;���&#10;,&#2;@&#16;&#4;

Comment 16

[Jens Hatlak (:InvisibleSmiley)]

I've also experienced mail corruption with SM 2.15b1, maybe already with 2.14 betas (I'm on the beta channel, Win 7 x64), through automatic filters moving messages from my IMAP account to Local Folders. Sounds like this bug. Mail is not completely lost; instead message parts appear when other messages of the respective folder are selected. In these cases, the header pane shows often shows no subject or is missing other header information (unlike the thread pane selection).

Last time I checked, there was a message in one of the affected folders that was at the end of the mbox file on disk. It was cut off in the middle of a MIME attachment part so the message was corrupt. Its contents were appearing when another message was selected in the UI. I guess that supports the "it is the message offsets in the database that are getting messed up" statement from comment 11.

I'll confirm this one now since I'm sure we do have a serious problem here. The reporter sees it, I do, and so does Michael Ströder on the newsgroups (mozilla.support.seamonkey, "mbox handling messed up in SeaMonkey 2.14?").

I've disabled all filters for my IMAP account for now since obviously data loss is not an option. I'll add a warning to SeaMonkey's Release Notes in a minute (not to be tracked here).

In case it's important, I regularly run filters manually and compact all folders, too. I do NOT use Maildir locally.

Comment 17

[Frank Wein [:mcsmurf]]

So just to write this down: This probably regressed between SeaMonkey 2.13/Thunderbird 16 and SeaMonkey 2.14/Thunderbird 17? Can someone confirm/deny this?

Comment 18

[michael]

With 2.13.2 I did not have this problem. Since 2.14 I have this problem. 2.14.1 did not fix it.

BTW: This occasionally also happens when manually moving IMAP messages within an IMAP account. Most times it happens with complicated HTML mails. Have there been any significant changes to the way this are parsed and stored?

Comment 19

[Frank Wein [:mcsmurf]]

Attachment: Checkins between TB 16.0.2. and TB 17

Maybe this is of help for the developers if someone wants to look at the checkins between TB 16.0.2 and TB 17. This file was created with the command
hg log --rev 1f02ae595203:2591738ac868
on comm-release. 1f02ae595203 is the THUNDERBIRD_17_0_RELEASE tag, 2591738ac868 is the THUNDERBIRD_16_0_2_RELEASE tag.

Comment 20

[Frank Wein [:mcsmurf]]

Attachment: Changed files between TB 16.0.2 and TB 17

Maybe this is of help for the developers if someone wants to look at list of changed files between TB 16.0.2 and TB 17. This file was created with the command
hg status --rev 1f02ae595203:2591738ac868
on comm-release. 1f02ae595203 is the THUNDERBIRD_17_0_RELEASE tag, 2591738ac868 is the THUNDERBIRD_16_0_2_RELEASE tag.

Comment 21

[Frank Wein [:mcsmurf]]

And yes, a lot has been changed, though quite a few changes were only cleanup (at least it looks like that).

Comment 22

[Frank Wein [:mcsmurf]]

@Kent: Almost a complete shot into the dark, but maybe Bug 790912 might be related to this bug here. Can you comment/check if your patch might have to do with this bug here?

For less check-ins to look at one can also use the command
hg log --rev 1f02ae595203:2591738ac868 -I mailnews
so that it only lists changes which modified files which have "mailnews" in its file path.

Comment 23

[peterj]

This bug affects me since the 17 update.

Seeing reports on other forums, it seems to affect W7 64 bit users most, perhaps only.

It is when a file is moved by a filter from one folder to another.
It also occurs during a folder compact, but more rarely, in my experience.
The data in the file is the random data copied from the original folder instead of the correct message data. It appears that the move operation has a pointer which becomes corrupt and reads from the wrong starting point. Data from that random point to the next end of message is moved to the new folder.

Turning off filter moves and not compacting, I have not seen it happen since and it was happening almost every time mail was retrieved.

Comment 24

[Wayne Mery (:wsmwk)]

(In reply to Frank Wein [:mcsmurf] from comment #22)
> @Kent: Almost a complete shot into the dark, but maybe Bug 790912 might be
> related to this bug here. Can you comment/check if your patch might have to
> do with this bug here?
> 
> For less check-ins to look at one can also use the command
> hg log --rev 1f02ae595203:2591738ac868 -I mailnews
> so that it only lists changes which modified files which have "mailnews" in
> its file path.

and, any relationship to Jeff's bug 753624?

Comment 25

[Wayne Mery (:wsmwk)]

possible reports:
- https://getsatisfaction.com/mozilla_messaging/topics/17_0_1_email_corrupted_when_moving_from_local_folder_inbox_to_another_subfolder
- https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_17_corrupts_emails
- https://getsatisfaction.com/mozilla_messaging/topics/upgrade_to_thunderbird_17_message_filters_corrupt_the_contents_of_the_email
- https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_17_0_message_corruption_on_local_folders_upon_applying_filters
- in mozilla.support.thunderbird, subject " TB17 Indexing/Compacting Errors"
On 12/6/2012 5:09 PM, Wayne wrote:
> On 12/6/2012 11:55 AM, News wrote:
>> Seeing many messages scrambled, mis-indexed (header and message body
>> misaligned/wrong body with header) after auto- or manual compacting.
>>
>> Anyone else seeing this?
>
> News, are these in folders that are receiving *filtered* messages?
Exactly.

Comment 26

[Kent James (:rkent)]

The key thing that this bug needs is a reproducible test case. Without that, it is extremely difficult to make any progress. So for anyone seeing this, any progress you could make toward developing a reproducible case would be greatly appreciated.

I'm not really sure what more I can say for feedback other than that. I've been fighting my own serious issues the last few weeks, so I have not been following this as closely as I would like. But I will look at any error that I can reproduce, if someone can suggest how to do that.

Comment 27

[Wayne Mery (:wsmwk)]

(In reply to Kent James (:rkent) from comment #26)
> The key thing that this bug needs is a reproducible test case. Without that,
> it is extremely difficult to make any progress. So for anyone seeing this,
> any progress you could make toward developing a reproducible case would be
> greatly appreciated.

kent, two, possibly three are already here in the bug -  comment 18 (michael) and comment 24 (peterj), comment 16 (Jens)

Can any of you narrow the regression range PLEASE?  two within a one to two day range would be great.  (mcsmurf's list is just way too long.)

Comment 28

[Frank Wein [:mcsmurf]]

I suggested one possible bug, should maybe produce a try build without the offending patch.

Comment 29

[Wayne Mery (:wsmwk)]

To narrow the range of when this started happening requires testing a few early builds
- 17.0a2 https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/08/2012-08-28-04-20-08-comm-aurora/
If it happens in 17.0a2, the next step is to test some 17.0a1 dailies (comm-central), using binary search method, starting roughly from https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/07/2012-07-26-03-02-01-comm-central/ through https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/08/2012-08-27-03-05-49-comm-central/

If it doesn't happen 17.0a2 then try...
- 17.0b1 https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/17.0b1/
- 17.0b2 https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/17.0b2/
- 17.0b3 https://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/17.0b3/

Comment 30

[Wayne Mery (:wsmwk)]

(In reply to Frank Wein [:mcsmurf] from comment #28)
> I suggested one possible bug, should maybe produce a try build without the
> offending patch.

Good point. 

Bug 790912 checked in on 17.0 at 2012-10-05.  So this bug should appear in
 https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/2012-10-05-04-20-11-comm-aurora/

but not in https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/2012-10-06-04-20-02-comm-aurora/

Comment 31

[Wayne Mery (:wsmwk)]

(comment 30 revised - drugs again)

(In reply to Frank Wein [:mcsmurf] from comment #28)
> I suggested one possible bug, should maybe produce a try build without the
> offending patch.

Good point. 

Bug 790912 checked in on 17.0 at 2012-10-05.  So this bug should appear in
https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/2012-10-06-04-20-02-comm-aurora/

but not in https://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2012/10/2012-10-05-04-20-11-comm-aurora/

Comment 32

[AirBoss]

Confirmed behavior on manual copies since cutover to TB17.0:

IMAP Message filtered to Local INBOX - Message OK

Local INBOX Message manually copied to Local Destination folder - Message corrupted

alt-e-u (undo) recover Message from Local Destination folder to Local INBOX - recovered Message remains corrupted

delete Local INBOX .msf and OK Message is recovered

Comment 33

[AirBoss]

As I'm operating in a production environment, without access to a sandbox, I am unable to test various distros of TB17.

Comment 34

[peterj]

(In reply to AirBoss from comment #32)
> Confirmed behavior on manual copies since cutover to TB17.0:
> 
> IMAP Message filtered to Local INBOX - Message OK
> 
> Local INBOX Message manually copied to Local Destination folder - Message
> corrupted
> 
> alt-e-u (undo) recover Message from Local Destination folder to Local INBOX
> - recovered Message remains corrupted
> 
> delete Local INBOX .msf and OK Message is recovered

Just a data point. In my case, it began immediately after the 17 update.

Manually MOVING from INBOX to other folders (dragging and dropping) has never resulted in a corruption.

Filtered move from INBOX to another folder has a high probability of corruption.

Comment 35

[Tony Mechelynck [:tonymec]]

Related with bug 776983?

Comment 36

[Kent James (:rkent)]

(In reply to Wayne Mery (:wsmwk) from comment #27)
> (In reply to Kent James (:rkent) from comment #26)
> > The key thing that this bug needs is a reproducible test case...
> 
> kent, two, possibly three are already here in the bug -  comment 18
> (michael) and comment 24 (peterj), comment 16 (Jens)

These cases are confirmations, but not reproducible. Or at least if I setup the basic situation they are describing (automatic move from IMAP to a local folder, followed by a manual filter move to another folder) I cannot reproduce this.

Right now, the only hook that I have to work on this is my own comment 11. At the moment, I can reproduce the "ASSERTION: didn't finish prev msg" message from the IMAP to Local move, but it does not seem to cause any ill effects. I can no longer reproduce the "whoops, something wrong with previous copy" which seems to indicate that something bad is about to happen which is probably this bug.

So from my perspective, I do not have a reproducible case that I can use to diagnose this. Any help in getting that would be appreciated.

Comment 37

[Kent James (:rkent)]

I seemed to now be reliably reproducing again the errors I posted in Comment 11, so I'll take a look at this now. There seems to be some unknown characteristic of one of the folders that is needed to keep seeing this, so hopefully I won't inadvertently cause it to go away. I've copied the test profile that I use to demo this, so hopefully that will allow be to make is happen again if it goes away again.

Comment 38

[Kent James (:rkent)]

I tried a test build with the patch from Bug 790912 removed and I could still reproduce the behavior of comment 11, so I don't think that bug is the regression.

I don't think that I need a specific regression to make progress on this at the moment.

Comment 39

[Jens Hatlak (:InvisibleSmiley)]

Addressing the needinfo request:

Since I had this problem with my regular account, I'm currently avoiding any steps (like going back to older test builds) that could further corrupt my data. However, like Peter I can confirm that this is definitely a problem that only occurs with filters, not with manual moves. I did many of the latter since I discovered the problem and had no further corruption happening (as I already said in comment 16, I've disabled all automatic filtering from my IMAP accounts for now).

Once we have better steps to reproduce or a patch, I'll give it a try. Sorry I cannot help more at this time.

Comment 40

[michael]

In opposite to other reporters I occasionally experienced this message corruption when manually moving messages, even within "Local Folders". But as said this happens rather rarely and I suspect it relates to more complex HTML mails. Maybe the code relies on another component to parse things correctly?

Comment 41

[Strauss van Dor]

In the hope that it may result useful to the cause, I'm copying the information I've been leaving in the thread: https://getsatisfaction.com/mozilla_messaging/topics/thunderbird_17_0_message_corruption_on_local_folders_upon_applying_filters

----

 I'm using Kaspersky, and I do NOT have the anti-virus option selected. 

--

I tried to recreate this issue in another machine yesterday, but couldn't.
The scenario wasn't exactly the same, tough. For instance, I created the (rather narrowed) folder structure and (simpler) filter after installing TB v17.0 whether on my real scenario those were inherited from previous versions. 

--

In response to Wayne who redirected us to comment 31 here:
-
This is what I did:
1. Full backup of my profile folder.
2. Downloaded both thunderbird-17.0a2.en-US.win32.zip to different folders in hope that using the zip archives will mess the less with my installation.
3. Run Earlybird from thunderbird.exe (one at a time, of course).
4. Applied one (different for each version) Filter on a Local Folder which already had downloaded messages; those I had been unable to move during these days.

Unfortunately, the result was the same with both versions:
Moved messages end up corrupted in their destination folder.

Bear in mind that both test where made with messages already downloaded from an IMAP account to Local Folders. Is this a valid try?
-
In case the problem starts in the "downloading" from IMAP (in my case) to Local Folders, I tried the whole process with both versions:
1. Downloaded a couple of messages from IMAP account to folder "A" in Local Folders.
2. Run a Filter to move those messages from A to B in Local Folders.

Same result with both versions:
Messages end up in B, but corrupted. 

--

I can also confirm this.
Manually moving messages corrupts them too.

--

I tried two other things:

1. Moving by means of drag and drop instead of Filters or Move command.
Obviously, same result: messages get corrupted in destination folder.
2. Repair Local source Folder before moving. This:
a. Made appear messages already moved or deleted.
b. Resulted in moved messages not to became corrupted.

Can anyone else please confirm this last point?
In summary:
If you run a Repair on the source folder before applying filters (or manually moving) it seems messages are moved without corruption. 

--

Could this bug be related?
https://bugzilla.mozilla.org/show_bug.cgi?id=769346

--

I finally reverted back to version 16.0.2 today.

In order to be able to move already downloaded messages, a Repair of the source folder was needed.
Newly downloaded messages are moved without corruption so far. 

----

That's it.
Thanks everyone for your time.

Comment 42

[Kent James (:rkent)]

So I now have the regression: Bug 769346

This is a one effective line change in nsMsgLocalMailFolder::InitCopyMsgHdrAndFileStream:

+  mCopyState->m_fileStream = NS_BufferOutputStream(mCopyState->m_fileStream, EIGHT_K);

Commenting out this line and recompiling with current trunk make the problem go away.

I found this by binary search on nightlies, but I see that comment 41 also pointed to it as a possible culprit.

Comment 43

[Frank Wein [:mcsmurf]]

I saw that bug when I looked at the checkin list, but did not think that it could have caused this bug here (as the patch looked quite simple and sane). Good that you found that out!

Comment 44

[Wayne Mery (:wsmwk)]

nice work!

so, this can happen both filter and not filter?


what about imap folder -> imap folder?   (if that is what michael is saying in comment 18)  should that be a different bug? 

(In reply to michael from comment #18)
> With 2.13.2 I did not have this problem. Since 2.14 I have this problem.
> 2.14.1 did not fix it.
> 
> BTW: This occasionally also happens when manually moving IMAP messages
> within an IMAP account. Most times it happens with complicated HTML mails.
> Have there been any significant changes to the way this are parsed and
> stored?

Comment 45

[:Irving Reid (No longer working on Firefox)]

(In reply to Kent James (:rkent) from comment #42)
> So I now have the regression: Bug 769346
> 
> This is a one effective line change in
> nsMsgLocalMailFolder::InitCopyMsgHdrAndFileStream:
> 
> +  mCopyState->m_fileStream =
> NS_BufferOutputStream(mCopyState->m_fileStream, EIGHT_K);
> 
> Commenting out this line and recompiling with current trunk make the problem
> go away.
> 
> I found this by binary search on nightlies, but I see that comment 41 also
> pointed to it as a possible culprit.

Good catch, Kent. The problem is that NS_BufferOutputStream isn't a transparent replacement for a standard output stream. If you don't call flush on the stream before you close it, the buffer is discarded. I'm sure I found documentation on that recently, but I can't track it down right now.

Comment 46

[Mark Banner (:standard8)]

(In reply to Irving Reid (:irving) from comment #45)
> (In reply to Kent James (:rkent) from comment #42)
> > So I now have the regression: Bug 769346
...
> > Commenting out this line and recompiling with current trunk make the problem
> > go away.
...
> Good catch, Kent. The problem is that NS_BufferOutputStream isn't a
> transparent replacement for a standard output stream. If you don't call
> flush on the stream before you close it, the buffer is discarded. I'm sure I
> found documentation on that recently, but I can't track it down right now.

I think there's also a potential problem in that we could be losing the reference for mCopyState->mfileStream, afaict NS_BufferOutputStream (and the items it creates) don't hold a reference to it.

I'm thinking we should back bug 769346 out from everywhere in time for the next releases, then re-work the fix for that bug with this issue in mind. Irving, Kent, thoughts on this?

Comment 47

[Kent James (:rkent)]

(In reply to Mark Banner (:standard8) from comment #46)

> 
> I'm thinking we should back bug 769346 out from everywhere in time for the
> next releases, then re-work the fix for that bug with this issue in mind.
> Irving, Kent, thoughts on this?

Yes that is the correct thing to do. Do we provide a patch to this bug to do that?

Comment 48

[Kent James (:rkent)]

AFAICT this does not depend on the complexity of the message, or on an filters. What is key to making it occur is to move more than one message at a time. In the IMAP case with incoming messages, this requires that the messages are sent while offline so that the filter moves more than one message at a time.

So I can reproduce with roughly these steps, on a profile with an IMAP account (and all folders containing existing messages, though I do not know if that it is important):

1) Select 2 messages in the IMAP inbox, and move them to a local folder (say "FauxInbox")

2) Select the 2 messages in "FauxInbox" and move them to another local folder (say "M3")

The moved messages will be corrupt at this point.

I suppose someone will ask me to do a unit test for this ...

Comment 49

[Mark Banner (:standard8)]

Having  patch attached to this bug would be easier for tracking purposes - we can then clone one of the bugs to re-fix the original.

Re unit test I suspect it may be difficult to get a test to consistently repeat this, though I could be wrong. There's hopefully a test around that it could be based on though.

Comment 50

[:Irving Reid (No longer working on Firefox)]

(In reply to Irving Reid (:irving) from comment #45)
> ... The problem is that NS_BufferOutputStream isn't a
> transparent replacement for a standard output stream. If you don't call
> flush on the stream before you close it, the buffer is discarded.

I was mis-remembering; it's the *safe* output stream (the one that writes to a temp file and then renames) that discards your work if you don't flush.

That said, if BufferedOutputStream is misbehaving, that could explain a *lot* of frustrating bugs - particularly our problems with finding corrupt config files at startup.

options for bug management:
1) Put the backout patch on this bug, and then either re-open or clone bug 769346 (with a clear comment about the problems with buffered output stream) to go back and revisit the performance problem.

2) Put the backout and re-open on bug 769346, and morph this bug into the "fix buffered output streams" bug. 

I think somebody does need to spend time investigating the buffered output streams; we use them in lots of places and need to be able to trust them.

Comment 51

[Mark Banner (:standard8)]

Attachment: The fix

Here's the backout patch (hg backout of https://hg.mozilla.org/comm-central/rev/a9f658106adb), I'm landing without review as Kent, Irving and myself have all agreed this is the right thing to do. Hence also a=me for all branches.

Comment 52

[Mark Banner (:standard8)]

Backed out bug 769346 from comm-central: https://hg.mozilla.org/comm-central/rev/1a577ed6eb9a

and branches:

https://hg.mozilla.org/releases/comm-aurora/rev/1204aac48e3a
https://hg.mozilla.org/releases/comm-beta/rev/1733d86c81ba
https://hg.mozilla.org/releases/comm-esr17/rev/fea4edb68f01

Comment 53

[peterj]

I just want to thank everyone who contributed to finding and stomping this bug.

I really appreciate the effort that was taken to localize the issue and understand it. I wish I had been able to help more.

Season's Greetings to everyone and my hat off to your expertise.

I look forward to turning my filters back on. Moving messages one by one by dragging and dropping is getting old.

Cheers,
Peter

Comment 54

[R.N. (Roger) Folsom]

(In reply to Mark Banner (:standard8) from comment #52)

I very much appreciate all of the work done by all of the contributors to solving this problem.

Unfortunately, I'm still stuck.  I have looked at the contents of your last two posts and at each of your last two messages links,
https://hg.mozilla.org/comm-central/rev/a9f658106adb
and
https://hg.mozilla.org/comm-central/rev/1a577ed6eb9a

but since I am not a programmer (I learned of this bug earlier today, from The SeaMonkey 2.14.1 Release Notes, at http://www.seamonkey-project.org/releases/seamonkey2.14/), I have no idea what I should do next.

So for now, I have to keep my 25 Message Filters disabled, until a new SeaMonkey version appears with the fix.  Correct?

I haven't had time to check the contents of my Message Filter folders (they're primarily "todo" messages), so I don't know if bug 815012 has bitten me.

But since I use POP and not IMAP, and am using WinXPsp3 rather than Windows7, and my Local Folders do not include an Inbox folder, maybe bug 815012 doesn't apply to me?

I hope that you or someone else responds to this message and explains what, if anything, non-programmer SeaMonkey 2.14.1 users should do.

On the other hand, everyone contributing to this location have already done much, so I will post a version of this message on Mozillazine.

Thanks to everyone who helped solve this problem.

R.N. (Roger) Folsom

Comment 55

[:aceman]

The fix is in the C++ part of Thunderbird. If you can't build Tb yourself please wait for TB 17.0.1 (or it's Seamonkey equivalent) that will contain the fix. It should be out soon (6 weeks after the release of TB 17).
You probably can't do anything else (short of downgrading to TB16 that does not contain the change that causes this problem).

Comment 56

[Frank Wein [:mcsmurf]]

The SeaMonkey 2.15 release will include the fix for this bug. You can enable your mail filters again as soon as you have SeaMonkey 2.15 installed. SeaMonkey 2.15 will probably be released in the second week of January.

Comment 57

[R.N. (Roger) Folsom]

(In reply to Frank Wein [:mcsmurf] from comment #56)
> The SeaMonkey 2.15 release will include the fix for this bug. You can enable
> your mail filters again as soon as you have SeaMonkey 2.15 installed.
> SeaMonkey 2.15 will probably be released in the second week of January.

Thanks VERY much for that information.  TheRube posted it also on MozillaZine, at
http://forums.mozillazine.org/viewtopic.php?f=40&t=2634017&p=12568733#p12568733

Roger Folsom

Comment 58

[AirBoss]

Agree with this:

"Just a data point. In my case, it began immediately after the 17 update."


DISAGREE with this; I see EXACTLY THE OPPOSITE BEHAVIOR:


"Manually MOVING from INBOX to other folders (dragging and dropping has never resulted in a corruption."

I see corruption on MOVE/COPY/COMPACT, NOT on filter use.


DISAGREE with this:

"Filtered move from INBOX to another folder has a high probability of corruption."  

NEVER SEE CORRUPTION w/FILTERS.

Comment 59

[AirBoss]

ABSOLUTELY NOT "FIXED"

Comment 60

[Wayne Mery (:wsmwk)]

(In reply to AirBoss from comment #59)
> ABSOLUTELY NOT "FIXED"

AirBoss, do you realize
a) different people are affected in different ways - some affected by filters, some not 
b) your comment is not specific enough to anyone to determine how you are being affected - to be more blunt, what version did you just test?
c) this isn't fixed in TB17.0 - it's fixed in 20, 19 and 18, and shortly 17.0.1 (ref comment 55)
d) the FIXED resolution setting below this comment means it is fixed in the *development* branch of Thunderbird

Comment 61

[AirBoss]

Wayne,

Thanks, but not clear the behavior will be "FIXED" in 17.0.1 since I don't see these errors on FILTER, whereas I always seeing them on MOVE/COPY and COMPACT.

Now using 17.0 and IMAP on WinXP

Comment 62

[AirBoss]

In addition to the observations in comment 58, which suggest the "fix" isn't even directed at the observed behavior.

Comment 63

[Conrad Yang]

I'm also having this issue with 17.0 on Ubuntu Linux (17.0+build2-0ubuntu0.12.10.1). Manually or Filter move of messages from IMAP folders to local folders WITH DELETED MESSAGES (without compact) will result in partial messages. 16.0.1+build1-0ubuntu1 doesn't have this issue.

Comment 64

[:aceman]

Please upgrade to 17.0.1 before reporting new findings.

Comment 65

[AirBoss]

How does that work when on 17.0, on the release channel, and no update is shown as available?

Comment 66

[:aceman]

Yeah, sorry it is not out yet, so please wait a few days.

Comment 67

[R.N. (Roger) Folsom]

(In reply to AirBoss from comment #58)
> . . . 
> I see corruption on MOVE/COPY/COMPACT, NOT on filter use.
> . . .

I am using SeaMonkey 2.14.1 rather than Thunderbird, and my internet connection is POP rather than IMAP, and I do realize that my questions below for you may be inappropriate, so please excuse me for trying anyway:

What do you mean by MOVE/COPY/COMPACT?  Is that a quick sequence of actions?  I think of move and copy as being alternatives, and I compact SeaMonkey mail accounts (I have three; two have inboxes and the third is Local Folders) only after I have deleted their trash folder contents, when I'm about to do a backup image (to an external hard drive).

So I am guessing that MOVE/COPY/COMPACT are not a quick sequence of actions, but instead that you are saying that your experience is that corruption can occur when ANY of the following actions are taken:  Move, Copy, or Compact.

Is my guess correct?  If my guess is wrong, please clarify what the package MOVE/COPY/COMPACT means.

Thanks for any comments, help, or answers!

P.S.  I have not yet discovered any corrupted messages, but that may be merely because I have been lucky, or because I have overlooked corrupted messages located in non-inbox folders.  (Some of my non-inbox folders got their content from filters;  others got their content by my dragging messages out of inboxes.)

Comment 68

[AirBoss]

Yes, corruption occurs on any MOVE, COPY or COMPACT of local files.

Comment 69

[R.N. (Roger) Folsom]

(In reply to AirBoss from comment #68)
> Yes, corruption occurs on any MOVE, COPY or COMPACT of local files.

Thanks very much for clarifying that.  But it is a bit discouraging.

Comment 70

[:aceman]

OK, TB 17.0.2 is out now, please try it.

Comment 71

[AirBoss]

17.0.2  -- so far, so good; no corruption; improvement over 17.0

Comment 72

[Conrad Yang]

On 17.0.2 (20130105220954, ubuntu raring 17.0.2+build1-0ubuntu1) for 2 days, no corruption.

Comment 73

[Brian Vanderburg]

On 17.0.2, it's working like normal again.  No corruption during first filter from IMAP to Local or second manual filter from Local to sorted folders.