95584 - Mail manged/lost (possibly filter problem)(possibly 'compact' problem)

Status: VERIFIED FIXED

(MailNews Core :: Filters, defect)

Description

[emmet]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2+)
Gecko/20010727
BuildID:    2001072703


Attachment 1 [details] [diff] : a clipped copy of the mail spool file off the
unix machine
Attachment 2 [details] [diff] : the first email shown in Mozilla to go wrong 
(inspect the end of the file)
Attachment 3 [details] [diff] : the second email shown in Mozilla to go wrong
(even more messed up)

There are 4 emails in the mail spool file (dean@cogs,
owner-zoology-l, jonathlm@cogs, [e-mail])
I have a filter set up that filters to the *delete* any mails
sent to: [e-mail] -- which should have
been the one from owner-zoology-l.

What happened -- three emails appeared in the email summary window of the
Inbox of Mozilla:

The 'Richard Coates' (dean@cogs) email was fine.

Attachment 2 [details] [diff] shows the contents as displayed by mozilla of
the one denoted by 'Johnathan Matthews' subject 'Domestic Diary Use'

Attachment 3 [details] [diff] show the contents of the one by 'Amanda B Hellberg' (subject '')
(actually this was the view shown by <ctrl-U>, the windows contents
showed the clipped HTML version.

CLEARLY the emails have been manged and the *actual* email
from Amanda has been lost.

This is the third time this has happened over the last couple of 
months and 3/4 versions of Mozilla but the first time I
delved into what was going on. Obviously it is a very
serious bug.


Reproducible: Didn't try
Steps to Reproduce:

Comment 1

[emmet]

Attachment: attachement 1: the clipped mail spool file

Comment 2

[emmet]

Attachment: attachment 2: the email under the name 'Johnathan Matthews', subject 'Domestic Diary Use'

Comment 3

[emmet]

Attachment: Attachment 3 : the email under name 'Amanda B Hellberg', subject ''

Comment 4

[emmet]

I just compacted my Inbox and found that all the manged messages (and ONLY
the mangled messages), even though they had been deleted in the mail summary
window, reappeared in their complete and proper form.

Might mean that the problem isn't a filter one but a mail database one?

Comment 5

[David Castro-Diephouse]

This bug has also been bothering me for several versions.  Here is how it
manifests for me:  

I have filters that move mailing-list messages to list-specific folders.  The
filter appears to move the headers properly from the Inbox.msf file to (in this
case) jboss-dev.msf.  However, the message bodies do not get moved properly. 
First, the message bodies are left unchanged in the Inbox file.  Second, mangled
versions of the messages are appended to the jboss-dev file.  Because the
messages are left intact in the original folder, they can be retrieved by
compacting the Inbox.

I will attach the relevant portions of rules.dat, Inbox, and jboss-dev.

Comment 6

[David Castro-Diephouse]

Attachment: A portion of the rules.dat file containing the filter

Comment 7

[David Castro-Diephouse]

Attachment: A portion of the mail file Inbox containing the filtered messages (unchanged by the filter)

Comment 8

[David :Bienvenu]

David, what platform do you run on, and what build are you running?

Comment 9

[David Castro-Diephouse]

Attachment: A portion of the mail file jboss-dev, containing the mangled messages after filtering

Comment 10

[David Castro-Diephouse]

I am on Win2K Pro SP1
Mozilla 0.9.3
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.3) Gecko/20010801

Comment 11

[Håkan Waara]

I don't understand much of this bug. Nonetheless, let me ask you a few questions:

1. Regarding the filter, did you check your "Local Folders" account's folders
for the messages?
2. Have you tried setting up a new profile and doing a new mail account? Was
this bug possible to reproduce?

Thanks

Comment 12

[David Castro-Diephouse]

The mangled messages (in jboss-dev) seem to consist of the following:

Message 1 with the following change
3c3
< X-Mozilla-Status: 0000
---
> X-Mozilla-Status: 0011

Message 1 (unchanged)

Message 2, first 1537 characters

Message 1, first 3059 characters

Message 3 does not appear at all

Comment 13

[Navin Gupta]

The filters do not work the way you have described. We truncate the inbox
when a message that fits the criteria has been moved to the destination 
folder. Not sure how the messages are still intact in inbox. looks like the 
message offsets are all wrong. 

Comment 14

[David Castro-Diephouse]

The target folder for the filter is in the Local Folders account.  That is where
the mangled messages are showing up.

I have reproduced this behavior several times (on different milestone builds). 
I have blown away the entire Application Data\Mozilla folder (which contains the
profiles) and started over.  

Basically I've never gotten mail to move messages to a folder without
encountering this bug.  I assume that if this problem were universal other
people would have reported it so I will keep trying.

Comment 15

[David Castro-Diephouse]

I believe I've tracked it down.  I was searching my hard drive and found old
copies of mozregistry.dat and mozversion.dat.  Since I blew those away, all of
the mozilla bugs that I have been cursing for the last 6 months went away.  I
don't know if this one will rear its head again tomorrow, but I have
successfully received several messages now without any mangling by the filters.
 I don't know if that is any help to emmet or not, but I wish him luck.

Comment 16

[Gervase Markham [:gerv]]

I've seen mail truncation issues recently; it's not a presentation issue because
when I looked with 4.x at my IMAP mail store, it was still truncated. I recently
 moved to a new server, so that _could_ be it - but, given this bug report, I
suspect not. This is very bad.

Gerv

Comment 17

[David Castro-Diephouse]

Well, it seems to have reared its ugly head again.  I had a few days off when
the filters seemed to do just the right thing.  Then, today, the 12:59 download
worked fine, but the 13:01 download was completely mangled.  Almost as if I left
mozilla running for too long and it got angry.

I also note that sometimes that when I move mail manually between folders, the
move seems to work fine on the front end, but if i look in the folder in a text
editor later, I find the messages still in the original folder ( as well as in
the new one ).  The front end seems to ignore the messages that aren't supposed
to be there, and everything works fine, but I thought Id point it out.

I can post more mangled mail if that is useful to anyone.  If I wanted to throw
in debug code at home to look for this, where would I look for the relevant
code?  Also, is it still necessary to get VC++ to build on windows?  

Comment 18

[Mark Scott]

I'm not sure if this should be a separate bug entry, but mail appears to be lost
 if Mozilla is compacting a folder whilst a new message is downloaded into it.
(Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.3) Gecko/20010801)

Comment 19

[Håkan Waara]

Mark, yes that's another bug. Try a newer build, however, before you report the
bug. Your build is roughly one month (!) old.

Comment 20

[Navin Gupta]

Mark your problem has been fixed. You should download  latest build and try it. 

Comment 21

[Håkan Waara]

Marking as worksforme.

Comment 22

[emmet]

I initally reported this bug (see the first two or three comments and
attachements) and am unconvinced it has been resolved since it has occured
two or three times over three months so requires more than just a circumspect
piece of attention considering its severity.

Comment 23

[Peter Arien]

I got hit very hard with something like this yesterday. I openend mozilla-mail
(build 2001082808). When I clicked the Inbox, it asked me if I wanted it to
compress all mailboxes. I said Yes. While it was compacting my mailboxes, I
pressed the 'Get Msg' button. Result: crash and EMPTY INBOX !!! Nothing there
anymore. The Inbox and Inbox.msf files were gone.

Comment 24

[Navin Gupta]

Peter, I am sorry. I made an attempt earlier of fixing this bug in bug 91731
and it did appear the bug was fixed because when I tested I was getting 
folder locked when both operation downloading message and compacting went 
off at the same time. But now I have found out that there are some 
problem with ownership issues of the semaphores. I will attach a new patch
to fix the problem. 
     
  

Comment 25

[Navin Gupta]

I don't know how you crashed, I am not able to crash. 

Comment 26

[Navin Gupta]

Attachment: proposed fix

Comment 27

[Navin Gupta]

The fix is to make sure the correct owner holds the lock on the folder. 
I have tested it rigorously both ways compacting go on -trying to get message
and vice-versa and it works ! david, need review. 

On a different note, I just want to say that there could be a scenario if
we are compacting all folders and we try to get messages. If inbox is not 
the folder being compacted at that moment, downloading messages will start 
but if a folder on which filter is set, is being compacted at that moment,
then filter will not work. All the messages that were supposed to be filtered
will be found in inbox.

Also in debug build I get this assertion after compact is over, when I click on 
any message. It has been there for a while. I don't think so it is harmful. 

###!!! ASSERTION: outEnv: '0', file d:\temp\mozilla\db\mork\src\morkConfig.cpp,
line 78

Comment 28

[David :Bienvenu]

what's the stack trace for the assertion? That worries me. And when you say "the
filter will not work", does it not work in an orderly fashion? I.e., does the
move code try to get the semaphore and fail so the move is not done? That's
fine, if that's the case. I just want to make sure why the filter doesn't work.

Comment 29

[Navin Gupta]

>I.e., does the
>move code try to get the semaphore and fail so the move is not done? That's
>fine, if that's the case.

yes, that is exactly what happens.

I have been living with the assertion for a while now and it also happens
when I click on the newly downloaded messages. Here is the stack trace..

nsDebug::Assertion(const char * 0x0487cba4, const char * 0x0487cab0, const char 
* 0x0487ca84, int 78) line 290 + 13 bytes
mork_assertion_signal(const char * 0x0487cba4) line 78 + 31 bytes
morkEnv::NewError(const char * 0x0487cfec) line 369 + 19 bytes
morkStore::AddAlias(morkEnv * 0x06f35ad0, const morkMid & {...}, unsigned int 0) 
line 975
morkBuilder::OnAlias(morkEnv * 0x06f35ad0, const morkSpan & {...}, const morkMid 
& {...}) line 635
morkParser::ReadAlias(morkEnv * 0x06f35ad0) line 1042
morkParser::ReadDict(morkEnv * 0x06f35ad0) line 1321
morkParser::ReadContent(morkEnv * 0x06f35ad0, unsigned char 1) line 1413
morkParser::ReadGroup(morkEnv * 0x06f35ad0) line 1207
morkParser::ReadAt(morkEnv * 0x06f35ad0, unsigned char 0) line 1239
morkParser::ReadContent(morkEnv * 0x06f35ad0, unsigned char 0) line 1416 + 16 
bytes
morkParser::OnPortState(morkEnv * 0x06f35ad0) line 1450 + 14 bytes
morkParser::ParseLoop(morkEnv * 0x06f35ad0) line 1506 + 12 bytes
morkParser::ParseMore(morkEnv * 0x06f35ad0, int * 0x0012d6d0, unsigned char * 
0x06f36c38, unsigned char * 0x06f36c39) line 1545
morkThumb::DoMore_OpenFileStore(morkEnv * 0x06f35ad0) line 433
morkThumb::DoMore(morkEnv * 0x06f35ad0, unsigned int * 0x0012d74c, unsigned int 
* 0x0012d744, unsigned char * 0x0012d750, unsigned char * 0x0012d748) line 353 + 
12 bytes
orkinThumb::DoMore(nsIMdbEnv * 0x06f33578, unsigned int * 0x0012d74c, unsigned 
int * 0x0012d744, unsigned char * 0x0012d750, unsigned char * 0x0012d748) line 
230
nsAddrDatabase::OpenMDB(nsAddrDatabase * const 0x06f2e1a0, nsFileSpec * 
0x06f2c560, int 1) line 653 + 34 bytes
nsAddrDatabase::Open(nsAddrDatabase * const 0x05bef340, nsFileSpec * 0x06f2c560, 
int 1, nsIAddrDatabase * * 0x038de3ec, int 1) line 566 + 20 bytes
nsAbAddressCollecter::OpenHistoryAB(nsIAddrDatabase * * 0x038de3ec) line 233 + 
35 bytes
nsAbAddressCollecter::CollectAddress(nsAbAddressCollecter * const 0x038de3e0, 
const char * 0x06f28240) line 114 + 38 bytes
nsAbAddressCollecter::CollectUnicodeAddress(nsAbAddressCollecter * const 
0x038de3e0, const unsigned short * 0x06f0b640) line 82 + 16 bytes
XPTC_InvokeByIndex(nsISupports * 0x038de3e0, unsigned int 4, unsigned int 1, 
nsXPTCVariant * 0x0012dcb0) line 139
XPCWrappedNative::CallMethod(XPCCallContext & {...}, XPCWrappedNative::CallMode 
CALL_METHOD) line 1952 + 42 bytes
XPC_WN_CallMethod(JSContext * 0x0260c560, JSObject * 0x02ac5ad8, unsigned int 1, 
long * 0x056b1f60, long * 0x0012dee8) line 1254 + 14 bytes
js_Invoke(JSContext * 0x0260c560, unsigned int 1, unsigned int 0) line 807 + 23 
bytes
js_Interpret(JSContext * 0x0260c560, long * 0x0012ec8c) line 2719 + 15 bytes
js_Invoke(JSContext * 0x0260c560, unsigned int 3, unsigned int 2) line 824 + 13 
bytes
nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJSClass * const 0x047382e0, 
nsXPCWrappedJS * 0x04738250, unsigned short 3, const nsXPTMethodInfo * 
0x00d2cba8, nsXPTCMiniVariant * 0x0012f1d4) line 1022 + 21 bytes
nsXPCWrappedJS::CallMethod(nsXPCWrappedJS * const 0x04738250, unsigned short 3, 
const nsXPTMethodInfo * 0x00d2cba8, nsXPTCMiniVariant * 0x0012f1d4) line 427
PrepareAndDispatch(nsXPTCStubBase * 0x04738250, unsigned int 3, unsigned int * 
0x0012f284, unsigned int * 0x0012f274) line 100 + 31 bytes
SharedStub() line 124
nsMimeHtmlDisplayEmitter::WriteHTMLHeaders(nsMimeHtmlDisplayEmitter * const 
0x06f407e0) line 214
nsMimeHtmlDisplayEmitter::EndHeader(nsMimeHtmlDisplayEmitter * const 0x06f407e0) 
line 297
mimeEmitterEndHeader(MimeDisplayOptions * 0x06f42730) line 1742 + 12 bytes
MimeMessage_write_headers_html(MimeObject * 0x06f42660) line 764 + 12 bytes
MimeMessage_close_headers(MimeObject * 0x06f42660) line 390 + 9 bytes
MimeMessage_parse_line(char * 0x05634b88, int 2, MimeObject * 0x06f42660) line 
256 + 9 bytes
convert_and_send_buffer(char * 0x05634b88, int 2, int 1, int (char *, unsigned 
int, void *)* 0x03ab5730 MimeMessage_parse_line(char *, int, MimeObject *), void 
* 0x06f42660) line 168 + 15 bytes
mime_LineBuffer(const char * 0x05d15aee, int 2373, char * * 0x06f42688, int * 
0x06f42690, unsigned int * 0x06f42698, int 1, int (char *, unsigned int, void 
*)* 0x03ab5730 MimeMessage_parse_line(char *, int, MimeObject *), void * 
0x06f42660) line 255 + 29 bytes
MimeObject_parse_buffer(char * 0x05d151f8, int 4667, MimeObject * 0x06f42660) 
line 255 + 49 bytes
mime_display_stream_write(_nsMIMESession * 0x06f425d0, const char * 0x05d151f8, 
int 4667) line 836 + 20 bytes
nsStreamConverter::OnDataAvailable(nsStreamConverter * const 0x06f42c40, 
nsIRequest * 0x06f33874, nsISupports * 0x06f36440, nsIInputStream * 0x06f32a60, 
unsigned int 148694938, unsigned int 4667) line 902 + 24 bytes
nsDocumentOpenInfo::OnDataAvailable(nsDocumentOpenInfo * const 0x06f33360, 
nsIRequest * 0x06f33874, nsISupports * 0x06f36440, nsIInputStream * 0x06f32a60, 
unsigned int 148694938, unsigned int 4667) line 244 + 46 bytes
nsMailboxProtocol::ReadMessageResponse(nsIInputStream * 0x06f32a60, unsigned int 
148694938, unsigned int 4667) line 543 + 78 bytes
nsMailboxProtocol::ProcessProtocolState(nsIURI * 0x06f36444, nsIInputStream * 
0x06f32a60, unsigned int 148694938, unsigned int 4667) line 634 + 20 bytes
nsMsgProtocol::OnDataAvailable(nsMsgProtocol * const 0x06f33870, nsIRequest * 
0x06f33d14, nsISupports * 0x06f36440, nsIInputStream * 0x06f32a60, unsigned int 
148694938, unsigned int 4667) line 245 + 32 bytes
nsOnDataAvailableEvent::HandleEvent() line 178 + 70 bytes
nsARequestObserverEvent::HandlePLEvent(PLEvent * 0x06f45fb4) line 65
PL_HandleEvent(PLEvent * 0x06f45fb4) line 590 + 10 bytes
PL_ProcessPendingEvents(PLEventQueue * 0x00a066b0) line 520 + 9 bytes
_md_EventReceiverProc(HWND__ * 0x0bb10738, unsigned int 49404, unsigned int 0, 
long 10512048) line 1071 + 9 bytes
USER32! 77e71820()
00a066b0()

Comment 30

[David :Bienvenu]

OK, your history address book is corrupted - you should move it away (rename it
from history.mab to something else), and start over, and make sure that it
doesn't get corrupted again when you test your fix again (it's very unlikely
that this bug or the fix corrupted your history ab - more likely you got bit by
turbo mode). If that works OK, then r=bienvenu

Comment 31

[Navin Gupta]

Thanks david for helping me to get rid of that annoying assertion. 
Everything works fine now. 

Comment 32

[Scott MacGregor]

Comment on attachment 49994 [details] [diff] [review]
proposed fix

sr=mscott. You still need to get an r= though =).

Comment 33

[Scott MacGregor]

plussing for the branch.

Comment 34

[Scott MacGregor]

.9.5

Comment 35

[Navin Gupta]

fixed on trunk

Comment 36

[Navin Gupta]

I missed one thing that is to clean up temp files in case compact cannot
start. need r/sr

+    NS_ASSERTION(0, "Some other operation is in progress on this folder");
+    m_folder->NotifyCompactCompleted();
+    if (m_compactAll)
+      CompactNextFolder();
+    else
+    {
+      CleanupTempFilesAfterError();
+      return rv;
+    }

Comment 37

[Navin Gupta]

More accurate patch...

@@ -303,8 +303,11 @@
     if (m_compactAll)
       CompactNextFolder();
     else
+    {
+      CleanupTempFilesAfterError();
       return rv;
   }
+  }

Comment 38

[David :Bienvenu]

r=bienvenu

Comment 39

[Navin Gupta]

*** Bug 100723 has been marked as a duplicate of this bug. ***

Comment 40

[Jaime Rodriguez, Jr.]

Pls come by @ 3:30 with Mscott.

Comment 41

[Jaime Rodriguez, Jr.]

check it in - PDT+

Comment 42

[Navin Gupta]

fixed the issue of temp files on trunk. 

Comment 43

[Navin Gupta]

fixed on branch as well. I cannot close this bug because only one of 
the issues raised in this bug has been fixed, the other one - as originally
filed by reported is not reproducible, leaving it open.

Comment 44

[Navin Gupta]

*** Bug 97572 has been marked as a duplicate of this bug. ***

Comment 45

[Scott MacGregor]

navin, we still need to mark this closed so QA knows to test it. Can you file a
spin off bug for the reporter's original problem then tell us that new bug
number here?

Comment 46

[laurel]

OK Sep24-25 commercial 0.9.4 branch win98.
Tested POP, normal default thread pane view/All/sort by date:
Auto-compact folders with biff on/auto-download new messages
Auto-compact folders with "get messages at startup"
Manual compact folder/Inbox biff on/auto-download new messages
Manual compact all folders biff on/auto-download new messages
Auto-compact with manual Get Msg
Manual compact folder with manual Get Msg
Manual compact all folders with Get Msg
Incoming messages mix of filtered, not filtered.
Messages filtered to folder on same account.
Messages filtered to folders under Local Folders.

In all cases:
1. If biff went off while compacting in progress, alert displayed that folder
was in use. (Side Issue here: is this alert meaningful for user when they didn't
activate Get Msg themselves, but was part of automatic process "biff"? May log
separate bug about alert in such case.)
2.  After compacting finished, biff picked right back up and all new messages,
filtered or not, were accounted for.
3.  After compacting finished, Inbox was in good state, no obvious corruption,
no missing messages, no blank pane.
4.  After compacting finished, filtered messages accounted for and able to be
read, no obvious corruption.
5.  After compacting finished for manual Get Msg cases, able to Get Msg again
and all new messages accounted for no corruption.

Comment 47

[laurel]

More verification info: 
Cases (still win98 sep25 branch) where the destination folder was being
compacted while a Get Msg was done on Inbox which had new messages to filter to
the folder in process of compaction were OK, too.  In these cases I tried, there
was no alert (makes sense, OK) and the message wound up remaining in the Inbox
(OK, filter couldn't complete because destination was unavailable) and the
filter was left enabled and further get messages on the same inbox/filter plan
(after compacting was completed) were successful.

Comment 48

[laurel]

Looks OK with sep25 branch build, linux rh6.2.

Comment 49

[emmet]

The original bug has been now registered under bug 101733

I have added some new details (the bug reoccured today) and it now looks
like a misregistration between the Inbox and the Inbox.msf file that
occurs after a mail fetch.

Comment 50

[laurel]

OK with sep26 commerical 0.9.4 branch build on Mac OS X.
Considering verified on branch.
Even though I did some testing on trunk several days ago, I'm adding vtrunk
keyword for more full verification on trunk. I won't get to that until branch
release.

Comment 51

[laurel]

OK with nov 19 commercial trunk build: win98, mac OS X, linux rh6.2