Closed Bug 403043 Opened 17 years ago Closed 14 years ago

Hangs when message filters are run automatic or manual (with > 2 GB folders)

Categories

(MailNews Core :: Backend, defect)

1.8 Branch
x86
Windows XP
defect
Not set
critical

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: schneegans, Unassigned)

Details

(Keywords: hang, Whiteboard: closeme 2010-02-15)

User-Agent:       Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
Build Identifier: version 2.0.0.6 (20070728)

When I have some message filter, TB freezes while reading the first message.
Any kind of message, apparently not related with spam.
When the TB window is closed (X top right), the window closes, but TB cannot be ren again, there is a message telling that it is still running.
The task manager shows that Thundebird.exe is still running.
TB has to be stopped from the task manager.
If the file MsgFilterRules.dat is deleted, all messages are read correctly (of course unfiltered) and the spam filter still works properly.

Curiously enough, I have several mail accounts, on the same and different servers, only on is affected. Others still have their filters working.


Reproducible: Always

Steps to Reproduce:
Happens every time I use TB.
Actual Results:  
TB freezes reading the first message (probably executing the filters)

Expected Results:  
Not freeze.

Just started to day, before it worked fine.
May be related to the Folder the filter is supposed to file the message?
Compacting the database does not help.
Condected some more testing:

The problem seems not to be related to message filters as such, but only to moving a message to its location because of a filter.
I deleted all filters and redefined just one with a condition that never happens. then I have no problem.
The problem is only when a message is found to match a filter.
imap?
Keywords: hang
Summary: Freezes while reading the first message when message filters are active → Freezes while reading the first message when message filters are active or msg moved manually, and thunderbird doesn't close properly
No,
In fact, the problem has nothing to do with getting messages, but only with running filters (which happens while reading messages).
I can read all messages with filters deactivated, then if I activate and run them, then I have the same problem: TB hangs up...
I can also run filters on by one, and suddenly TB hangs on one of them.
If I slightly change the string in this filter, it will run with no problem, but then another on will make TB hangs up.
Ah ah! Now I know that the problem only arises when the input folder contains a large amount of messages. I deleted half of them and filters are running fine now.
I keep messages for one year, and this can make quite a big folder.

So it must be some kind of overflow in the process, or so. 
If you posted your report with an interest in getting the problem fixed then what you've posted so far, while interesting, is not sufficient.  What's the filter criteria and filter action that fails and the one that works?  how many messages in folder when it failed, when it worked?  source and target folder types (imap, pop, local)?  etc.
>>What's the filter criteria

None in particular.

>> and filter action that fails

Just have filters active while receiving mail, or execute filters once messages are recieved.

>> and the one that works?

Do not use filters!

>>  how many messages in folder when it failed, 

Not sure it depends on the number of messages. I always kept messages for one year and never had a problem with filters before. Now I don't really recieve more messages, but much more and more with videos attached, you know, jokes etc.
So I think it more depends on the size of the input bin.
All I can say is that it is now about 1.6 Gb and I have no problem.
When it was 4 Gb, I was having the problem.
IMHO it is some kind of overflow when the file is too big.
BTW, I can't believe the messages are kept in their integrity with attachements
all in the same file. This is also the reason re-indexing takes so long.
All attachement should be saved as files. Really poor design :-(

>>source and target folder

Inbox

>>types (imap, pop, local)?  etc.

Doesn't matter
(In reply to comment #6)
> >>What's the filter criteria
> 
> None in particular.

contradicts comment 3

 
> >> and filter action that fails
> 
> Just have filters active while receiving mail, or execute filters once messages
> are recieved.
> 
> >> and the one that works?
> 
> Do not use filters!
> 
> >>  how many messages in folder when it failed, 
> 
> Not sure it depends on the number of messages. 

contradicts comment 4


> So I think it more depends on the size of the input bin.
> All I can say is that it is now about 1.6 Gb and I have no problem.
> When it was 4 Gb, I was having the problem.

Despite your assumptions it would still be helpful to hear you say how many messages. (total message count for inbox)


> >>source and target folder
> Inbox

target is inbox?


> >>types (imap, pop, local)?  etc.
> 
> Doesn't matter

this is not a helpful statement unless you tested all of them and all failed.  Assuming you didn't, are we to assume you are using pop? (comment 4)  If pop, do you have multiple accounts, and do you use a global inbox?
>> None in particular.

>>contradicts comment 3

Exact, now I know better about the problem.

>>contradicts comment 4

Absolutely not: what I mean is that it depends on the size of the file, not on the number of messages in the file. One could have a small file with thousands of messages, or one huge file with only a hundred of messages.
My feeling is that the main factor here is the size, not the number. 


> So I think it more depends on the size of the input bin.
> All I can say is that it is now about 1.6 Gb and I have no problem.
> When it was 4 Gb, I was having the problem.

>>Despite your assumptions it would still be helpful to hear you say how many
>>messages. (total message count for inbox)

I'd be glad to... where do I find the "total message count"?

>>target is inbox?

Of course not, but I tell you the target does not matter.
All my filters have a "move to" action. 
Anyway, all targets are folders specific to my implementation.



>>types (imap, pop, local)?  etc.
> 
> Doesn't matter

>>this is not a helpful statement unless you tested all of them and all failed. 

Although I fist got the problem while recieving messages, as I specified later,
the problem has nothing to do with the recieving process.
I can deactivate all filters, get all messages with no problem, then close the connection, run filters by hand on the inbox, and pouf! It freezes.
So gime a break with the type of connection, the problem is only with the filtering process, period.
(In reply to comment #8)
>>
> >>contradicts comment 4
> 
> Absolutely not: what I mean is that it depends on the size of the file, not on
> the number of messages in the file. One could have a small file with thousands
> of messages, or one huge file with only a hundred of messages.
> My feeling is that the main factor here is the size, not the number. 

then what's approximate smallest message that fails?


> > So I think it more depends on the size of the input bin.
> > All I can say is that it is now about 1.6 Gb and I have no problem.
> > When it was 4 Gb, I was having the problem.
> 
> >>Despite your assumptions it would still be helpful to hear you say how many
> >>messages. (total message count for inbox)
> 
> I'd be glad to... where do I find the "total message count"?

tools > optons > advanced > general pane: check off "show expanded columns"
in folder pane click small box above slider on the right.

sending to filters, but not convinced this is strictly a filter bug.
signing off, hopefully someone else can help you.
Component: General → MailNews: Filters
Product: Thunderbird → Core
QA Contact: general → filters
>>then what's approximate smallest message that fails?

I'm talking of the size of the INBOX as Gb, not the size of a particular message.

>>tools > optons > advanced > general pane: check off "show expanded columns"
in folder pane click small box above slider on the right.

Still do not see any total message count :-( 
All I ca see is the total messages per thread.
I have same problem.
- imported 6GB of mails from Outlook
- added 5 POP3s accounts
- it worked until I created a simple filter (if sender is xxx move to folder yyy, added to the Local). Size of Inbox is 2.5GB, the yyy folder is 8MB.
- TB freezes while receiving first message
James, please file a new bug.

Claude, let's start over from the beginning 
- can you cause this with version 2.0.0.14, new profile + newly created filter?
- if that fails, does it also fail using trunk version?
 ftp://ftp.mozilla.org/pub/thunderbird/nightly/latest-trunk/


the shutdown problem is more interesting to me, so once this is figured out we need to return to that.
and please specify what you tested, whether ...
target folder is : imap, local, pop
source folder is : imap, local, pop


(another potentially similar bug is bug 383027)
Summary: Freezes while reading the first message when message filters are active or msg moved manually, and thunderbird doesn't close properly → Hangs when message filters are run automatic or manual
>>Claude, let's start over from the beginning

There is nothing new. I got the problem again a couple of weeks ago.
My inbox bin was over 2 gig.
AGAIN, it has nothing to do with the retrieving of the messages, should it be imap, local, pop or whatever. The problem is with moving a message to an other folder, depending on the filter.
The proof is that you can disable retrieving of messages, just apply the filters on the inbox, and poof! The only thing you can do is to call the task manager and kill the program.
NOTHING to do with the type of filter either.

IMO, the problem is some kind of overflow in some calculation of addresses or something while relocating the message or updating the index, or whatever.
It ONLY happens when the inbox is over 2 gig.

PLEASE do not ask me any question again about the mail protocol or message retrieving. The problem is not there.
Claude don't be so ignorant if people asking for something particular that really matters! As example I wonder how do you disable retrieving messages on IMAP and move to another IMAP folder?
Anyway what File System on your primary partition if its FAT32 there is limit 4GB per file and because TB using mailbox format this may cause of your problem.
I will do test on my own, cuz I have user with really HUGE Inbox and about 20K messages in it and see if it reproducible. 
>>Claude don't be so ignorant

C'mon, I'm not "ignorant", I happen to have about 40 years experience in programming ;-)

>>As example I wonder how do you disable retrieving messages on
IMAP and move to another IMAP folder?

Obviously because my folders are not IMAP, but POP.

>>Anyway what File System on your primary partition if its FAT32 there is limit
4GB per file

My file system is NTFS, with a limit of 2TB per file.

>> and because TB using mailbox format this may cause of your
problem.

If it was the case, then it would still be TB responsability to handle the issue nicely.

IMO, the idea of storing all messages with files attached in their original form is a serious flaw in design. It causes inbox to reach an insane size.
It makes compacting and indexing last for hours. 
Attached files should be handled separately, using the OS file handling facilities. 

SMTP is a communication protocole, not a database system.
(In reply to comment #16)
> >>Claude don't be so ignorant
> 
> C'mon, I'm not "ignorant", I happen to have about 40 years experience in
> programming ;-)

I think Nikolay means ignorant in terms of you letting other people help *from their perspective, not yours*.  And, I don't mean this in a nasty way, but if your 40 years experience counted here, then your problem should have already been solved weeks ago by you, not by us _volunteers_. But it hasn't.


> >>As example I wonder how do you disable retrieving messages on
> IMAP and move to another IMAP folder?
> 
> Obviously because my folders are not IMAP, but POP.
 
finally we get to some facts.  So lets repeat part of comment 12...
- can you cause this with version 2.0.0.14, *new profile* + newly created filter?
and comment 13
- is target folder pop?


> >>Anyway what File System on your primary partition if its FAT32 there is limit
> 4GB per file
> 
> My file system is NTFS, with a limit of 2TB per file.
> 
> >> and because TB using mailbox format this may cause of your
> problem.
> 
> If it was the case, then it would still be TB responsability to handle the
> issue nicely.
> 
> IMO, the idea of storing all messages with files attached in their original
> form is a serious flaw in design. It causes inbox to reach an insane size.
> It makes compacting and indexing last for hours. 

please stick to one problem
>>is target folder pop?

This is unbelievable how you can stick on asking irrelevant questions!
I told here several times that the problem occurs even when the retrieving of messages is disabled, the Internet connection being open or closed doesn't matter.
In my previous message, you quoted yourself "my folders are not IMAP, but POP."
When I say "my folders", this include ALL my folders, ANY folder, INCLUDING target folders.

I tell you, the problem is with the way TB is calculating addresses inside a file larger than 2 GB, probabily some overflow or so, when moving a message from one LOCAL file to another LOCAL folder.

>>please stick to one problem

But this is still the same problem : TB does not handle large files properly, period!
I don't think those questions are irrelevant. Obviously not everyone has your problem, so we can't test "everything", but have to narrow it down.

Anyway, the > 2 GB file size sounds like the problem. There are also other bugs about that (though I don't know if any of them is exactly the same.)
Summary: Hangs when message filters are run automatic or manual → Hangs when message filters are run automatic or manual (with > 2 GB folders)
so perhaps this is a dupe...
Mac: bug 294647, bug 343907
Necko: bug 184452 (unrelated?)

from fixed bug 207400 (OS=ALL)...

David Bienvenu  2004-01-18 10:20:28 PST
there are several places in the code that limit mailboxes to 2GB (0x7fffffff) -
one is nsOutputFileStream (used when copying messages to local folders), an
other is nsInputStreamTransport, used when reading a mail message (which is a
limitation of nsISeekableStream)

Comment #11 [reply] David Bienvenu  2004-04-13 07:42:44 PST
fixed by checkin to 180154 - now mail folders can be up to 4GB
Component: MailNews: Filters → MailNews: Backend
QA Contact: filters → backend
>>I don't think those questions are irrelevant.

They are, because I already asked them to myself even before I reported this bug, and I know the answers and already gave them here.

>>Obviously not everyone has your problem, 

No, only those receiving large attachments and keeping more than 2 GB archives.

>>so we can't test "everything", but have to narrow it down.

I DID narrow it down: the problem is with > 2 GB file size. 

>>Anyway, the > 2 GB file size sounds like the problem. 

You see!

>>There are also other bugs about that (though I don't know if any of them is exactly the same.)

Then start digging this avenue, this seems the better approach to me.
>>there are several places in the code that limit mailboxes to 2GB 
>>now mail folders can be up to 4GB


I have no problem with this limitation.
The bug is that TB should handle the overflow gracefully, not just hang.
Claude, one of the reasons I pursued other possibilities is David's comments that I cite in comment 20 about 4GB being the limit (which I knew when I started asking).  So your issue then is 4GB, not 2?  If so then this is a dupe.

Perhaps (we will see) I was too skeptical about your conclusion :)  Glad we can move on.

Related bug 387502. bug 387502 comment 1 David says there should be a warning.  perhaps this warning is not issued when filters are involved??
>>So your issue then is 4GB, not 2?

I'm not categoric about the exact size limit. All what I can say is that 
1. the problem occurs after several month of no trouble operation
2. when the problem occurs, it is permanent, no interruption;
3. the inbox file is pretty big indeed, several Gig for sure;
4. if filters are deactivated, the problem dispears, I can still receive and read messages;
5. if I substancially reduce the size of inbox, I can reactivate filters again, and it works fine.
So my conclusions are:
1. the problem is definitely related to the size in the inbox, but is it 2 or 4 Gb, I can't say;
2. the problem may be not related to increasing the size of inbox (reading new messsages), but rather moving one message from inbox to another folder. 
Actually, moving one message does not remove it from the inbox, but just copies it to the new folder, this is why compacting is necessary sometimes.
So it may just be this copying process wich has a 2Gb limit, not the inbox per se.
Product: Core → MailNews Core
FYI.
Some problems when 2GB<file_size<=4GB on MS Win are already resolved.
  Bug 450359 & Bug 450991 : fixed1.8.1.18 (i.e. fixed by Tb 2.0.0.18).
Claude, can you confirm this now works in version 3, or 2?
Whiteboard: closeme 2010-02-15
Version: unspecified → 1.8 Branch
>>can you confirm this now works in version 3, or 2?

Actually no; since it was established that the problem was due to file size, I took measure to maintain it below a certain limit. So the problem never had a chance to occur agoin.
WFM per comment 28
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.