90452 - threading by subject slow when threads are large

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[wkfx2003-bugzilla]

From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2+)
Gecko/20010711
BuildID:    2001071104

I upgraded to 2001071104 build and
when I click on one of the POP3 account folder, it pops up a dialog showing:
"Do you wish to compact all local and offline folders because the total wasted
space in all accounts exceeds the purge threshhold?"
I answered "Ok" and all the CPU time are grabed by Mozilla.
I waited for quite a long time and Mozilla disappear from Windows
"Alt-tab (task switching)" windows but remains only in task manager
BTW,What is the wasted space and purge thershold?

Reproducible: Always
Steps to Reproduce:
As described in description

Actual Results:  Mozilla Hang

Expected Results:  Mozilla completed the compaction using normal CPU time.

Comment 1

[David :Bienvenu]

reassign to Navin.

Comment 2

[Oliver Klee]

During the hang, can you still use the menus, toolbar etc. in the other Mozilla
windows?

Comment 3

[wkfx2003-bugzilla]

I can't. The Mozilla lose control at all, cannot refresh the screen,
browser windows, mail windows all no response.

Comment 4

[Navin Gupta]

Do you have folders configured for offline use ? 

Comment 5

[wkfx2003-bugzilla]

POP3 Account, has no "offline" options.
IMAP Account has not such problem

Comment 6

[Navin Gupta]

I cannot reproduce this problem. 

Comment 7

[Scott MacGregor]

nsEnterprise+ eMojo bugs should be in the .9.4 milestone so I'm tweaking the
milestone here.  Although this may be a WFM if we can't figure out how to
reproduce this in house.

Comment 8

[Sheela Ravindran]

Reporter,
Are you still seeing this problem on the recent builds?  If so can you tell us
more information about your pop3 account? How many folders you have. Do you have
filters set up to any of your local folders? How many messages do you have etc.  
I will try to set up my test account to fairly match yours and see if I can
reproduce this problem.  
Can you also give us your system information too. thanks.  

Comment 9

[Scott MacGregor]

If we can't find a reproduceable test case for this, then we won't be fixing
this for the branch. So leaving off the nsBranch+ status right now. Leaving the
nsEnterprise nomination alone.

Comment 10

[Jussi-Pekka Mantere]

Triaging nsenterprise-.

Comment 11

[scottputterman]

moving to 0.9.6

Comment 12

[scottputterman]

[e-mail], are you still seeing this?  If so, could you provide Sheela with
the information she asked about?

moving to 0.9.7 since it sounds like we won't be able to fix this until we can
reproduce it.

Comment 13

[scottputterman]

Moving out.  If we can get a reproduceable case then we can move back in.

Comment 14

[Philippe Mignard]

I confirm this bug on 2002060308
Note this is a huge compacting as I have deleted a lot of messages

Mozilla hangs and I have no control. It appears both when asking for compacting
by the menu and when Mozilla proposes "compact folders" after launch.

Note: I use Windows XP.

Comment 15

[Aleksander Adamowski]

I see this on Linux, however, Mozilla stays responsive although sluggish
(possibly because Linux kernel has better resource management?).

When using a file manager I enter the disk directory containing the compacted
folder's file, I can see that there's a temporary file nstmp there which grows
slowly:


[olo@tuxia ~/.mozilla/winda/4oali5gr.slt/Mail/Local Folders-1/Admin.sbd 18:39:33]$ll
razem 92372
-rw-r--r--    1 olo      olo      52886332 lut 20 17:10 NMAIL Message Notifications
-rw-r--r--    1 olo      olo       3573388 lut 20 18:20 NMAIL Message
Notifications.msf
-rw-r--r--    1 olo      olo      38123978 lut 20 18:39 nstmp
-rw-r--r--    1 olo      olo             0 lut 20 17:47 nstmp.msf
[olo@tuxia ~/.mozilla/winda/4oali5gr.slt/Mail/Local Folders-1/Admin.sbd 18:39:34]$ll
razem 92376
-rw-r--r--    1 olo      olo      52886332 lut 20 17:10 NMAIL Message Notifications
-rw-r--r--    1 olo      olo       3573388 lut 20 18:20 NMAIL Message
Notifications.msf
-rw-r--r--    1 olo      olo      38127466 lut 20 18:39 nstmp
-rw-r--r--    1 olo      olo             0 lut 20 17:47 nstmp.msf


eventually, the compacting finishes (but after a veeeery long time), the
temporary file seems to be renamed to replace the old folder file.

So to summarize, Mozilla actually doesn't hang, it's only that compacting is
extremely slow (rebuilding speed in my case was roughly 6 KB/sec on a 50 MB
folder! it took hours!).

Dureing the compacting Mozilla users all available CPU power, so it looks like
the compacting algorithm is some kind of bogosort ;)

Comment 16

[Aleksander Adamowski]

BTW this is the same folder for which building summary takes long time too, see
bug 172786.

It seems strange, because when the folder is smaller (below 20 MB) all those
operations are relatively fast, but when the folder grows, the execution time of
compacting and building summary file increases exponentially!

Comment 17

[Aleksander Adamowski]

See this URL for a testcase (tar-gzipped folder that gives me all those
headaches, 3.8 MB in size when compressed):
http://olo.office.altkom.com.pl/domowa/qa/mozilla/2003_01_24_Large_Mail_Folder_loading/

Comment 18

[Aleksander Adamowski]

BTW I've placed more testcases, all are compressed archives containing versions
of the big folder I have trouble with.

The latest ones will be compressed with bzip2 instead of gzip - now they only
take 1.9MB.

Comment 19

[wkfx2003-bugzilla]

I haven't specificly test this bug for recent build but I did remember 1 or 2
months ago, I need to move some mail to a new folder so that I compact the
original folder

Comment 20

[(not reading, please use [e-mail] instead)]

mass re-assign.

Comment 21

[David :Bienvenu]

Have you tried this with a 1.5 Mozilla or Thunderbird build? Compact has been
sped up a great deal.

Comment 22

[David :Bienvenu]

actually, this folder seems to have a very large number of messages in the same
thread. That makes reparsing and compacting the folder very slow.

Comment 23

[David :Bienvenu]

When you try 1.5b/final or thunderbird, you should set the following pref:

user_pref("mail.thread_without_re", false);

either by using about:config, or shutting down mozilla and editing prefs.js by
hand. This will cause messages with the same subject but no Re: or references to
be put in separate threads, but it will also speed up reparsing and compact
enormously.

Comment 24

[David :Bienvenu]

*** Bug 191890 has been marked as a duplicate of this bug. ***

Comment 25

[David :Bienvenu]

changing summary - it's not hang; it's just very slow...

Comment 26

[David :Bienvenu]

Attachment: proposed fix

This is an ugly but simple fix for a difficult problem. Basically, when we add
a message to a thread, we have to run through the thread to see if the new
message is a parent of an existing message in the thread, and adjust things
accordingly. If you thread by subject, and you have a large folder with
messages w/ all the same subject, this code can take a really long time. So the
pragmatic thing is to say that for threads with more than 1000 messages, it's
simply not worth dealing with the case where the parent comes in after the
child. Threads with more than 1000 messages are pretty unwieldy anyway.

Comment 27

[Scott MacGregor]

Comment on attachment 190755 [details] [diff] [review]
proposed fix

how about a comment describing why we are skipping threads with more than 1,000
children

Comment 28

[David :Bienvenu]

Comment on attachment 190755 [details] [diff] [review]
proposed fix

I've basically added the bugzilla comment to the code.

Comment 29

[Benjamin Smedberg]

Comment on attachment 190755 [details] [diff] [review]
proposed fix

I assume that since this is a -w patch the indentation will be fixed on
checkin. A comment referencing this bug would be good also.

Comment 30

[David :Bienvenu]

added a ref to this bug in comment. The indentation was fine - I just had some
whitespace cleanups in that file that I didn't want to complicate the diff with.