541001 - TB 3.0 hangs for long periods, accompanied by high memory usage (header download of all articles of huge newsgroup consumes too large memory and causes long hang)

Status: RESOLVED DUPLICATE of bug 185634

(Thunderbird :: General, defect)

Description

[Joe Sewell]

User-Agent:       Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7
Build Identifier: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

Periodically, with no apparent consistency in user action, Thunderbird 3.0 will completely stall, not responding to user input at all. Sometimes it even happens when a menu (from the menu bar or a pop-up menu) pops up, causing that menu to remain floating above other applications' windows as well. CPU is usually completely busy, and memory consumption tends to go up to maximum RAM, then dropping back down; this RAM "breathing" may repeat 1 or more times before Thunderbird responds to user input. If other programs, especially Firefox, are consuming a significant portion of available RAM (even on a 1GB machine), Thunderbird may never return to a responsive mode, requiring a Force Quit to free it up.

Profile in use was created "fresh" when installing TB 3.

Reproducible: Always

Steps to Reproduce:
No steps available to reproduce; happens even when otherwise idle (from a user perspective).

Comment 1

[Joe Sewell]

Attachment: Output of Activity Monitor's Sample command on stall when trying to close tab

Comment 2

[Joe Sewell]

Attachment: Output of Activity Monitor's Sample command on hang bringing TB to front

Comment 3

[Wayne Mery (:wsmwk)]

need to get more specific ...
how much memory is *thunderbird* using?
is it a solid hang? or does it go away after 15-30 seconds? 
how often is "period"?
does this happen started in safe mode?  
 http://kb.mozillazine.org/Safe_mode

Comment 4

[Joe Sewell]

(In reply to comment #3)
> need to get more specific ...
> how much memory is *thunderbird* using?
> is it a solid hang? or does it go away after 15-30 seconds? 
> how often is "period"?
> does this happen started in safe mode?  
>  http://kb.mozillazine.org/Safe_mode

Excuse me. Sheesh.

I just tried it again in safe mode. To answer your last question first, it does indeed happen in safe mode, in this particular case about 10 minutes into running. Thunderbird effectively hung for 13 minutes, although there were 2 quick periods where it actually did ping its event queue; it appeared, though, that it didn't do much during those times before freezing up again. After about 13-15 minutes it did allow me to regain control, though from past experience it would happen again within 5-10 minutes.

RAM usage for Thunderbird alone peaked at 740 MB, with virtual memory at around 880 MB at that same time. Once Thunderbird came out of the hang for more than a second or two, memory usage did *not* drop back down, although during the freeze-up it did drop from 735 MB down to 395 MB, then rose back up again.

I will attach another sample obtained during this safe-mode hang.

I do not understand the question "how often is 'period'?" I assume you are referring to my initial "Periodically" in the bug report. I did not time it precisely, but it appears to start at roughly 10 minutes into usage. Subsequent periods depend greatly on Thunderbird usage as well as other processes taking up system resources. Likewise the "hang" period lasts for several minutes each time with only Thunderbird and normal background processes running; if I have another heavy memory consumer running, such as Firefox, the hang will continue indefinitely as it thrashes in a page-faulting frenzy.

If this is inadequate, then *you* need to get more specific about "specific."

Comment 5

[Joe Sewell]

Attachment: Safe Mode Sample output

Output of Activity Monitor's Sample command about 8-10 minutes into the latest hang while in Safe mode, as requested.

Comment 6

[Wayne Mery (:wsmwk)]

thanks for the info. I failed to complete the word "periodic", but you've answered the question. I wasn't complaining, just asking for detail to get an accurate, full picture needed to narrow the issue. to some people, a freeze is anything over 5 seconds :)

had you done any type of searches (quick, search all, etc)?

do you see this problem if gloda indexing is disabled? (Tools|Options|Advanced|General|Enable Global Search and Indexer) 

if seen with gloda off, it might be  Bug 526568 -  Consumes too much memory (nearly 2 GB)

The current list of Mac perf and hang bugs is https://bugzilla.mozilla.org/buglist.cgi?bugidtype=include;bug_id=388347%2C442674%2C522986%2C456993%2C439897%2C479712%2C531455%2C294647%2C463970%2C482811%2C531428%2C526568%2C494849%2C538162%2C506473%2C533441%2C470001%2C373040%2C531549%2C462013%2C536155%2C538610%2C534403%2C508263%2C541001;query_format=advanced

Comment 7

[Wayne Mery (:wsmwk)]

eek, I forgot to ask, when it's in high memory situation, what folders are reported open by Mac's lsof?
 http://www.macosxhints.com/article.php?story=20040121001144687

lsof | grep ".msf"

Comment 8

[WADA:World Anti-bad-Duping Agency]

Same problem as bug 540952?

Comment 9

[Joe Sewell]

(In reply to comment #8)
> Same problem as bug 540952?

I don't believe so. I turned off the offline sync, as mentioned in the cited bug, plus I ensured it was disabled in the Account Settings. This problem does not happen immediately, either, as it did with the cited bug.

I'm currently researching the other two questions.

Comment 10

[Joe Sewell]

(In reply to comment #6)
> thanks for the info. I failed to complete the word "periodic", but you've
> answered the question. I wasn't complaining, just asking for detail to get an
> accurate, full picture needed to narrow the issue. to some people, a freeze is
> anything over 5 seconds :)

Good point.

> had you done any type of searches (quick, search all, etc)?

Nope.

> do you see this problem if gloda indexing is disabled?
> (Tools|Options|Advanced|General|Enable Global Search and Indexer) 

It did hang (it was in the background, so I don't know the actual amount of time it hung), but it came back significantly more quickly with indexing off! (In fact, I wondered if that was part of the issue, but for some silly reason I couldn't see the option right in front of me.) The large memory consumption was **not** seen!

Actually, I just turned gloda back on, and TB immediately seized up! Memory consumption is increasing, albeit not as quickly as usual. ... Wait, after a few minutes, it's back up again. Sounds like indexing is, at best, the trigger.

> if seen with gloda off, it might be  Bug 526568 -  Consumes too much memory
> (nearly 2 GB)

In this case it seems more like gloda is triggering the memory consumption.

So, since it's locked up, the output of the lsof | grep ".msf" command is:

thunderbi 239 joesewell   48u    VREG      14,15     213101   787990 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/ImapMail/imap.spamcop.net/INBOX.msf
thunderbi 239 joesewell   50u    VREG      14,15      24867   788019 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/ImapMail/imap.spamcop.net/INBOX.sbd/Held Mail.msf
thunderbi 239 joesewell   61u    VREG      14,15       1723   910535 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart mailboxes/Archives.msf
thunderbi 239 joesewell   62u    VREG      14,15       2101   808022 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart mailboxes/Inbox.msf
thunderbi 239 joesewell   63u    VREG      14,15       1421   808025 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart mailboxes/Sent.msf
thunderbi 239 joesewell   64u    VREG      14,15       1916   787977 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart mailboxes/Trash.msf
thunderbi 239 joesewell   65u    VREG      14,15       1424  1379564 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart mailboxes/Drafts.msf
thunderbi 239 joesewell   67u    VREG      14,15    3574041  1384149 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/News/isp5.newshosting.com/rec.arts.comics.dc.universe.msf
thunderbi 239 joesewell   71u    VREG      14,15     423199   788011 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/ImapMail/imap.spamcop.net/INBOX.sbd/Trash.msf
thunderbi 239 joesewell   72u    VREG      14,15    2826003  1384151 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/News/isp5.newshosting.com/rec.arts.comics.marvel.universe.msf
thunderbi 239 joesewell   73u    VREG      14,15   20705840   795465 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/News/isp5.newshosting.com/alt.binaries.mac.osx.apps.msf
thunderbi 239 joesewell   74u    VREG      14,15       2690   801020 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/Feeds/Mozilla Add-ons.msf
thunderbi 239 joesewell   75u    VREG      14,15  256512548   795455 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/News/isp5.newshosting.com/alt.binaries.mac.applications.msf
thunderbi 239 joesewell   91u    VREG      14,15       2085   802042 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/Feeds/Softpedia.msf
thunderbi 239 joesewell  106u    VREG      14,15      54065  1100094 /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/Feeds/Trash.msf

(Please pardon any spurious line breaks the copy & paste may have introduced.)

Comment 11

[Joe Sewell]

(In reply to comment #10)
> So, since it's locked up, the output of the lsof | grep ".msf" command is:
> 
> thunderbi 239 joesewell   48u    VREG      14,15     213101   787990
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/ImapMail/imap.spamcop.net/INBOX.msf
> thunderbi 239 joesewell   50u    VREG      14,15      24867   788019
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/ImapMail/imap.spamcop.net/INBOX.sbd/Held Mail.msf
> thunderbi 239 joesewell   61u    VREG      14,15       1723   910535
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart
> mailboxes/Archives.msf
> thunderbi 239 joesewell   62u    VREG      14,15       2101   808022
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart
> mailboxes/Inbox.msf
> thunderbi 239 joesewell   63u    VREG      14,15       1421   808025
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart
> mailboxes/Sent.msf
> thunderbi 239 joesewell   64u    VREG      14,15       1916   787977
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart
> mailboxes/Trash.msf
> thunderbi 239 joesewell   65u    VREG      14,15       1424  1379564
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default User/Mail/smart
> mailboxes/Drafts.msf
> thunderbi 239 joesewell   67u    VREG      14,15    3574041  1384149
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/rec.arts.comics.dc.universe.msf
> thunderbi 239 joesewell   71u    VREG      14,15     423199   788011
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/ImapMail/imap.spamcop.net/INBOX.sbd/Trash.msf
> thunderbi 239 joesewell   72u    VREG      14,15    2826003  1384151
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/rec.arts.comics.marvel.universe.msf
> thunderbi 239 joesewell   73u    VREG      14,15   20705840   795465
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/alt.binaries.mac.osx.apps.msf
> thunderbi 239 joesewell   74u    VREG      14,15       2690   801020
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/Mail/Feeds/Mozilla Add-ons.msf
> thunderbi 239 joesewell   75u    VREG      14,15  256512548   795455
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/alt.binaries.mac.applications.msf
> thunderbi 239 joesewell   91u    VREG      14,15       2085   802042
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/Mail/Feeds/Softpedia.msf
> thunderbi 239 joesewell  106u    VREG      14,15      54065  1100094
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/Mail/Feeds/Trash.msf
> 
> (Please pardon any spurious line breaks the copy & paste may have introduced.)

Since some of these paths include "smart mailboxes," I should note that I have not done anything explicit with smart folders; I have neither disabled them nor used them. I know there are some out there that TB did for me, but I haven't touched them.

Comment 12

[WADA:World Anti-bad-Duping Agency]

> thunderbi 239 joesewell   73u    VREG      14,15   20705840   795465
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/alt.binaries.mac.osx.apps.msf

> thunderbi 239 joesewell   75u    VREG      14,15  256512548   795455
> /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> User/News/isp5.newshosting.com/alt.binaries.mac.applications.msf

alt.binaries.mac.applications is probably huge newsgrop.
What will happen by "unsubscribe of alt.binaries.mac.applications"?

Comment 13

[Joe Sewell]

(In reply to comment #12)
> > thunderbi 239 joesewell   73u    VREG      14,15   20705840   795465
> > /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> > User/News/isp5.newshosting.com/alt.binaries.mac.osx.apps.msf
> 
> > thunderbi 239 joesewell   75u    VREG      14,15  256512548   795455
> > /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> > User/News/isp5.newshosting.com/alt.binaries.mac.applications.msf
> 
> alt.binaries.mac.applications is probably huge newsgrop.
> What will happen by "unsubscribe of alt.binaries.mac.applications"?

Yikes! You're right. I didn't expect TB to download the entire contents of messages (I'm used to the NewsWatcher series of programs). Okie-dokie, they're gone!

Thanks for the heads-up. I'll have to dig through the mozilla-messaging site and mozillazine to see how to get TB to avoid downloading the entire messages behind my back. Then again, if that's just the header information, maybe it's time to use an OS X version of MT-NewsWatcher.

Comment 14

[Joe Sewell]

(In reply to comment #12)
> > thunderbi 239 joesewell   73u    VREG      14,15   20705840   795465
> > /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> > User/News/isp5.newshosting.com/alt.binaries.mac.osx.apps.msf
> 
> > thunderbi 239 joesewell   75u    VREG      14,15  256512548   795455
> > /Users/joesewell/Library/Thunderbird/Profiles/9ck4odcf.Default
> > User/News/isp5.newshosting.com/alt.binaries.mac.applications.msf
> 
> alt.binaries.mac.applications is probably huge newsgrop.
> What will happen by "unsubscribe of alt.binaries.mac.applications"?

Oh, sorry, what happened is that TB "breathed" again during the unsubscribe process, but recovered significantly more quickly than before. Considering the size of these files, this was an acceptable stall time. I saw no other change, now that I've turned off Gloda.

Comment 15

[Wayne Mery (:wsmwk)]

so alt.binaries helped kill performance? 

perhaps Bug 539194?  -  MailNews reported as high CPU being caused by having news downloaded without bounds, i.e. no purging

Comment 16

[Wayne Mery (:wsmwk)]

Gloda doesn't index newsgroups. So perhaps the combination of normal gloda indexing and high newsgroup activity made the impact of both activities worse than they would be alone?  What do you think?

Comment 17

[Joe Sewell]

I just confirmed that, indeed, removing the alt.binaries.* groups did indeed solve the problem; I turned Gloda back on without injury.

I did not have the newsgroup accounts set to read new messages at startup, though I've noticed TB has a knack for doing that when it feels like it anyhow (probably when opening the account's top "folder").

So what we have is that Gloda played a part in it, and so did the large newsgroups. Whether it's a match with any existing bug or not, though, I'm not too sure.

Comment 18

[Joe Sewell]

Oh, forgot to mention, that last check was with TB 3.0.1. BuildID = Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1

Comment 19

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #16)
> Gloda doesn't index newsgroups. So perhaps the combination of normal gloda
> indexing and high newsgroup activity made the impact of both activities worse
> than they would be alone?

If similar issue to Bug 540214(Tb somehow deletes .msf of IMAP by himself) or bug 541917(Tb somehow re-downloads all mail of an IMAP folder) occurs on newgroup, re-download of all news articles may happen. If huge newsgroup, it takes very long and it may cause large virtual memory and high CPU.

Joe Sewell, can you trace file I/O for alt.binaries.mac.applications and alt.binaries.mac.applications.msf by tool or monitor?

Comment 20

[Joe Sewell]

(In reply to comment #19)
> (In reply to comment #16)
> > Gloda doesn't index newsgroups. So perhaps the combination of normal gloda
> > indexing and high newsgroup activity made the impact of both activities worse
> > than they would be alone?
> 
> If similar issue to Bug 540214(Tb somehow deletes .msf of IMAP by himself) or
> bug 541917(Tb somehow re-downloads all mail of an IMAP folder) occurs on
> newgroup, re-download of all news articles may happen. If huge newsgroup, it
> takes very long and it may cause large virtual memory and high CPU.
> 
> Joe Sewell, can you trace file I/O for alt.binaries.mac.applications and
> alt.binaries.mac.applications.msf by tool or monitor?

Not completely, as I had already removed the binary newsgroups. However, my attempt to add them back produced some interesting data.

Specifically adding alt.binaries.mac.applications, with 388997 headers to download, produced the same action as I originally reported. This happened with Gloda turned on or off.

A ktrace -t i of the process showed essentially *no* I/O during the lockup period. (I missed the file's creation and the initial I/O activity, sorry.) Activity Monitor confirmed that there was essentially no file I/O going on system-wide while TB was allocating tons of memory again.

Killing TB and relaunching it kept everything stable, until I opened the account for the news server in question. At that point TB stalled for about 10 seconds, then checked its event queue long enough for me to Unsubscribe from the group. Another 10-20 second churn, with only moderate memory consumption, and everything was freed up and back to normal.

I have that newsgroup account set up to download everything, not just the most recent n days' worth of items.

It begins to sound like something triggers TB to do a full download of the headers, perhaps as a periodic update. I do have that server set up to check for new messages every 10 minutes, but not to check at startup.

Comment 21

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #20)
> Specifically adding alt.binaries.mac.applications, with 388997 headers to
> download, produced the same action as I originally reported.
> This happened with Gloda turned on or off.
> A ktrace -t i of the process showed essentially *no* I/O during the lockup period.

It sound build-index process for very many deep/long thread.
Threading of deep/long thread has performance problems;
  (a) (re-)build index takes long
  (b) threaded view display takes long
  (c) delete of mails takes very long (bug 452221)
Can you check next cases?
 1. Subscribe, download     10 articles only
 2. Subscribe, download    100 articles only
 3. Subscribe, download  1,000 articles only
 4. Subscribe, download 10,000 articles only
Roughly O(num_of_articles**2)? Or O(num_of_articles)?

Comment 22

[Joe Sewell]

(In reply to comment #21)
> (In reply to comment #20)
> > Specifically adding alt.binaries.mac.applications, with 388997 headers to
> > download, produced the same action as I originally reported.
> > This happened with Gloda turned on or off.
> > A ktrace -t i of the process showed essentially *no* I/O during the lockup period.
> 
> It sound build-index process for very many deep/long thread.
> Threading of deep/long thread has performance problems;
>   (a) (re-)build index takes long
>   (b) threaded view display takes long
>   (c) delete of mails takes very long (bug 452221)
> Can you check next cases?
>  1. Subscribe, download     10 articles only
>  2. Subscribe, download    100 articles only
>  3. Subscribe, download  1,000 articles only
>  4. Subscribe, download 10,000 articles only
> Roughly O(num_of_articles**2)? Or O(num_of_articles)?

I can confirm that the threaded display is not part of this issue, as I am not displaying the items. Item (a) sounds like a possibility, though.

1. 10 articles takes a second, essentially no memory increase
2. 100 articles about 5 seconds, some memory increase to inactive memory
3. 1000 articles about 10 seconds, roughly 4MB increase
4. 10000 articles about 3 minutes, 105MB memory increase
5. 1025 articles about 15 seconds, 1MB increase
6. 4097 articles, about 30 seconds, no memory increase
7. 20000 articles just under 4 minutes, around 120MB increase

(I wanted to try a few powers of 2, just in case. I also wanted to try some linear tests, just to see what happened.)

Methodology:

1. Set account to ask if downloading more than 500 messages.
2. With newsgroup account collapsed, right-click and Subscribe...
3. Search for mac.app, subscribe to alt.binaries.mac.applications
4. Expand newsgroup account in All Folders pane, exposing new subscription.
5. When prompted, enter desired # of articles.
6. Do my best to examine real memory & time after dismissing prompt, and again when TB responds to events.

I did not quit & restart TB through any of this. What I found curious is that the real memory consumed did not drop, although how much may have been reused is unknown.

Comment 23

[Wayne Mery (:wsmwk)]

huge newsgroup problem can be related to Bug 185634 -  Mozilla News consumes unbearable amounts of memory in big newsgroups.

Joel, perhaps you answered this already, but did you see this problem in version 2?  (or are you a new user)

Comment 24

[Joe Sewell]

(In reply to comment #23)
> huge newsgroup problem can be related to Bug 185634 -  Mozilla News consumes
> unbearable amounts of memory in big newsgroups.
> Joel, perhaps you answered this already, but did you see this problem in
> version 2?  (or are you a new user)

No, I did not see it in 2.0x.

Comment 25

[WADA:World Anti-bad-Duping Agency]

Large memory consumption probably occurs in threading process.
1. news server=news.mozilla.org, newsgroup=mozilla.support.firefox
   number of articles=111069
2. Start Tb 3.0.1 on Win-XP, virtual memory size of Tb is around 100MB.
3. subscribe, download all 111069 headers
   => Virtual memory size increases till 180MB according to download progress
4. After "Downloading <around 111000>" of status bar,
   (end of download, 111069 is not displayed due to bug 388527)
   virtual memory size start to increases from 180 MB to 400MB.
   While increasing, virtual memory size sometimes drops, and increases again. 
   CPU utilization was 50%(Dual Core, 100% if single CPU).
   I think threading is executed in this step.
5. CPU utilization returns to 0%, virtaul memory size drops to 180MB.

(In reply to comment #24)
> No, I did not see it in 2.0x.

Did you do "header download of all articles" just after subscribe of the huge alt.binaries.mac.applications when you were using Tb 2?
"Adding some articles to already threaded long thread" doesn't consume large memory and doesn't take long, but "threading of all articles of long thread at once" consumes large memory and takes long.
I guess "recursive call" like method is used for threading, and/or inefficient search(not binary tree search) is used, and/or inefficient sort(bubble sort like) is used.

Note:
Next problems are easily be reproduced by long single thread(all mails has same subject) in a local mail folder.
(a) Rebuild-index takes long. I think this bug corresponds to (a).
(b) Change to threaded view takes long.
(c) Delete many mails from long thread takes very long and causes long freeze.
    Bug 452221. I guess;
    If N mails is deleted from thread of length=N, in worst case,
    O(factorial of N) times of parent/child pointer update happens.

Comment 26

[Joe Sewell]

(In reply to comment #25)
> Large memory consumption probably occurs in threading process.
> 1. news server=news.mozilla.org, newsgroup=mozilla.support.firefox
>    number of articles=111069
> 2. Start Tb 3.0.1 on Win-XP, virtual memory size of Tb is around 100MB.
> 3. subscribe, download all 111069 headers
>    => Virtual memory size increases till 180MB according to download progress
> 4. After "Downloading <around 111000>" of status bar,
>    (end of download, 111069 is not displayed due to bug 388527)
>    virtual memory size start to increases from 180 MB to 400MB.
>    While increasing, virtual memory size sometimes drops, and increases again. 
>    CPU utilization was 50%(Dual Core, 100% if single CPU).
>    I think threading is executed in this step.
> 5. CPU utilization returns to 0%, virtaul memory size drops to 180MB.

This sound similar to what I experienced, except I'm on an old PowerMac G4, single CPU.

> (In reply to comment #24)
> > No, I did not see it in 2.0x.
> 
> Did you do "header download of all articles" just after subscribe of the huge
> alt.binaries.mac.applications when you were using Tb 2?
> "Adding some articles to already threaded long thread" doesn't consume large
> memory and doesn't take long, but "threading of all articles of long thread at
> once" consumes large memory and takes long.

I believe I did have TB2 download the headers of all articles. I know I did not limit it by date.

I do not believe threading is an issue here, since alt.binaries.mac.applications rarely has threads. Instead, each piece of a segmented upload shows up as a new article. Cues in the subject of each article determine the order each piece should be added to the final file.

> I guess "recursive call" like method is used for threading, and/or inefficient
> search(not binary tree search) is used, and/or inefficient sort(bubble sort
> like) is used.
> 
> Note:
> Next problems are easily be reproduced by long single thread(all mails has same
> subject) in a local mail folder.
> (a) Rebuild-index takes long. I think this bug corresponds to (a).
> (b) Change to threaded view takes long.
> (c) Delete many mails from long thread takes very long and causes long freeze.
>     Bug 452221. I guess;
>     If N mails is deleted from thread of length=N, in worst case,
>     O(factorial of N) times of parent/child pointer update happens.

Let me remind you that disabling global search did resolve the problem.

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #26)
> Let me remind you that disabling global search did resolve the problem.

I was talking about remained prolem after you disabled "Global Search and Indexer".
For problems(other than huge newsgroup related) before disabling Indexer.
Tb looks to index already downloaded IMAP offline-store data, even after change to offline use=OFF. (auto-sync and Indexer is independent. auto-sync'ed data is not killed by change to auto-sync=off)
(1) Disable "Global Search and Indexer"
(2) [Gmail]/All Mail : offline use=ON => auto-sync'ed to offline store 
(3) [Gmail]/All Mail : offline use=OFF
(4) Enable "Global Search and Indexer" and restart Tb3
    => Indexer indexed all mails of [Gmail]/All Mail
I think it's the reason why your main problem remained even after auto-sync=off until Indxer=off.

By the way, Tb 3.0 has at least two big issues, Bug 539389, Bug 540214, which will produce large memory use and/or performance issue, in addition to Global Indexer related issues. If huge mail folder/newsgroup or very many mail folders/newsgroups are accessed, problems may easily be exposed.

Comment 28

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #26)
> I believe I did have TB2 download the headers of all articles. I know I did not limit it by date.

(For remained problem after you disabled "Global Search and Indexer")
Difference between Tb3 and Tb2 may be caused by bug 548468, because you subscribe many newsgroups and because re-download of all headers of all newsgroup might happened (it looks to have happened at least on alt.binaries.mac.applications in your environment). 
> Bug 548468 3.0 fetches all newsgroup headers for ALL groups asynchronously; TB 2 fetched as needed (visited, read)

Comment 29

[Wayne Mery (:wsmwk)]

what antivirus sofwtare or firewall do you run?
is problem better with v3.1beta2? 
 http://www.mozillamessaging.com/en-US/thunderbird/early_releases/downloads/

Comment 30

[Joe Sewell]

3.1b2 is better. While it does consume extra memory, it's significantly better behaved, and does not interfere significantly with other processes. Gloda is not an issue in 3.1b2. There is still a delay as it's processing headers, but not an unexpected delay.

I use ClamAV for anti-virus protection (not set to auto-scan email). Local network is firewalled at the router.

Comment 31

[Wayne Mery (:wsmwk)]

Joe, your opinion on whcih bug we dup this to? bug 185634 or bug 548468?

Comment 33

[[:Aureliano Buendía]]

Cleanup *dupeme* whiteboard flag from bugs that are marked as Resolved
Duplicate!