561272 - UI unresponsive for several minutes with high CPU while .msf files are thrashed with reads and lseeks. global indexing is disabled (linux?)

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Bill McGonigle (not currently reading bugmail; please contact directly)]

At seemingly random times, I'll have Thunderbird become unresponsive in the GUI while it goes and thrashes on some .msf files - usually ones for large mailboxes.  I can see via strace that thunderbird is quite busy and can see it's read'ing and lseek'ing the .msf's the whole time, but if there's housekeeping work to be done the UI shouldn't be hung for it.  Perhaps demorkifying the .msf's will solve this, but AFAIK that's not happening so some other way of not locking the UI is needed.

I'm currently on Thunderbird 3.0.4, but it's been a problem for a while.  This is on a quad-core 2.8GHz box running off a raid volume - normally a peppy machine, but I can hear my fans kick on as the CPU is pegged during this period.

I'm adding some stack info - if it'd be worthwhile getting more detail I can take the time to redact the full backtrace.  The actual top of the stack varies among different mork calls depending on when I hit the process with gdb.

#0  morkArray::CutSlot 
#1  0x00007f5498fa74fc in morkTable::CutRow
#2  0x00007f5498f95afb in morkBuilder::OnNewRow 
#3  0x00007f5498f9d9eb in morkParser::ReadRow
#4  0x00007f5498f9dc31 in morkParser::ReadTable
#5  0x00007f5498f9dd04 in morkParser::ReadContent
#6  0x00007f5498f9deb3 in morkParser::ReadGroup 
#7  0x00007f5498f9df30 in morkParser::ReadAt
#8  0x00007f5498f9ddc2 in morkParser::OnPortState
#9  0x00007f5498f9dfe0 in morkParser::ParseLoop
#10 0x00007f5498f9e0b7 in morkParser::ParseMore 
#11 0x00007f5498fa878f in morkThumb::DoMore_OpenFileStore
#12 0x00007f5498fa88e6 in morkThumb::DoMore 
#13 0x00007f5498fa89cc in morkThumb::DoMore
#14 0x00007f5499366d16 in nsMsgDatabase::OpenMDB 
#15 0x00007f5499367079 in nsMsgDatabase::Open 
#16 0x00007f5499367cc7 in nsMsgDBService::OpenFolderDB 
#17 0x00007f5499399681 in nsImapMailFolder::GetDatabase 
#18 0x00007f549924e3b3 in nsMsgDBFolder::GetMsgDatabase 
#19 0x00007f54993d165a in nsAutoSyncState::ProcessExistingHeaders
#20 0x00007f54993d45db in nsAutoSyncManager::TimerCallback

Comment 1

[Ludovic Hirlimann [:Usul]]

do you compact your folders once in a while ?

Comment 2

[Wayne Mery (:wsmwk)]

thanks for the information. but no response, so => incomplete.

if you see this problem when using v3.1 which will be out soon, please attach a *hang* stacktrace and reopen this bug - https://developer.mozilla.org/En/How_to_get_a_stacktrace_for_a_bug_report

Comment 3

[Wayne Mery (:wsmwk)]

Bill, you could check if it's gloda related activity by disabling gloda (that's the only major new consumer/user of folders in v3 afaik).  Tools>options>advanced>general>uncheck global search and index

if you do so, please post results.

Comment 4

[Miroslav Suchy]

I hit the same issues (Thunderbird 24.6.0) and I can provide requested data.

Yes I compact folders once in while. But even compacted my folders are quite large. My Local Folders is 6.6 GB big.

I have 'global search and index' switched off (for years actually).

Comment 5

[Miroslav Suchy]

@wsmwk The link you provide does not show how to generate stacktrace of *hang* system. Where the hang is in order of few seconds, so the thunderbird does not crash.

If you need more information, then I'm happy to give it you.

Comment 6

[Wayne Mery (:wsmwk)]

Bill, do you still see this problem?

(In reply to Miroslav Suchy from comment #5)
> @wsmwk The link you provide does not show how to generate stacktrace of
> *hang* system. Where the hang is in order of few seconds, so the thunderbird
> does not crash.
> 
> If you need more information, then I'm happy to give it you.

Please see https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems#Diagnosis_Steps and file a new bug report if needed

Comment 7

[Bill McGonigle (not currently reading bugmail; please contact directly)]

Attachment: Backtrace of thread 1 when Thunderbird is unresponsive

Attaching a backtrace of the main thread during a long hang.  For others who might need to do the same, what worked for me to generate the backtrace was:
---
screen -S thunderbird
thunderbird -g
handle SIG33 noprint nostop
run

and then ctrl-c to interrupt when stuck and then:

thread apply all bt full

hitting return through it, and then for screen:

ctrl-a : hardcopy -h mythunderbirdbacktrace
--

I haven't examined the code yet, but things that I'm noticing:

1) It's deep in mork
2) the folder it's busy with is not set for 'enabled' or 'inherit' in its properties for the indexer (looks like the synchronizer running?)
3) moving mail out of the folder does not trim up the .msf file.  The .msf for this folder was 410MB.  It had 6 mail messages in (years ago it had hundreds of thousands).  I did a 'repair' and it went down to 7KB.
4) another giant folder had a 300MB .msf.  That one did have lots of mail messages.  A 'repair' brought it down to 32MB.  
5) The IMAP folder (server-side) hasn't had a new message since April.  If the synchronizer is running frequently on this local folder, it's not using the available information.
6) Based on the amount of thrash, it seems to be looking at every record in the .msf.  I thought mork had an index, but maybe I'm mistaken.
7) looking at my backups, neither the file sizes nor the mtime on the local .msf files are changing over time on the old folders.
8) I'm unable to move 300,000 messages atomically from one folder to another on the client without exhausing 16GB of RAM and being reaped by the OOM killer.  `find` on the server and then 'repair' on the client is usable.
98) sqlite
99) running long operations s in the UI thread


#0  0x00007ffff55be9eb in morkAtomBodyMap::Equal (this=<optimized out>, ev=0x7fffae5eeb00, inKeyA=0x7fff4d2da350,
#1  0x00007ffff55c379d in morkMap::find (this=this@entry=0x7ffef3bfc730, ev=ev@entry=0x7fffae5eeb00,
#2  0x00007ffff55c3fa8 in morkMap::Get (this=this@entry=0x7ffef3bfc730, ev=ev@entry=0x7fffae5eeb00,
#3  0x00007ffff55bec63 in morkAtomBodyMap::GetAtom (this=this@entry=0x7ffef3bfc730, ev=ev@entry=0x7fffae5eeb00,
#4  0x00007ffff55cb975 in morkStore::YarnToAtom (this=this@entry=0x7ffeb64ef000, ev=ev@entry=0x7fffae5eeb00,
#5  0x00007ffff55bfdbb in morkBuilder::OnValue (this=0x7fff0ccf0800, ev=0x7fffae5eeb00, inSpan=..., inBuf=...)
#6  0x00007ffff55c62dc in morkParser::ReadCell (this=0x7fff0ccf0800, ev=0x7fffae5eeb00)
#7  0x00007ffff55c676c in morkParser::ReadRow (this=this@entry=0x7fff0ccf0800, ev=ev@entry=0x7fffae5eeb00, c=40, c@entry=91)
#8  0x00007ffff55c6cc8 in morkParser::ReadContent (this=this@entry=0x7fff0ccf0800, ev=ev@entry=0x7fffae5eeb00,
#9  0x00007ffff55c6f04 in morkParser::OnPortState (this=0x7fff0ccf0800, ev=0x7fffae5eeb00)
#10 0x00007ffff55c707c in morkParser::ParseMore (this=0x7fff0ccf0800, ev=0x7fffae5eeb00, outPos=outPos@entry=0x7fffffffb95c,
#11 0x00007ffff55d0939 in morkThumb::DoMore_OpenFileStore (this=this@entry=0x7fffae2839d0, ev=<optimized out>)
#12 0x00007ffff55d0a05 in morkThumb::DoMore (this=this@entry=0x7fffae2839d0, ev=ev@entry=0x7fffae5eeb00,
#13 0x00007ffff55d0a98 in morkThumb::DoMore (this=0x7fffae2839d0, mev=<optimized out>, outTotal=0x7fffffffba10,
#14 0x00007ffff54bd2dd in nsMsgDatabase::OpenMDB (this=0x7ffef1342400, dbName=<optimized out>, create=<optimized out>,
#15 0x00007ffff54c3418 in nsMsgDatabase::OpenInternal (this=0x7ffef1342400, summaryFile=0x7ffeeffeb540, aCreate=<optimized out>,
#16 0x00007ffff54bf8c2 in nsMsgDBService::OpenFolderDB (this=this@entry=0x7fffcc011e20, aFolder=aFolder@entry=0x7fffcc020038,
#17 0x00007ffff54e4014 in nsImapMailFolder::GetDatabase (this=0x7fffcc020000)
#18 0x00007ffff53c06f5 in nsMsgDBFolder::GetMsgDatabase (this=0x7fffcc020000, aMsgDatabase=0x7fffffffbdd8)
#19 0x00007ffff54cfb8e in nsAutoSyncState::ProcessExistingHeaders (this=0x7fffcc01b7f0, aNumOfHdrsToProcess=250,
#20 0x00007ffff54cbe4a in nsAutoSyncManager::TimerCallback (aTimer=<optimized out>, aClosure=0x7fffe7013940)
#21 0x00007ffff592413a in nsTimerImpl::Fire (this=0x7ffe519f2c40)
#22 0x00007ffff59241f5 in nsTimerEvent::Run (this=0x7fff60b3dbf8)
#23 0x00007ffff5921afc in nsThread::ProcessNextEvent (this=0x7ffff7d3cac0, mayWait=<optimized out>, result=0x7fffffffbfaf)

Comment 8

[Miroslav Suchy]

Attachment: backtrace from my machine

Comment 9

[Miroslav Suchy]

Attachment: another backtrace

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Bill McGonigle from comment #7)
> ...
> 3) moving mail out of the folder does not trim up the .msf file.  The .msf
> for this folder was 410MB.  It had 6 mail messages in (years ago it had
> hundreds of thousands).  I did a 'repair' and it went down to 7KB.
> 4) another giant folder had a 300MB .msf.  That one did have lots of mail
> messages.  A 'repair' brought it down to 32MB.  
> 7) looking at my backups, neither the file sizes nor the mtime on the local
> .msf files are changing over time on the old folders.

these sound like Bug 65086 - sometimes mork doesn't automatically compress commit - and therefore the folder does not shrink, even though some records in the msf file are obsolete


> 6) Based on the amount of thrash, it seems to be looking at every record in
> the .msf.  I thought mork had an index, but maybe I'm mistaken.

msf (mork) files are a flat file, tablular index into the corresponding *message folder*.  They are not themselves an index in the classic sense.  https://wiki.mozilla.org/Mork


> 5) The IMAP folder (server-side) hasn't had a new message since April.  If
> the synchronizer is running frequently on this local folder, it's not using
> the available information.

I don't understand this point. However, perhaps you are experiencing uid roll in imap. This will cause messages to be redownloaded, and it has been known to happen that msf and message folders grow in size as a result.  Are you seeing redownloads?


> 8) I'm unable to move 300,000 messages atomically from one folder to another
> on the client without exhausing 16GB of RAM and being reaped by the OOM
> killer.  `find` on the server and then 'repair' on the client is usable.

different issue

> 98) sqlite
> 99) running long operations s in the UI thread

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to Miroslav Suchy from comment #4)
> I hit the same issues (Thunderbird 24.6.0) and I can provide requested data.
> 
> Yes I compact folders once in while. But even compacted my folders are quite
> large. My Local Folders is 6.6 GB big.

Note, there can be performance issues with Thunderbird profiles when they are in NFS.

You say "Local folders", but by that do you mean the profile and all the locally stored/synced mail including imap?

> I have 'global search and index' switched off (for years actually).

Do you see this problem if you use a recent beta version from http://www.mozilla.org/en-US/thunderbird/channel/ ?

Comment 12

[Wayne Mery (:wsmwk)]

Bill,

(In reply to Wayne Mery (:wsmwk) from comment #10)
> (In reply to Bill McGonigle from comment #7)
> > ...
> > 3) moving mail out of the folder does not trim up the .msf file.  The .msf
> > for this folder was 410MB.  It had 6 mail messages in (years ago it had
> > hundreds of thousands).  I did a 'repair' and it went down to 7KB.
> > 4) another giant folder had a 300MB .msf.  That one did have lots of mail
> > messages.  A 'repair' brought it down to 32MB.  
> > 7) looking at my backups, neither the file sizes nor the mtime on the local
> > .msf files are changing over time on the old folders.
> 
> these sound like Bug 65086 - sometimes mork doesn't automatically compress
> commit - and therefore the folder does not shrink, even though some records
> in the msf file are obsolete

https://bugzilla.redhat.com/show_bug.cgi?id=493000#c35 plus the comments above suggest strong relationship to Bug 65086.

On how many folders are you seeing this type of behavior?
And for the most active folders, please give some example sizes of the largest files.


> > 5) The IMAP folder (server-side) hasn't had a new message since April.  If
> > the synchronizer is running frequently on this local folder, it's not using
> > the available information.
> 
> I don't understand this point. However, perhaps you are experiencing uid
> roll in imap. This will cause messages to be redownloaded, and it has been
> known to happen that msf and message folders grow in size as a result.  Are
> you seeing redownloads?

?


> > 8) I'm unable to move 300,000 messages atomically from one folder to another
> > on the client without exhausing 16GB of RAM and being reaped by the OOM
> > killer.  `find` on the server and then 'repair' on the client is usable.
> 
> different issue

and to be more clear, we have other bug reports on file that cover this issue.


Lastly, to cover a different performance possibility, in Edit | Preferences (on Windows Tools|Options)->Advanced->Config editor, find these 2 preferences with these values:
mail.db.idle_limit 300000
mail.db.max_open 30

Are they at those value

Then, try to increase mail.db.max_open to 10000 or more than the number of your folders in total.
Is your overall performance better?

Comment 13

[Miroslav Suchy]

Ad comment #11 - profiles is stored on local disk (magnetic HDD) no NFS is used. I see this problem with Thunderbird 31.3.0. I did not tried newer one.

Ad comment #12
my sizes of files inside of ImapFolder:
75M     announce                                                                                             
4.0M    announce.msf                                                                                         
8.0K    Archives-1.sbd                                                                                       
8.0K    Archives.msf                                                                                         
1.4M    Archives.sbd                                                                                         
271M    arm-list                                                                                             
12M     arm-list.msf                                                                                         
6.2M    beaker                                                                                               
1.8M    beaker.msf                                                                                           
361M    brno                                                                                                 
60M     brno.msf                                                                                             
51M     bugzilla                                                                                             
29M     bugzilla.msf                                                                                         
41M     bugzilla.sbd                                                                                         
3.8M    copr                                                                                                 
1.3M    copr.msf                                                                                             
16M     copr.sbd                                                                                             
19M     devexp                                                                                               
2.4M    devexp.msf                                                                                           
1.1M    Drafts                                                                                               
24K     Drafts.msf                                                                                           
8.0K    filterlog.html                                                                                       
49M     github                                                                                               
5.9M    github.msf                                                                                           
28M     INBOX
3.4M    INBOX.msf
32K     INBOX.sbd
2.9M    Junk
108K    Junk.msf
20K     msgFilterRules.dat
388M    Sent
3.5M    Sent.msf
68K     simulator
20K     simulator.msf
139M    spacewalk
20M     spacewalk.msf
12M     Trash
3.6M    Trash.msf
40K     Trash.sbd

and sizes of LocalFolders:
284K    bugzilla.sbd/katello bugs
24K     bugzilla.sbd/katello bugs.msf
7.3M    bugzilla.sbd/mine comments
1.1M    bugzilla.sbd/mine comments.msf
35M     list.sbd/android-users
5.1M    list.sbd/android-users.msf
3.5M    list.sbd/dsc
532K    list.sbd/dsc.msf
2.5M    list.sbd/env-and-stack
1.1M    list.sbd/env-and-stack.msf
932M    list.sbd/fedora-devel
361M    list.sbd/fedora-devel.msf
93M     list.sbd/fedora-infrastructure
12M     list.sbd/fedora-infrastructure.msf
16M     list.sbd/fedora-packaging
5.8M    list.sbd/fedora-packaging.msf
35M     list.sbd/infrastructure-mails
13M     list.sbd/infrastructure-mails.msf
6.0M    list.sbd/infrastructure-tickets
2.5M    list.sbd/infrastructure-tickets.msf
36M     list.sbd/katello-devel
3.8M    list.sbd/katello-devel.msf
308M    list.sbd/katello-internal
26M     list.sbd/katello-internal.msf
136M    list.sbd/memo
34M     list.sbd/memo.msf
637M    list.sbd/memo.sbd
17M     list.sbd/nagios
13M     list.sbd/nagios.msf
24M     list.sbd/obs
7.4M    list.sbd/obs.msf
12M     list.sbd/os1-cloud-users
2.6M    list.sbd/os1-cloud-users.msf
58M     list.sbd/os-devel
8.0M    list.sbd/os-devel.msf
11M     list.sbd/rhel7-planning
2.3M    list.sbd/rhel7-planning.msf
27M     list.sbd/rhos-tech
4.0M    list.sbd/rhos-tech.msf
86M     list.sbd/spacewalk
4.8M    list.sbd/spacewalk.msf
42M     list.sbd/systemd-devel
11M     list.sbd/systemd-devel.msf
13M     x-list-not-interesting.sbd/brno-pto
8.3M    x-list-not-interesting.sbd/brno-pto.msf
44M     x-list-not-interesting.sbd/centos-devel
11M     x-list-not-interesting.sbd/centos-devel.msf
17M     x-list-not-interesting.sbd/eng
1.3M    x-list-not-interesting.sbd/eng.msf
83M     x-list-not-interesting.sbd/epel-devel
8.3M    x-list-not-interesting.sbd/epel-devel.msf
41M     x-list-not-interesting.sbd/europe
1.4M    x-list-not-interesting.sbd/europe.msf
5.3M    x-list-not-interesting.sbd/fedora-legal
1.2M    x-list-not-interesting.sbd/fedora-legal.msf
60M     x-list-not-interesting.sbd/fedora-perl-devel
4.9M    x-list-not-interesting.sbd/fedora-perl-devel.msf
132M    x-list-not-interesting.sbd/fedora-testing
15M     x-list-not-interesting.sbd/fedora-testing.msf
29M     x-list-not-interesting.sbd/fedora-translation
4.2M    x-list-not-interesting.sbd/fedora-translation.msf
7.8M    x-list-not-interesting.sbd/katello-fedorahosted
588K    x-list-not-interesting.sbd/katello-fedorahosted.msf
208K    x-list-not-interesting.sbd/nagios (fedora)
64K     x-list-not-interesting.sbd/nagios (fedora).msf
6.4M    x-list-not-interesting.sbd/outages
4.0K    x-list-not-interesting.sbd/outages.msf
8.2M    x-list-not-interesting.sbd/prod-dept
1.3M    x-list-not-interesting.sbd/prod-dept.msf
20M     x-list-not-interesting.sbd/pulp
3.5M    x-list-not-interesting.sbd/pulp.msf
45M     x-list-not-interesting.sbd/rhel-people-statuses
15M     x-list-not-interesting.sbd/rhel-people-statuses.msf
21M     x-list-not-interesting.sbd/rhn-list
876K    x-list-not-interesting.sbd/rhn-list.msf
988K    x-list-not-interesting.sbd/rhn-proxy
100K    x-list-not-interesting.sbd/rhn-proxy.msf
126M    x-list-not-interesting.sbd/rhn-satellite
6.5M    x-list-not-interesting.sbd/rhn-satellite.msf
22M     x-list-not-interesting.sbd/rhn-satellite-user
1.9M    x-list-not-interesting.sbd/rhn-satellite-user.msf
96K     x-list-not-interesting.sbd/rh-scl
40K     x-list-not-interesting.sbd/rh-scl.msf
2.7M    x-list-not-interesting.sbd/rpm-list
688K    x-list-not-interesting.sbd/rpm-list.msf
4.7M    x-list-not-interesting.sbd/ruby-sig
1.2M    x-list-not-interesting.sbd/ruby-sig.msf
232M    x-list-not-interesting.sbd/satellite-dept
55M     x-list-not-interesting.sbd/satellite-dept.msf
177M    x-list-not-interesting.sbd/satellite-devel
12M     x-list-not-interesting.sbd/satellite-devel.msf
276M    x-list-not-interesting.sbd/tech-list
32M     x-list-not-interesting.sbd/tech-list.msf
9.6M    x-list-not-interesting.sbd/yum
20M     x-list-not-interesting.sbd/yum-devel
8.5M    x-list-not-interesting.sbd/yum-devel.msf
1.2M    x-list-not-interesting.sbd/yum.msf
61M     z-archive.sbd/2007
736K    z-archive.sbd/2007.msf
88M     z-archive.sbd/2008
1.4M    z-archive.sbd/2008.msf
51M     z-archive.sbd/2009
2.1M    z-archive.sbd/2009.msf
201M    z-archive.sbd/2010
2.6M    z-archive.sbd/2010.msf
97M     z-archive.sbd/2011
2.7M    z-archive.sbd/2011.msf
184M    z-archive.sbd/2012
4.4M    z-archive.sbd/2012.msf
130M    z-archive.sbd/2013
13M     z-archive.sbd/2013.msf
141M    z-archive.sbd/2014
16M     z-archive.sbd/2014.msf
1.1M    z-archive.sbd/ackbar
92K     z-archive.sbd/ackbar.msf
20M     z-archive.sbd/brno-admins
1.6M    z-archive.sbd/brno-admins.msf
1.5M    z-archive.sbd/entitlement-feedback
68K     z-archive.sbd/entitlement-feedback.msf
3.7M    z-archive.sbd/google-calendar-pilot
180K    z-archive.sbd/google-calendar-pilot.msf
42M     z-archive.sbd/imanage
1.9M    z-archive.sbd/imanage.msf
7.2M    z-archive.sbd/oss-sat
288K    z-archive.sbd/oss-sat.msf
11M     z-archive.sbd/rhn-user
840K    z-archive.sbd/rhn-user.msf
152M    z-archive.sbd/satellite-commit
7.3M    z-archive.sbd/satellite-commit.msf
20M     z-archive.sbd/satellite-release-team
1.2M    z-archive.sbd/satellite-release-team.msf
1.1G    z-archive.sbd/satellite-se
6.5M    z-archive.sbd/satellite-se.msf
96M     z-archive.sbd/sent-2006-2010
648K    z-archive.sbd/sent-2006-2010.msf
1.7M    z-archive.sbd/star-network
120K    z-archive.sbd/star-network.msf
1.3M    brew
652K    brew.msf
4.0K    bugzilla
8.0K    bugzilla.msf
8.0K    bugzilla.sbd
14M     cron
5.2M    cron.msf
380K    Drafts
68K     Drafts.msf
8.0K    filterlog.html
0       Inbox
4.0K    Inbox.msf
72K     Junk
4.0K    Junk.msf
4.0K    list
8.0K    list.msf
8.0K    list.sbd
56M     LOGWATCH
20M     LOGWATCH.msf
4.0K    msgFilterRules.dat
8.0K    Sent
12K     Sent.msf
8.0K    Templates.msf
44M     Trash
856K    Trash.msf
4.0K    Trash.sbd
0       Unsent Messages
4.0K    Unsent Messages.msf
4.0K    x-list-not-interesting
8.0K    x-list-not-interesting.msf
8.0K    x-list-not-interesting.sbd
4.0K    z-archive
8.0K    z-archive.msf
8.0K    z-archive.sbd

[msuchy@dri/~/.thunderbird/2nn7z7bh.default/Mail/Local Folders]$ du -sh .
7.1G    .

I will try to set this value in config editor next week as I'm out of office this week. I will provide reports then.

Comment 14

[Wayne Mery (:wsmwk)]

(In reply to Miroslav Suchy from comment #13)
> Ad comment #11 - profiles is stored on local disk (magnetic HDD) no NFS is
> used. I see this problem with Thunderbird 31.3.0. I did not tried newer one.
> 

> and sizes of LocalFolders:
> 932M	list.sbd/fedora-devel
> 361M	list.sbd/fedora-devel.msf
> ...
> 637M    list.sbd/memo.sbd

Of all the files listed, these are the only ones that stand out for me.  Are these files (or any others) bigger than they should be for number and size of messages in those folders?


> During the 100% CPU utilization the strace shows read() and lseek() on various *.msf files.

For just a few .msf files? Or are you seeing this for all of them

Comment 15

[Bill McGonigle (not currently reading bugmail; please contact directly)]

if I understand bug 65086 correctly, I'd have a large panacea.dat if this were the same, but it looks sane to me:

  -rw-rw-r-- 1 user user 2.8M Feb 25 16:55 panacea.dat

however, rebuilding the msf index files ("Repair Folder") does make a big difference, so maybe it has something to do with it:

  https://bugzilla.redhat.com/show_bug.cgi?id=493000#c35   (2010-04)

I have to say I'm not experiencing the UI lockup regularly because I've gotten in the habit of breaking up my large folders into sub-folders by year every January (logs/, logs/2014, logs/2013, logs/2012) as a workaround.

Anecdotally, if the message count stays below ~ 256K in a folder, I don't seem to see the .msf thrash, but if I let it creep up over that (non-scientific observation) it gets very bad very fast.  I don't know if this might be a limit within Mork or Thunderbird - the size of the messages don't seem to matter as much as the number of messages (possibly the size of the .msf then).

I checked mail.db.idle_limit and mail.db.max_open and they were set to defaults.  Bumped max_open to 10000 as I currently have 265 folders.  All but 32 of my .msf files are under 10M - the top 10 are:

49M
57M
63M
64M
71M
81M
90M
101M
115M
163M

Just for grins, I found the folder with the 163M .msf, pressed 'Repair Folder' and now it's 39M.  The modification date on both the mbox and the .msf were previously 2014-12-10, and after 'Repair' the .msf is dated now.  So, if something is supposed to be optimizing/compacting .msf files at idle time, that's not happening (at least effectively).

-rw------- 1 user user 754M Dec 10 17:01 ImapMail/example.com/Logs to Review.sbd/2009
-rw-rw-r-- 1 user user 163M Dec 10 17:14 ImapMail/example.com/Logs to Review.sbd/2009.msf

Re: comment #10, item 5) - what I meant there is that the IMAP folder isn't downloading any new messages (I can see the filesystem modification date doesn't change for the mailbox file) but I was seeing the associated .msf being thrashed on a regular basis.  If this is the Global Indexer (suggested elsewhere), it should be able to figure out that the .msf hasn't changed since the global index was last updated, so it doesn't need to index anything for the given folder. 

I have Thunderbird 31.4.0 currently.

Comment 16

[Wayne Mery (:wsmwk)]

Global indexing will have nothing to do with this. The only reason I put it in the summary was to remind ourselves that problem exists without it.

panacea.dat it need not be corrupted to have performance issues, which is size dependant. 
And folders not properly compacting is no doubt also involved.
What is not yet known, is what to do about it.

What is the compact size specified at tools | options | advanced | network & disk?
And Is the compact enabled?

Comment 17

[Wayne Mery (:wsmwk)]

Bill, Miroslav, in addiiton to questions above, does problem go away if you DISABLE message sync in account settings at Sync & Storage?

Comment 18

[Wayne Mery (:wsmwk)]

Bill...

(In reply to Bill McGonigle (not currently reading bugmail; please contact directly) from comment #15)
> if I understand bug 65086 correctly, I'd have a large panacea.dat if this
> were the same, but it looks sane to me:
> 
>   -rw-rw-r-- 1 user user 2.8M Feb 25 16:55 panacea.dat
> 
> however, rebuilding the msf index files ("Repair Folder") does make a big
> difference, so maybe it has something to do with it:
> 
>   https://bugzilla.redhat.com/show_bug.cgi?id=493000#c35   (2010-04)
> ...
> Just for grins, I found the folder with the 163M .msf, pressed 'Repair
> Folder' and now it's 39M.  The modification date on both the mbox and the
> .msf were previously 2014-12-10, and after 'Repair' the .msf is dated now. 
> So, if something is supposed to be optimizing/compacting .msf files at idle
> time, that's not happening (at least effectively).

compress commit is what reduces the size of panacea.dat. compress commit is also what reduces the size of .msf files. This *should* happen when a folder is compacted. But it is possible for compress commit to fail, or not be called.  Hence, my question to you in comment 16 about compact.

 
> -rw------- 1 user user 754M Dec 10 17:01 ImapMail/example.com/Logs to
> Review.sbd/2009
> -rw-rw-r-- 1 user user 163M Dec 10 17:14 ImapMail/example.com/Logs to
> Review.sbd/2009.msf

That's a hefty change!

> Re: comment #10, item 5) - what I meant there is that the IMAP folder isn't
> downloading any new messages (I can see the filesystem modification date
> doesn't change for the mailbox file) but I was seeing the associated .msf
> being thrashed on a regular basis.  If this is the Global Indexer (suggested
> elsewhere), it should be able to figure out that the .msf hasn't changed
> since the global index was last updated, so it doesn't need to index
> anything for the given folder. 
> 
> I have Thunderbird 31.4.0 currently.


(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #16)
> ...
> What is the compact size specified at tools | options | advanced | network &
> disk?
> And Is the compact enabled?

question ^

Also, do you have folder retention settings at an account or folder level, to automatically delete messages after x days?

Comment 19

[Miroslav Suchy]

The problem still persist even when I disable message sync in account settings (I have two accounts and I disabled both).

> Then, try to increase mail.db.max_open to 10000 or more than the number of your folders in total.
Is your overall performance better?

Hm... I have ~60 folders. I increased it to 1000 and it seem to behave nice. (Shame on me that I did not tried this one at February). I will give it one more day testing if this is really the fix. At least for me.

Comment 20

[Wayne Mery (:wsmwk)]

(In reply to Miroslav Suchy from comment #19)
> The problem still persist even when I disable message sync in account
> settings (I have two accounts and I disabled both).
> 
> > Then, try to increase mail.db.max_open to 10000 or more than the number of your folders in total.
> Is your overall performance better?
> 
> Hm... I have ~60 folders. I increased it to 1000 and it seem to behave nice.
> (Shame on me that I did not tried this one at February). I will give it one
> more day testing if this is really the fix. At least for me.

Thanks for checking. Based on your check ... 

1. your problem _should_ be fixed in version 38 with mail.db.max_open reset to defalt values.  You find 38 beta from https://www.mozilla.org/en-US/thunderbird/all-beta.html

2. This is not to say that you might not have other issues. For example, from comment 14
> and sizes of LocalFolders:
> 932M	list.sbd/fedora-devel
> 361M	list.sbd/fedora-devel.msf

fedora-devel.msf is about half the size of fedora-devel, which is unusual for a large folder. How many messages in the folder?

Comment 21

[Miroslav Suchy]

Few weeks ago I purged some old email, but it is still:
477M ./Mail/Local Folders/list.sbd/fedora-devel
110M ./Mail/Local Folders/list.sbd/fedora-devel.msf
Which is 42488 pieces of emails. I suppose that originally it was over 100 thousands of emails in this folder.

Nevertheless, after one day of usage I do no see theses symptoms any more. From my POV this report can be closed.
It would be nice if Bill can confirm it too.

Comment 22

[Wayne Mery (:wsmwk)]

(In reply to Miroslav Suchy from comment #21)
> Few weeks ago I purged some old email, but it is still:
> 477M ./Mail/Local Folders/list.sbd/fedora-devel
> 110M ./Mail/Local Folders/list.sbd/fedora-devel.msf
> Which is 42488 pieces of emails. I suppose that originally it was over 100
> thousands of emails in this folder.

Miroslav, Do you have custom headers defined?  
check at Tools | Options | Advanced | General | Config Editor | paste in mailnews.customHeaders

I ask because 4:1 size ratio folder:msf is much better than 2:1, but the .msf is still 2.5k per message. 

> Nevertheless, after one day of usage I do no see theses symptoms any more.
> From my POV this report can be closed.

That's good news. But it's premature to close this bug, even if Bill responds, until it has been confirmed that the issue is also gone for you and Bill when using version 38 with default settings.  

That goes double for https://bugzilla.redhat.com/show_bug.cgi?id=493000, which has many users reporting issues, and we also have not heard from the author of the bug report.

Comment 23

[Ludovic Hirlimann [:Usul]]

Removing myslef on all the bugs I'm cced on. Please NI me if you need something on MailNews Core bugs from me.

Comment 24

[Wayne Mery (:wsmwk)]

(attempted to contact Bill the reporter via a couple different email addresses (mail rejected) and by business phone (system hangs up before a message can be left) but all attempts fail.)

Comment 25

[Wayne Mery (:wsmwk)]

I can imagine that bug 1116055 will have an effect on this.  It fits the bulk of Bill's problem. When it lands it would be nice if Bill comments

But other factors are involved..

Miroslav in https://bugzilla.redhat.com/show_bug.cgi?id=493000 writes that setting mail.db.max_open to values ~60 helps greatly, for imap accounts with 100s of folders, 14 of which have server side filters putting messages into them. 

Also bug 588952 should help Miroslav - open folders async for autosync. 
Unclear whether bug 551209 is relevant - aync folder open for gloda (it certainly will not help bill)

Comment 26

[doug2]

daily 9/22/17 (? about) using from 30% to 48% cpu and over 500MB memory, no Internet.  Not able to open mail.  
Shows "Not responding" at top of window.

Comment 27

[doug2]

see above,  57.0a1 9/22/17 Had to kill TBird with task manager.  Started back up ok. Now running 2%-3% with 100MB. Received several pieces of mail when restarted, may have been trying to receive?  POP, SMTP, Windows 10 64 bit.

Comment 28

[Wayne Mery (:wsmwk)]

(In reply to doug2 from comment #26)

daily 9/22/17 (? about) using from 30% to 48% cpu and over 500MB memory, no
Internet. Not able to open mail.
Shows "Not responding" at top of window.

Doug, do you still see this ?

Comment 29

[Wayne Mery (:wsmwk)]

(correcting NI to doug)

Comment 30

[doug2]

No. Haven't seen any "spinning" problems.

Comment 31

[Miroslav Suchy]

I can confirm that since https://bugzilla.redhat.com/show_bug.cgi?id=493000 has been resolved, this stopped. I believe it is the same issue.

Comment 32

[doug2]

I believe bug 711204 is also resolved "coincidentally?" in Daily. I have hesitated to close it but have not seen that strange closing window action in a while (which I believe is related to a compaction happening when I first open TBird after a "nap").

Comment 33

[Wayne Mery (:wsmwk)]

(In reply to Miroslav Suchy from comment #31)

I can confirm that since https://bugzilla.redhat.com/show_bug.cgi?id=493000 has been resolved, this stopped. I believe it is the same issue.

That bug of course didn't change any code itself, so it remains to be seen whether Bill's issue is really gone.

Miroslav, which version are you using? And the issue is gone for you with no mail.db.xxx settings being used?

Comment 34

[Miroslav Suchy]

I am using 68.10.0. No manual changes to mail.db.xxx.

Comment 35

[Wayne Mery (:wsmwk)]

(In reply to doug2 from comment #32)

I believe bug 711204 is also resolved "coincidentally?" in Daily. I have hesitated to close it but have not seen that strange closing window action in a while (which I believe is related to a compaction happening when I first open TBird after a "nap").

Compact certainly could cause high IO, which is why bug 520115 was associated to your bug 711204.

Perhaps a way to test for compact effects is to turn off automatic compact for some extended period of time, so that many messages deletes and moves can be accumulated. Then turn compact on, set it to a low value, and restart thunderbird a couple times.

Comment 36

[Wayne Mery (:wsmwk)]

(Bill is gone.)

Comment 37

[Wayne Mery (:wsmwk)]

WFM per comment 31