876978 - sent box very, very slow after Thunderbird has been running for a while

Status: RESOLVED WORKSFORME

(Thunderbird :: Untriaged, defect)

Description

[Todd]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0 (Beta/Release)
Build ID: 20130511120803

Steps to reproduce:

Scientific Linux 6.4, 64 bit (Red Hat Enterprise Linux clone).
Thunderbird 17.0.6, 64 bit

Would one of our intrepid heroes please fix this for me?

I have five POP3 eMail accounts.  When I have left Thunderbird running for several hours, trying to open my Sent box is a nightmare.  It is so very, very slow.  I have to exit Thunderbird and restart it to be able to see my Sent box.  (This is a pain in the neck as my passwords are not remembered and I have to reenter them.  They are very long passwords too.)  After restarting, I am okay for about 30 minutes, then the problem comes back.

Comment 1

[WADA:World Anti-bad-Duping Agency]

What is file size? (file named Sent, not Sent.msf. See Folder Properties of Sent folder, General Tab, Location: field for directory path)
Is Sent, Sent.msf writable?

What is your auto-compact setting?
(Tools/Options/Advaced/Network&Disk Space, Disk Space, [?] Compact all folders...)
What is relevant setting in prefs.js?
(via Tools/Options/Advanced/General, Config Editr)
   mail.prompt_purge_threshhold  : Do auto-compact, or not
   mail.purge_threshhold_mb      : Threshold value in MB    
   mail.purge.ask  : Confirm before start auto-compact, or Do silently

Comment 2

[Todd]

Permissions on all Sent and Sent.msf are 766.  They do not change while he program is running.  (Remember this problem takes time to occur.  If the permissions were wrong, they'd never work).

Account #1:   Sent=46.4 KiB;   Sent.msf=22.5 Kib
Account #2:   Sent=173.3 KiB;  Sent.msf=40.6 Kib
Account #3:   Sent=9.1 MiB;    Sent.msf=331.7 Kib
Account #4:   Sent=10.4 MiB;   Sent.msf=571.3 Kib
Account #5:   Sent=785.4 KiB;  Sent.msf=191.3 Kib

Tools/Options/Advaced/Network&Disk Space, Disk Space, [?] Compact all folders...
--> compact over 1MB
That is actually "Edit","Preferences" in Linux.  It would be cool if you made Linux and Windows the same at some point.

user_pref("mail.purge.ask", false);
user_pref("mail.purge_threshhold", 2000);
user_pref("mail.purge_threshhold_mb", 1);
user_pref("mail.purge_threshold_migrated", true);

If I missed anything, please let me know.  This problem has been going on for at least two years.

-T

Comment 3

[WADA:World Anti-bad-Duping Agency]

"Too big Sent folder" case is ruled out.
But, mail.purge_threshhold_mb=1 and mail.purge.ask=false (I assume mail.prompt_purge_threshhold=true) indicates that "auto-compact of all folders where delete mail occurred" is executed silently after one hour from last auto-compact execution, if more than 1MB data is deleted from mail folders of your Tb after the last Compact.

First, set mail.purge.ask=true, restart Tb(restart of Tb is mandatory).
When confirmation dialog is shown before start of auto-compact, reply Cancel.
Do you still see your problem?

(In reply to Todd from comment #0)
> I have to exit Thunderbird and restart it to be able to see my Sent box.
> (This is a pain in the neck as my passwords are not remembered
> and I have to reenter them.  They are very long passwords too.)

Is actually Sent folder problem?

If mail send via SMTP was successful once and password was saved by password manager, saved password won't be cleared and is re-used by next mail send.

Is mail sending successful?

(1) Disable "Sent Mail Copy" of an identity(Copies&Folders)
(2) Compose a small text mail using the identity(From: address),
    and set your mail address in To:.
(3) Send Later => mail is saved in Outbox of Local Folders".
(4) At Outbox, Send Unsent Messages.
    By (1), sent mail is not saved in Sent folder by Tb.
    So, Sent folder is absolutely irrelevant in this operation.

Comment 4

[Todd]

(In reply to WADA from comment #3)
 
> First, set mail.purge.ask=true, restart Tb(restart of Tb is mandatory).
> When confirmation dialog is shown before start of auto-compact, reply Cancel.
> Do you still see your problem?

I will leave TH up all day and get back to you.

 
> Is mail sending successful?

No problem.  Send and receive work perfectly


> (1) Disable "Sent Mail Copy" of an identity(Copies&Folders)
> (2) Compose a small text mail using the identity(From: address),
>     and set your mail address in To:.
> (3) Send Later => mail is saved in Outbox of Local Folders".
> (4) At Outbox, Send Unsent Messages.
>     By (1), sent mail is not saved in Sent folder by Tb.
>     So, Sent folder is absolutely irrelevant in this operation.

Got a test message setting in drafts.  Turned sent copy back on afterward.

By the way, drafts gets real slow too after a bit.

Comment 5

[Wayne Mery (:wsmwk)]

(In reply to Todd from comment #4)
> By the way, drafts gets real slow too after a bit.

what is your interval in preferences | composition | general | autosave __ minutes ?

Comment 6

[Todd]

(In reply to Wayne Mery (:wsmwk) from comment #5)
> (In reply to Todd from comment #4)
> > By the way, drafts gets real slow too after a bit.
> 
> what is your interval in preferences | composition | general | autosave __
> minutes ?

five minutes

Additional info: it have been 2 hours now.  I have been prompted to compact once and pressed cancel.  And! My Sent folder is all jammed up (slow to respond).

Comment 7

[Todd]

Is this a clue?

$ ps -eLF | grep -i thunderb
UID        PID  PPID   LWP  C NLWP    SZ   RSS PSR STIME TTY          TIME CMD
todd     12608     1 12608  3   19 267829 220128 0 09:35 ?        00:04:27 /usr/bin/thunderbird
todd     12608     1 12610  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12611  0   19 267829 220128 6 09:35 ?        00:00:02 /usr/bin/thunderbird
todd     12608     1 12612  0   19 267829 220128 2 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12613  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12614  0   19 267829 220128 1 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12615  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12616  0   19 267829 220128 3 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12617  0   19 267829 220128 7 09:35 ?        00:00:05 /usr/bin/thunderbird
todd     12608     1 12619  0   19 267829 220128 5 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12622  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12628  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12629  0   19 267829 220128 6 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12631  0   19 267829 220128 7 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12632  0   19 267829 220128 2 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12648  0   19 267829 220128 4 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12651  0   19 267829 220128 7 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 12652  0   19 267829 220128 5 09:35 ?        00:00:00 /usr/bin/thunderbird
todd     12608     1 25132  0   19 267829 220128 4 11:45 ?        00:00:00 /usr/bin/thunderbird
todd     25736 25712 25736  0    1 25812   880   0 11:51 pts/2    00:00:00 grep -i thunderb

Comment 8

[Todd]

Attachment: My Open Files

These are my open files assigned to process 12608 (Thunderbird)

Comment 9

[Todd]

Attachment: Process When Try To Read Sent Folder

For the fun of it, I monitored my processes when clicking on my Sent box.  There was a freak out.  I thought it might be a clue.

Comment 10

[Todd]

Thunderbird memory usage:

At start up:
# ps p 31892 o pid,rss,vsz,comm
  PID   RSS    VSZ COMMAND
31892 177824 943820 thunderbird


After running five hour
# ps p 12608 o pid,rss,vsz,comm
  PID   RSS    VSZ COMMAND
12608 235092 1093976 thunderbird


Doesn't look like a memory leak

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to Todd from comment #6)
> Additional info: it have been 2 hours now.
> I have been prompted to compact once and pressed cancel.

Prompted ONLY ONCE after change to mail.purge.ask=true, even though mail.purge_threshhold_mb=1 is intentionally kept and Cancel is intentionally replied for problem determiation?

What is currently set in mail.purge.ask? (check via Config Editor)

Comment 12

[Todd]

(In reply to WADA from comment #11)
> (In reply to Todd from comment #6)
> > Additional info: it have been 2 hours now.
> > I have been prompted to compact once and pressed cancel.
> 
> Prompted ONLY ONCE after change to mail.purge.ask=true, even though
> mail.purge_threshhold_mb=1 is intentionally kept and Cancel is intentionally
> replied for problem determiation?

Oh it prompted sever times.  I clicked Cancel each time.  The problem came back anyway.

> 
> What is currently set in mail.purge.ask? (check via Config Editor)

I set it to false after being prompted several times and the problem had already comes back.  I had to restart TH several times to dig things out of my Sent box while ask was true.  

Why the processor storm when I click on the Sent box?

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to Todd from comment #12)
> Oh it prompted several times.  I clicked Cancel each time.  The problem came back anyway.

It indicates that cause of your problem on Sent is not auto-compact.

> I set it to false after being prompted several times and the problem had already comes back.

If auto-compact is being silently executed when you tries to access a folder, it may interfere folder access. This is always true regardless of cause of your problem is auto-compact or not.
The purpose of "mail.purge.ask=true and Cancel reply always" is to always gurantee that auto-compact is surely not executed when your problem occurs.

Keep following, and reply Cancel usually.
  mail.purge.ask=true
   (If changed from false to true, restart of Tb is MANDATORY.)
   (To keep true, never check [] do this automatically ... at Dialog.)
  mail.prompt_purge_threshhold=true
   (Keep true, don't kill auto-compact, to know "Compact needed")
   (status by dialog before auto-compact by mail.purge.ask=true.) 
  mail.purge_threshhold_mb=1 => increase. 100 MB or several handreds MB
If Cancel is replied to Dialog, dialog is perhaps shown again after one hour. If you see dialog several times, do File/"Compact Folders" manually when Tb is not so busy, to force execution of Compact. 

> By the way, drafts gets real slow too after a bit.

How long does it take to save draft usually?
(0) Check "Show confirmation dialog ..." at Copies&Folders.
    Disable auto-save, to measure time needed to Save as draft.
(1) Compose a large mail with attaching some files,
    with attaching some mails(Drag&Drop some mails to attachment box),
    do "Save" several times, and close composition window.
(2) Create a local mail folder(call FolderX), 
    and move the draft mail to FolderX for testing.
(3) Copy test mail in FolderX to Drafts
(4) Copy test mail in FolderX to Sent
How long does it take to Save or Copy?
(5) Edit the mail in Drafts, Send Later.
    => mail data is generated in Outbox of "Local Folders"
How long does it take to do "Send Later"?

Comment 14

[WADA:World Anti-bad-Duping Agency]

(In reply to Todd from comment #0)
> When I have left Thunderbird running for several hours,
> trying to open my Sent box is a nightmare.  It is so very, very slow.
> I have to exit Thunderbird and restart it to be able to see my Sent box. 
> After restarting, I am okay for about 30 minutes, then the problem comes back.

Questions about phenomenon.

What operation do you call by "trying to open my Sent box"?
Does the "It is so very, very slow" occur on any Sent folder of any account?

What folder do you call by "Sent"?
Any folder named "Sent" of any account? Any folder used as "folder to save sent mail copy" which is set in Copies&Folders of each Identity? Both?

When the "It is so very, very slow", what kind of other phenomena are observed? (Click menu=>no responding, Minimize Tb=>Nothing occur, CPU 100%, try to open Tab=>Tb's window is not re-painted, ...)

How long do you wait before restart Tb when the problem occurs?

After restart of Tb, before "30 minutes from restart", is any mail in any Sent folder shown normally as expected?

Do you enable automatic deletion of mails by retention policy?
  POP3 : Account Settings, Disk Space
  IMAP : Account Settings, Synchronization&Disk Space
  Each folder : Folder Properties, Retention Policy

Comment 15

[Wayne Mery (:wsmwk)]

you might take a spin through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

Comment 16

[Todd]

(In reply to Wayne Mery (:wsmwk) from comment #15)
> you might take a spin through
> https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems

Take a look at comment 10.  Do you think a leak would not show up in that test?

Comment 17

[Todd]

> If auto-compact is being silently executed when you tries to access a
> folder, it may interfere folder access. This is always true regardless of
> cause of your problem is auto-compact or not.

This won't cause a CPU storm.  Just slow disk access.

> If you see dialog several times, do File/"Compact Folders" manually
> when Tb is not so busy, to force execution of Compact. 

There is no change if you repair or compact the folder

> 
> > By the way, drafts gets real slow too after a bit.
> 
> How long does it take to save draft usually?

Instant.  Trying to get them back out is a chore.

Comment 18

[Todd]

(In reply to WADA from comment #14)
> (In reply to Todd from comment #0)
> > When I have left Thunderbird running for several hours,
> > trying to open my Sent box is a nightmare.  It is so very, very slow.
> > I have to exit Thunderbird and restart it to be able to see my Sent box. 
> > After restarting, I am okay for about 30 minutes, then the problem comes back.
> 
> Questions about phenomenon.
> 
> What operation do you call by "trying to open my Sent box"?
> Does the "It is so very, very slow" occur on any Sent folder of any account?

In the left pane, the are folder names.  One is called “Sent”.  It has an arrow 
pointing at it.  Clicking on the word “Sent” or clicking on the arrow to
open the drop down with the five sent boxes plus local causes the CPU storm.

During the storm, the entire windows will not paint either.  Meaning, if you go to
another program which paints on top of Thunderbird, going back to Thunderbird
leaves the artifacts of the previous program in Thunderbird's windows.  When
the store subsides, the windows will paint properly and get rid of the artifacts.

> 
> What folder do you call by "Sent"?

The one in the left pane that says “Sent”.

> Any folder named "Sent" of any account? Any folder used as "folder to save
> sent mail copy" which is set in Copies&Folders of each Identity? Both?

All of them.

> When the "It is so very, very slow", what kind of other phenomena are
> observed? (Click menu=>no responding, Minimize Tb=>Nothing occur, CPU 100%,
> try to open Tab=>Tb's window is not re-painted, …)

CPU storm.  Window won't repaint.  See above.

> 
> How long do you wait before restart Tb when the problem occurs?

Usually I do not have time to wait, so I restart after about 30 seconds.
The times I have let it run its course, up to 10 minutes and it will
eventually show the Send box(es)

> After restart of Tb, before "30 minutes from restart", is any mail in any
> Sent folder shown normally as expected?

Yes.  After a restart, I can operate all my boxes without problem.  It
is annoying to hae to restart all the time though

> Do you enable automatic deletion of mails by retention policy?
Yes

>   POP3 : Account Settings, Disk Space
Everything is default

>   IMAP : Account Settings, Synchronization&Disk Space
No using imap

>   Each folder : Folder Properties, Retention Policy
Everything is default

Comment 19

[Wayne Mery (:wsmwk)]

(In reply to Todd from comment #16)
> (In reply to Wayne Mery (:wsmwk) from comment #15)
> > you might take a spin through
> > https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems
> 
> Take a look at comment 10.  Do you think a leak would not show up in that
> test?

That amount of change is not significant. Nor is it necessarily a key indicator to the cause of your CPU problem 

Again I suggest looking at what's in the doc

Comment 20

[WADA:World Anti-bad-Duping Agency]

(In reply to Todd from comment #18)

Some other questions.

It sounds you use Folder View = "Unified". Right?
If so, do you see your problem with Folder View = All?

> > Do you enable automatic deletion of mails by retention policy?
> Yes
> >   POP3 : Account Settings, Disk Space
> Everything is default
> >   Each folder : Folder Properties, Retention Policy
> Everything is default

IIRC, default of "Disk Space" is "Don't delete any messages", and many users usually calls it "delete by retention policy is disabled".
Does "Everything is default" for "Disk Space" and per folder "Retention Policy" mean "Don't delete any messages" is checked?

As you say "CPU storm", phenomenon you saw is not Freeze/Wait type(no CPU hog, no memory usage change or pretty small memory usage change, ...).
Loop like one?
Or O(2^n)/O(n^X)/O(n!) CPU consumption like one?
(if O(2^n) problem like one, "wait 30 second only" is too short for problem determination...)

Is "heavy disk access" like phenomenon observed when the "CPU storm" happens?
Is externally observed phenomenon "CPU storm forever(or too long)" only?

Is there any specific operation other than "click Sent" which may be a trigger of the "CPU storm"?
Is there something funny in Error Console or Activity manager before your "CPU storm problem occur?

No problem if any folder other than "Sent" folder?

Do you see your problem with Tb's -safe-mode? (thunderbird.exe -safe-mode if Win).

Comment 21

[WADA:World Anti-bad-Duping Agency]

Additional quetion.
When dis your problem start to occur? After upgrade to Tb 17.0.6? Or Tb 17.0.6 is your first Tb?

Comment 22

[Todd]

(In reply to WADA from comment #21)
> Additional quetion.
> When dis your problem start to occur? After upgrade to Tb 17.0.6? Or Tb
> 17.0.6 is your first Tb?

It has always been there.  Can't remember when I first noticed it.

Comment 23

[Todd]

(In reply to WADA from comment #20)
> (In reply to Todd from comment #18)
> 
> Some other questions.
> 
> It sounds you use Folder View = "Unified". Right?
Yes

> If so, do you see your problem with Folder View = All?
Testing in Safe mode as I write this.  Safe mode removes the
unified.
 
> No problem if any folder other than "Sent" folder?
Other folders with problems are Drafts, Templates, Trash.  I use sent the most often.

There is no HUGE disk access during a CPU storm.

> 
> Do you see your problem with Tb's -safe-mode? (thunderbird.exe -safe-mode if
> Win).

"thunderbird --safe-mode" in Linux

In Progress.  I am zipping around with the greatest of ease between Sent Boxes in Safe mode.  I will get back in an hour or two.

Comment 24

[Todd]

Safe mode passed.  The only extensions I have are Ligtning 1.9.1, SmtpSelect 1.1.1, and Test Pilot 1.3.9.

I am now testing View, Folders, All.

Comment 25

[Todd]

There does not seem to be any problem opening my Sent boxes with View, Folders, All.  Seems we have isolated this to View, Folders, Unified.

What next?

Comment 26

[WADA:World Anti-bad-Duping Agency]

(In reply to Todd from comment #25)
> Seems we have isolated this to View, Folders, Unified.

You look to forget "no problem with --safe-mode" even with "View, Folders, Unified".
It looks for me that "performance issue with Lightning in Folder View=All" is resolved or reduced in recent releases(Tb, Lightning) but "performance issue with Lightning in Folder View=Unified" is still critical.

> What next?

(A) In daily use, stop using Folder View = Unified,
    or uninstall Lightning if Lightning is cause.

    Because "Unified folder" is same as "Search folder" except that,
    if "Unified", search target folder is automatically maintained by
    Tb and "View Refresh" is perhaps invoked by Tb when new mail is
    added, check can be continued with Search folder of "search target
    == all Sent folders you have" and "condition == any mail". 
(B) Check with newer Tb and Lightning.

FYI.
Tb daily is Tb 23.0aX even in Earlybird. Aurola and comm-central is 24.0ax.
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-earlybird/
However, Lightning Beta supports till Tb 22 only.
> https://addons.mozilla.org/en-US/thunderbird/addon/lightning/versions/
If you do test with newer Tb/Lightning, use candidate build2, please.
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/22.0b1-candidates/build2/linux-x86_64/en-US/

Comment 27

[WADA:World Anti-bad-Duping Agency]

> What next?

(C) Continue problem analysis with Tb 17 and Ligtning 1.9.1.
  Reduce number of mails in each actual Sent folder by moving many mails
  in actual Sent folder to backup folder such as Sent-Backup,
  and execute Compact of each actual Sent folder.
  If problem occurs again, measure time to show Unified "Sent" folder.
  If too long and can't wait, force restart of Tb and reduce number of
  mails in each actual Sent folder.

Comment 28

[Todd]

I only keep six month of mail in my Sent box.

Back in 17.0.6, with all extensions removed and Unified folders turned on, Th starts with a 10 second delay trying to open the Sent box and up to 10 minutes after running for 10 or more hours.  After all the Sent boxes open, then you can maneuver around them with alacrity.  With Ligtning installed, the Sent boxes would never open after running 10 hours.

Now for some good news.  This problem can not be duplicated in ftp://ftp.mozilla.org/pub/thunderbird/candidates/22.0b1-candidates/build2/linux-x86_64/en-US/thunderbird-22.0b1.tar.bz2

Comment 29

[Wayne Mery (:wsmwk)]

Thanks for being persistent Todd.
Ah, the wonders of trying a beta :)
Sorry I didn't suggest it sooner.

not many performance bugs have been fixed since last september https://bugzilla.mozilla.org/buglist.cgi?f10=CP&f1=OP&o3=substring&list_id=6721046&f0=OP&f8=short_desc&v3=perf&o9=allwordssubstr&resolution=FIXED&classification=Client%20Software&classification=Components&o2=anywordssubstr&f9=short_desc&j7=OR&f4=CP&chfieldto=Now&chfield=resolution&query_format=advanced&j1=OR&chfieldfrom=2012-09-01&f3=keywords&f2=short_desc&chfieldvalue=fixed&f11=CP&f5=CP&f6=OP&v2=slow&product=MailNews%20Core&product=Thunderbird&f7=OP&o8=substring

it's a tiny possibility but my guess is bug 750781. Or something gmail related

Comment 30

[Todd]

(In reply to Wayne Mery (:wsmwk) from comment #29)
> Thanks for being persistent Todd.

Oh, this Beta is really nice is a number of ways.  Very quick and snappy.  Love the new customizable tags feature.

Thank you for the Kings treatment!

-T