Closed Bug 1192827 Opened 9 years ago Closed 2 years ago

On multiple win7/win8.1 machines with SSD, completely unresponsive almost all the time, caused by 100k messages in Trash folder and large spam folder. EX. compose locked up for 10-60 seconds. Low CPU.

Categories

(Thunderbird :: Folder and Message Lists, defect)

Unspecified
Windows 8
defect
Not set
major

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: fst, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: perf, Whiteboard: [needs profile])

User Story

Happens in:
* All TB versions (comment 10)
* safe mode (comment 10)

Environment:
* SSD
* largest inbox 2k messages
* ~200gb messages 
* imap server Dovecot, Fedora
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Build ID: 20150806001005

Steps to reproduce:

I have several local email accounts with mllions of emails, nearly 200 GB in total size. Workstations are decent on SSD. Server is high-end, SSD. IMAP, no SSL. Dovecot, Fedora. The problem is with the responsiveness of Thunderbird. My Inbox has been growing for 15 years. Emails are sorted by year in Folders. The folders currently in use do not have excessive amount of emails.




Actual results:

While typing an email (or before being able to do anything), TB is locked up for 10-60 seconds. After regaining consciousness, I am usually able to write a few words before TB locks up again. If that happens, sometimes the main window is activataed after that lockup period for no apparent reason where all the keystrokes made are executed as shortcuts (like manipulation mails and folders) which is very annoying.


Expected results:

Thunderbird should have it's IO operations correctly decoupled from it's UI. As this problem has occured only recently and the inbox has been constantly growing, I think there have been some code changes to the worse in some recent versions.
(In reply to Frederic Steinfels from comment #0)
>...
> As this problem has occured only recently 

Please be more specific with a version number (preferably) and/or date time frame

Also, please verify that the problem does not occur in safe modes...
Please start *Windows'* safe mode with networking enabled
- win7 http://windows.microsoft.com/en-us/windows/start-computer-safe-mode#start-computer-safe-mode=windows-7
Still In Windows safe mode, start thunderbird in safe mode
- http://support.mozillamessaging.com/en-US/kb/safe-mode

Does problem go away?
Flags: needinfo?(fst)
Keywords: perf
Puh... this is an awful lot of work to figure out what version number it occurs. Is there an archive of old versions?

Futhermore I have about 5 work stations and notebooks with different windows versions (7 / 8.1). There are no comparable problems with other software. And the symptoms are the same on all machines.
Flags: needinfo?(fst)
(In reply to Frederic Steinfels from comment #2)
>> Please be more specific with a version number (preferably) and/or date time frame
> Puh... this is an awful lot of work to figure out what version number it
> occurs. Is there an archive of old versions?

check tools | options | advanced | updates | show update history.
Failing that, surely you can estimate a rough date time frame.

> Futhermore I have about 5 work stations and notebooks with different windows
> versions (7 / 8.1). There are no comparable problems with other software.
> And the symptoms are the same on all machines.

Useful info, but that doesn't provide the type of info I seek for comment 1. No need to test all machines, just test one
Flags: needinfo?(fst)
Whiteboard: [closeme 2015-08-25]
Since some days, on Windows 8.1, when returning from Windows sleep mode, TB is regularly completely non responsive. Only closing and restarting TB helps. I haven't waited 60 seconds, and I won't test that.
Whiteboard: [closeme 2015-08-25]
(In reply to Thomas D. from comment #4)
> Since some days, on Windows 8.1, when returning from Windows sleep mode, TB
> is regularly completely non responsive. Only closing and restarting TB
> helps. I haven't waited 60 seconds, and I won't test that.

please file a new bug.
please add stackgrace for the hang
 https://developer.mozilla.org/en-US/docs/How_to_get_a_stacktrace_with_WinDbg
actually, first determine if hang has zero CPU or high CPU. If zero CPU please include stacktrace
I have tested various versions but I could not find one that actually works. So I have to correct myself. TB can not handle very large inboxes, no matter what version I use.
Flags: needinfo?(fst)
(In reply to Frederic Steinfels from comment #7)
> I have tested various versions but I could not find one that actually works.
> So I have to correct myself. TB can not handle very large inboxes, no matter
> what version I use.

You're writing a lot, but missing giving us some important details. For starters...
What happened in windows and thunderbird safe modes?
What version nubmers did you test?  
Your comment 0 implies you've been running Thunderbird for a long period of time - so when was the last version that worked for you? Last year? 

Without sufficient detail there's not much we can do for you.
More facts please - less observations.
Flags: needinfo?(fst)
p.s. how many messages in your largest imap Inbox folders?
So far I have tested 40.0b1 in safe mode and normal mode: lockups
My coworker has tested 
40.0b1/    25-Jul-2015 02:13    
38.2.0/    13-Aug-2015 22:06    
34.0b1/    13-Nov-2014 20:32  
31.6.0/    30-Mar-2015 20:52 
28.0b1/    06-Mar-2014 18:23
24.3.0/    01-Feb-2014 02:20
24.0/    12-Sep-2013 03:46
17.0b2/    31-Oct-2012 00:30
17.0.3/    19-Feb-2013 14:11
13.0/    07-Jun-2012 14:30
10.0/    31-Jan-2012 14:02
3.0.6/    20-Jul-2010 11:12    
No change - lockups.
We will do further tests

Mailbox sizes:
trash has 100k messages
order confirmation box has 300k messages
most other boxes have a few 10k messages

I do have teamviewer ready.
Flags: needinfo?(fst)
> trash has 100k messages
#1 Empty your trash. 


how many messages in your largest imap *Inbox* folders?
Flags: needinfo?(fst)
Largest Inbox folder has 2k messages. Usually I do delete them periodically.

Anyway, Thunderbird should have IO Operations decoupled from UI responsiveness.
Flags: needinfo?(fst)
Is "order confirmation box" an archive folder that is frequently referenced?
After emptying trash, please post with the results.
Flags: needinfo?(fst)
Emptying the Trash improved the Situation.
Flags: needinfo?(fst)
Thanks for the update.
Is this trash folder part of an imap account?
Or is it local?
Severity: normal → major
Flags: needinfo?(fst)
Summary: Thunderbird is completely unresponsive almost all the time → Thunderbird is completely unresponsive almost all the time, caused by 100k messages in Trash folder
It happens on all our workstations. It is an IMAP folder. I have moved that folder out of Maildir because I dont want to really delete it.
Flags: needinfo?(fst)
What I have to add is that the unresponsiveness has been reduced by about 60%. The symptoms mentioned above are still here. There are other folders with 10k-100k messages. I have the impression that thunderbird is checking all folders at a fixed interval. For exmaple, if folders are checked every 30 seconds and the checking takes 25 seconds, tb is almost completely dead.
Suggest you go through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems which lists most common causes of performance issues.
Fredick wrote a couple months ago "its not 40 seconds. its more 15 seconds. but still annoying."

Is that with an empty trash folder?
Flags: needinfo?(fst)
I have just deleted another few 10k mails from trash and spam and the lockups are are much more infrequent and shorter. Before that, I could barely use thunderbird and I had to force close it a 100 times a day. I do not want to empty my trash completely.
Frederic,
What is your mail provider/server?
And what is size of your drafts folder?
And please, when Thunderbird is locked, what is the CPU and memory usage of thunderbird.exe process according to the taskmgr process tab
Blocks: tbbigfolder
OS: Unspecified → Windows 8
Summary: Thunderbird is completely unresponsive almost all the time, caused by 100k messages in Trash folder → Thunderbird is completely unresponsive almost all the time, caused by 100k messages in Trash folder. EX. compose locked up for 10-60 seconds. (still don't know if high or low CPU, high IO or low IO)
Hi Wayne
My server is IMAPS on dovecot on F24. But honestly, it should not matter because any Server IO should be async with the GUI. This is not the case right now. By the way, is there any extension that shows me exactly what thunderbird is doing? Thunderbird is still unresponsive but it currently happens less often and for a shorter time period. If I do open a rather large email, thunderbird locks up as well until the email has been loaded completely.
There are only a few mails in the Drafts folder. For CPU usage, I will check.
Thunderbird is at 0.0 or 0.1% of cpu usage while hanging "spontanously" (probably doing some background batch job). However if I do open a message with 8 MB of text, Thunderbird will use about 33% of cpu power and hang for several minutes. These are two different cases. Both should be fixed. But there should probably two tickets for each one.
> if I do open a message with 8 MB of text, Thunderbird will use about 33% of cpu power and hang for several minutes

Slowness displaying large text messages is pretty well covered in other bug reports and has nothing to do with this. Let's just stick to the general lockup issue here.


>  is there any extension that shows me exactly what thunderbird is doing?

There are multiple tools. For example tools | acvitity manager will show you some message and folder activity. You should check it out. Another is https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Reporting_a_Thunderbird_Performance_Problem_with_G - please use it to get a profile and post the URL here. Please do that BEFORE you do anything I mention below.


> I do not want to empty my trash completely.

Then please move all the messages to an archive folder - get them out of trash.


Did you go through https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems (comment 18)

(again, do the profile first)
User Story: (updated)
Summary: Thunderbird is completely unresponsive almost all the time, caused by 100k messages in Trash folder. EX. compose locked up for 10-60 seconds. (still don't know if high or low CPU, high IO or low IO) → On multiple win7/win8.1 machines with SSD, completely unresponsive almost all the time, caused by 100k messages in Trash folder. EX. compose locked up for 10-60 seconds. Low CPU.
I tried installing the Performance Reporting addon on my thunderbird 49 beta but the buttons do not appear in the status bar. My Profile folder on my local harddrive is 153 GB in 840000 files. I have not moved any messages so far because I had no time to do that. The Activity Manager does not show anything shortly before or after such a 30 second freeze. During the freeze, the activity manager is frozen itself. Recently, after the freeze has stopped, a message does popup: "A script is busy and does not respond. You can stop that script. chrome://messenger/content/folderPane.js:2229". The original intention was most likely to show that requester during the freeze but thunderbird does not seem to be able to open that requester at that time and can only open it after the freeze.
Today I had another "Not responding Scripct": chrome://messenger/content/folderDisplay.js:1449
I've seen similar behaviour, I built trunk today and it too behaved the same way.

The UI freezes for 15-180 seconds, CPU usage is typically 100% of one core.

The Thrash.msf file is 478MB.
Not sure if related, but I've seen massive delays when opening a saved draft with a handful (8+) of very small plaintext attachments. It takes 60+ seconds to open that small and otherwise totally unsuspicious message. The more attachments, the longer it takes. text attachments have only 16 kb each.
(In reply to me from comment #27)
> The Thrash.msf file is 478MB.

And Trash file 823M. So size of folder.msf file is >50% of folder.  This is odd because .msf should be in the 10-25% range. Do you have any custom header filter fields defined? 

usre profile at https://allg.one/MgjR shows ~3% actiivty in folderpane.js and 3% dbViewWrapper.js
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(me)
not sure yet where this belongs
Component: Untriaged → Folder and Message Lists
Depends on: 1507709
Flags: needinfo?(me)
Flags: needinfo?(fst)
Whiteboard: [needs profile]

Frederic (fst) reports "I have verified, thunderbird works ok here now." Dec 2018

(In reply to Thomas D. (:thomas8) | PTO from comment #28)

Not sure if related, but I've seen massive delays when opening a saved draft
with a handful (8+) of very small plaintext attachments. It takes 60+
seconds to open that small and otherwise totally unsuspicious message. The
more attachments, the longer it takes. text attachments have only 16 kb each.

Thomas, does that still reproduce?

Summary: On multiple win7/win8.1 machines with SSD, completely unresponsive almost all the time, caused by 100k messages in Trash folder. EX. compose locked up for 10-60 seconds. Low CPU. → On multiple win7/win8.1 machines with SSD, completely unresponsive almost all the time, caused by 100k messages in Trash folder and large spam folder. EX. compose locked up for 10-60 seconds. Low CPU.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.