1264743 - "Not Responding" for long periods (nsMsgDatabase open takes 4-10 minutes) on startup and checking mail on Windows 7. Large 2gb root folder of 190k messages is from mailman, smartd servivces fed by server side sieve (no TB filters). Compact failed?

Status: RESOLVED WORKSFORME

(MailNews Core :: Database, defect)

Description

[Torkil Svensgaard]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36

Steps to reproduce:

Start the program. I have a rather large profile directory (~26 GB), due to large IMAP accounts, which is part of the problem. I'm currently on 47.0a1 x64 for Windows but the latest x32 bit version had the same issue. I also run Thunderbird on my Linux desktop at work and although it sometimes time out for a few seconds it doesn't stop responding in the same way, even though it has the exact same IMAP accounts configured.


Actual results:

After some seconds it turns "Not Responding" for long periods of time, like 10 minutes or more. 


Expected results:

Program should respond.

Comment 1

[Wayne Mery (:wsmwk)]

Start *Windows'* safe mode with networking enabled
- win10 http://windows.microsoft.com/en-us/windows-10/change-startup-settings-in-windows-10
- win8 http://windows.microsoft.com/en-us/windows-8/windows-startup-settings-safe-mode
- win7 http://windows.microsoft.com/en-us/windows/start-computer-safe-mode#start-computer-safe-mode=windows-7
- XP http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/boot_failsafe.mspx

Still In Windows safe mode, start thunderbird in safe mode
- https://support.mozilla.org/kb/safe-mode-thunderbird

Does problem go away?

Comment 2

[Torkil Svensgaard]

Still "Not Responding" after a short period of time.

Comment 3

[Wayne Mery (:wsmwk)]

Go back to 32 bit - 64 bit isn't likely to help you and it's not fully supported. 

Please next visit the steps at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems and post your results.

Also, please define "a short period of time".
And, what is your largest Inbox (number of messages)?

Comment 4

[Torkil Svensgaard]

Attachment: imap.log

Left it running with log overnight

Comment 5

[Torkil Svensgaard]

Attachment: thunderbird_error_console.png

Screen shot of error console

Comment 6

[Torkil Svensgaard]

Back on 32 bit.

Steps:

1) On newest version yes
2) Safe mode didn't help (had disabled extensions etc. already)
3) Windows safe mode with networking didn't help
4) Not subscribed to "All Mail"
5) No newsgroups
6) Disabled all indexing, didn't help
7) Default profile location (on SSD)
8) Log attached
9) Globat indexing disabled
10) Renamed global-messages-db.sqlite. Old was 3 GB, new is 1.200 KB, no change.
11) Several folders are in the 1-2 GB range. 
12) Changed check interval to 5 mins, no change
13) I only have IMAP folders
14) I have 552 contacts
15) Panacea.dat is 48K
16) Don't think it's a script causing it, but screen shot of error console attached
17) Not using particularly high CPU, steady 13% for thunderbird.exe on a core i7 920. Updater not running.
18) All on system SSD
19) No RSS
20) Core i7 with 12GB RAM, no hardware problems I know of (and no problems with the system otherwise). Two of my inboxes have ~30.000 messages, but that isn't a problem on Ubuntu.
21) Windows 7
22) No real match in references I think.

Short period of time = within one minute. Largest inbox is ~30.000 messages.

Comment 7

[Torkil Svensgaard]

Turned of checking for new messages at startup for all accounts and it worked fine until i manually checked for new messages = freeze within one minute.

Comment 8

[Kent James (:rkent)]

It's hard to make sense of a large log with no context. What would be useful is to correlate times in the log with the observed behavior in the bug, like "freeze occurred at 2016-04-14 22:38:19.292000 in the log."

Comment 9

[Torkil Svensgaard]

Ok, I'll attach new log in a minute, with the following chronology:

21:11:12 Started program, pressed get all messages and started typing a new message
21:12:35 Noticed program lockup while typing
21.16.30 Noticed program responded again

From the log:
2016-04-20 21:12:32.312000 UTC - 0[2611140]: nsMsgDatabase::Open(C:\Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's

That folder has a size of 2.16 GB from 190.283 messages and I'm now compacting it to see if that makes any difference.

Comment 10

[Torkil Svensgaard]

Attachment: imap.log

Comment 11

[Tim Johnson]

I am seeing a similar issue of protracted periods of "Not Responding" behavior. I'm using 64 bit Win 10 (ver 1511) with latest updates, and TB 45.2.0. This has been happening for over half a year, and seems to be getting worse. 

I see it most often after I've deleted some messages, or requested the Junk or Trash folders to be emptied. I have automatic compacting on and set to compact with over 20 MB would be saved. When going into or out of the Not Responding period, I have observed a very brief flash of the TB window.

I have the following extensions: Dictionary Search, FoxClocks, Lightning, Provider for Google Calendar, Mail Merge, QuoteCollapse and StartupMaster. Except for Lightning, Provider for Google Calendar and StartupMaster, these extensions have been there for years. I have the calendar set to refresh every 30 minutes.

On my wife's 64 bit Win 7 computer, with the same extensions, we see this Not Responding behavior infrequently.

Comment 12

[Tim Johnson]

Oh yes, my e-mail accounts are all pop3.

Comment 13

[Tim Johnson]

I've been running TB in safe mode (extensions temporarily disabled). This issue still persists, although the duration of the hang is considerably shorter. I'll run like this for the rest of the day, then start adding back extensions to see if there is one which brings back the much longer hang.

Comment 14

[Kent James (:rkent)]

Lightning is known to cause this issue in some cases. In theory THunderbird has lots of activities that might cause a hang, but they are surprisingly rare in my experience.

Then again, when I travel I run Thunderbird on a Microsoft Surface tablet, and I expect and see a lot of hangs there.

I think that we realize that there are way too many places where Thunderbird does sync processing on the IU thread. IMAP uses its own thread, POP3 does not. It's amazing that there are not more reports like this.

Recommendation? Disable Lightning and see if that is the issue. If not, if you can find a reproducible case where you can cause a hang in Thunderbird, we might be able to devise a kludge to stop that case. But there are underlying architectural issues, which to some extent we share with Firefox. They have been working for about 5 years on this, and I believe are now slowly starting to role out e10s. So this is not a simple change to fix.

Comment 15

[Tim Johnson]

With all my extensions disabled, I had very few hands, and when I got them, they were very short.

I enabled all extensions except Lightning and Provider for Google Calendar and for the next day did not see a single hang. 

Yesterday, I enabled Lightning and Provider for Google Calendar as well, and after over a day of running, I have not had a single hang. Go figure. If the hangs come back I'll post again.

This does smell like a thread race condition with timeouts to keep the race from going indefinitely. Why it has seemed to settle out for me is hard to explain.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Torkil Svensgaard from comment #9)
> Ok, I'll attach new log in a minute, with the following chronology:
> 
> 21:11:12 Started program, pressed get all messages and started typing a new
> message
> 21:12:35 Noticed program lockup while typing
> 21.16.30 Noticed program responded again
> 
> From the log:
> 2016-04-20 21:12:32.312000 UTC - 0[2611140]:
> nsMsgDatabase::Open(C:
> \Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.
> default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
> 2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's
> 
> That folder has a size of 2.16 GB from 190.283 messages and I'm now
> compacting it to see if that makes any difference.

4 minutes is a long time to load from SSD what is probably a ~300MB .msf index file. However, 190k is a lot of messages. 

Torkil,
1. Do many of the messages in that file have the same subject?
2. why would Thunderbird be opening that file?

Comment 17

[Wayne Mery (:wsmwk)]

Torkil,...


(In reply to Torkil Svensgaard from comment #5)
> Screen shot of error console

Thanks. unfortunately nothing illuminating there


What did you find after compacting the folder Admin-1.sbd\root?
And is that a folder of server log files?


In reply to Torkil Svensgaard from comment #10)
> Created attachment 8743533 [details]
> imap.log

2016-04-20 21:12:32.312000 UTC - 0[2611140]: nsMsgDatabase::Open(C:\Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's

There is a 4 minute gap between those two log lines.  It seems to me there should have nearly the same time stamp.  Does that 4 minute gap match your delay time?

Comment 18

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #16)
> (In reply to Torkil Svensgaard from comment #9)
> > Ok, I'll attach new log in a minute, with the following chronology:
> > 
> > 21:11:12 Started program, pressed get all messages and started typing a new
> > message
> > 21:12:35 Noticed program lockup while typing
> > 21.16.30 Noticed program responded again
> > 
> > From the log:
> > 2016-04-20 21:12:32.312000 UTC - 0[2611140]:
> > nsMsgDatabase::Open(C:
> > \Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.
> > default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
> > 2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's
> > 
> > That folder has a size of 2.16 GB from 190.283 messages and I'm now
> > compacting it to see if that makes any difference.
> 
> 4 minutes is a long time to load from SSD what is probably a ~300MB .msf
> index file. However, 190k is a lot of messages. 
> 
> Torkil,
> 1. Do many of the messages in that file have the same subject?

Yes, lots of emails from root accounts from various machines, with identical subjects.

> 2. why would Thunderbird be opening that file?

Not sure, as I've un-ticked "When getting new messages for this account, always check this folder"

Comment 19

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #17)
> What did you find after compacting the folder Admin-1.sbd\root?
> And is that a folder of server log files?

Problem went away, pretty much, after compacting, IIRC. It's various service emails from mailman, smartd etc.

> In reply to Torkil Svensgaard from comment #10)
> > Created attachment 8743533 [details]
> > imap.log
> 
> 2016-04-20 21:12:32.312000 UTC - 0[2611140]:
> nsMsgDatabase::Open(C:
> \Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.
> default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
> 2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's
> 
> There is a 4 minute gap between those two log lines.  It seems to me there
> should have nearly the same time stamp.  Does that 4 minute gap match your
> delay time?

I guess so, as I wrote this in another comment:

21:11:12 Started program, pressed get all messages and started typing a new message
21:12:35 Noticed program lockup while typing
21.16.30 Noticed program responded again

I've since archived a lot of emails to speed things up (yearly folders) so my setup is not the same anymore.

Comment 20

[Wayne Mery (:wsmwk)]

Thanks!  A couple more questions embeded below...


(In reply to Torkil Svensgaard from comment #18)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #16)
> > (In reply to Torkil Svensgaard from comment #9)
> > > Ok, I'll attach new log in a minute, with the following chronology:
> > > 
> > > 21:11:12 Started program, pressed get all messages and started typing a new
> > > message
> > > 21:12:35 Noticed program lockup while typing
> > > 21.16.30 Noticed program responded again
> > > 
> > > From the log:
> > > 2016-04-20 21:12:32.312000 UTC - 0[2611140]:
> > > nsMsgDatabase::Open(C:
> > > \Users\torkil\AppData\Roaming\Thunderbird\Profiles\2z29axja.
> > > default\ImapMail\drcmr.dk\Admin-1.sbd\root.msf, FALSE, 81288d0, FALSE)
> > > 2016-04-20 21:16:28.819000 UTC - 0[2611140]: 13 open DB's
> > > 
> > > That folder has a size of 2.16 GB from 190.283 messages and I'm now
> > > compacting it to see if that makes any difference.
> > 
> > 4 minutes is a long time to load from SSD what is probably a ~300MB .msf
> > index file. However, 190k is a lot of messages. 

> > Torkil,
> > 1. Do many of the messages in that file have the same subject?
> 
> Yes, lots of emails from root accounts from various machines, with identical
> subjects.

good info - identical subjects in the UI can be a performance issue. also good - is that unless you actually clicked the folder, it does not present a problem.

  
> > 2. why would Thunderbird be opening that file?
> 
> Not sure, as I've un-ticked "When getting new messages for this account,
> always check this folder"

Oh. Getting new mail in the folder with TB filters or "always check this folder" explains why it was opening, even though you didn't click on the folder.

Is/was that folder populated by a server-side filter?  How recently did you untick that option? 
Are you using Thunderbird filters now to categorize those mails?

And do you have "check this folder" set for any other folders? 


> I've since archived a lot of emails to speed things up (yearly folders) so my setup is not the same anymore.
> Problem went away, pretty much, after compacting.

You say "pretty much". Does that mean you still see some delays?  And how many messages in the folder now?


I'm still a bit stumped - you have SSD so opening even a 200mb msf index file shouldn't cause minutes of delay - because we can rule out dozens of possible causes using your comment 6. Unless "check this folder" was still enabled at the time.  

Not to mention, that you say the linux system didn't have this problem. So I'll put it out there even though you said you still had trouble in WIndows safe mode, that this kind of extreme/patholigical performance degradation is what we sometimes see when Windows-based antivirus programs are involved.

Comment 21

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #20)
> Thanks!  A couple more questions embeded below...
> 
   
> > > 2. why would Thunderbird be opening that file?
> > 
> > Not sure, as I've un-ticked "When getting new messages for this account,
> > always check this folder"
> 
> Oh. Getting new mail in the folder with TB filters or "always check this
> folder" explains why it was opening, even though you didn't click on the
> folder.

I meant it was un-ticked at the time (and still is), so no idea why the folders was checked.
 
> Is/was that folder populated by a server-side filter?  How recently did you
> untick that option? 
> Are you using Thunderbird filters now to categorize those mails?

Server side by sieve on the IMAP server. I'm not using Thunderbird filters

> And do you have "check this folder" set for any other folders? 

No.
 
> > I've since archived a lot of emails to speed things up (yearly folders) so my setup is not the same anymore.
> > Problem went away, pretty much, after compacting.
> 
> You say "pretty much". Does that mean you still see some delays?  And how
> many messages in the folder now?

Still delays sometimes, but in the less than a minute range. The folder contains about 47.000 emails currently but, perhaps more important, size on disk is ~200 mb. As I wrote earlier, I think the folder was 2.16 gb before I compacted it, and afterwards it was much smaller.
 
> I'm still a bit stumped - you have SSD so opening even a 200mb msf index
> file shouldn't cause minutes of delay - because we can rule out dozens of
> possible causes using your comment 6. Unless "check this folder" was still
> enabled at the time.  

It wasn't

> Not to mention, that you say the linux system didn't have this problem. So
> I'll put it out there even though you said you still had trouble in WIndows
> safe mode, that this kind of extreme/patholigical performance degradation is
> what we sometimes see when Windows-based antivirus programs are involved.

The linux system still/also have small delays, but also in the less than a minute range, and subjectively much shorter delays than on Windows.

Comment 22

[Wayne Mery (:wsmwk)]

> The folder contains about 47.000 emails currently but, perhaps more important, size on disk is ~200 mb. As I wrote earlier, I think the folder was 2.16 gb before I compacted it, and afterwards it was much smaller.

So it got much smaller than 2.1gb after the compact, but before you archived messages out of it??

If so, then automatic compact was failing on that folder.

Comment 23

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #22)
> > The folder contains about 47.000 emails currently but, perhaps more important, size on disk is ~200 mb. As I wrote earlier, I think the folder was 2.16 gb before I compacted it, and afterwards it was much smaller.
> 
> So it got much smaller than 2.1gb after the compact, but before you archived
> messages out of it??
> 
> If so, then automatic compact was failing on that folder.

Exactly. That sounds plausible.

Comment 24

[Wayne Mery (:wsmwk)]

> Still delays sometimes, but in the less than a minute range. The folder contains about 47.000 emails currently but, perhaps more important, size on disk is ~200 mb

If you keep the folder open in a separate tab (you don't have to keep it in the foreground) does it impact the delays you see?

Comment 25

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #24)
> > Still delays sometimes, but in the less than a minute range. The folder contains about 47.000 emails currently but, perhaps more important, size on disk is ~200 mb
> 
> If you keep the folder open in a separate tab (you don't have to keep it in
> the foreground) does it impact the delays you see?

I've had it on a separate tab for a couple of days, seems about the same.

Comment 26

[Mike]

Have same problem as Tim Johnson (comment 11)  Windows Ver1607 64 bit
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.7.1 
TB handles mix of POP3, IMAP, RSS and NNTP accounts

Not responding states seem to last a minute or so before system comes back to life and repeat frequently, say every 4 or so minutes. Most often happens whilst composing a new mail message editor, responding to an email downloaded from pop3 server.  UI in complete non-responding state.  Despite so many good features, this is making Thunderbird almost unworkable, just about to throw in the towel.

Concurrent with issue, the error console has reported:
Error: uncaught exception: 2147746065
Source File: resource:///modules/activity/autosync.js
Line: 210

Kent James message 14 seems pertinent:
"I think that we realize that there are way too many places where Thunderbird does sync processing on the IU thread. IMAP uses its own thread, POP3 does not. It's amazing that there are not more reports like this."

Comment 27

[Wayne Mery (:wsmwk)]

Torkil,
Do you have account settings for delete set to IMAP delete model (Mark as Deleted)?

Comment 28

[Torkil Svensgaard]

It's set to move to a trash folder on my Windows, probably the same on my Linux at work

Comment 29

[Wayne Mery (:wsmwk)]

(In reply to Mike from comment #26)
> Have same problem as Tim Johnson (comment 11)  Windows Ver1607 64 bit
> Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
> Thunderbird/45.7.1 
> TB handles mix of POP3, IMAP, RSS and NNTP accounts

(In reply to Tim Johnson from comment #11)
> I am seeing a similar issue of protracted periods of "Not Responding"
> behavior. I'm using 64 bit Win 10 (ver 1511) with latest updates, and TB
> 45.2.0. This has been happening for over half a year, and seems to be
> getting worse. 
> 

You may be seeing similar symptoms, but unless you have very large folders it is unlikely the cause is the same.  There is a litany of possible causes - too many to list here, but listed at https://wiki.mozilla.org/Thunderbird:Testing:Memory_Usage_Problems  For example check your junk and trash folder sizes.

1. you must eliminate all other software as potential cause, per comment 1.
2. RSS is a possible problem - turn off RSS checking at startup and for intervals. If that helps, with RSS checking try the nightly build which has some changes https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ (backup your Thunderbird profile first.

Comment 30

[Wayne Mery (:wsmwk)]

(In reply to Torkil Svensgaard from comment #28)
> It's set to move to a trash folder on my Windows, probably the same on my
> Linux at work

Cool. What's clear is the problem is related to that "root" folder and perhaps others like it. What isn't clear, is whether it was device-related, Thunderbird compact repeatedly failing or some mork-related issue (compress commit failing), or something else.

If you don't use global search against large folders like "root" you can save yourself some overhead by disabling global search indexing on those high volume folders (n.b. comment 6 item 10 where index size dropped by 2/3) by installing glodaquilla addon and set those specific folder properties to not index.

Comment 31

[Wayne Mery (:wsmwk)]

(In reply to Mike from comment #26)
> Have same problem as Tim Johnson (comment 11)  Windows Ver1607 64 bit
> Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
> Thunderbird/45.7.1 
> TB handles mix of POP3, IMAP, RSS and NNTP accounts
> 
> Not responding states seem to last a minute or so before system comes back
> to life and repeat frequently, say every 4 or so minutes. Most often happens
> whilst composing a new mail message editor, responding to an email
> downloaded from pop3 server.  UI in complete non-responding state.  Despite
> so many good features, this is making Thunderbird almost unworkable, just
> about to throw in the towel.
> 
> Concurrent with issue, the error console has reported:
> Error: uncaught exception: 2147746065
> Source File: resource:///modules/activity/autosync.js
> Line: 210
> 
> Kent James message 14 seems pertinent:
> "I think that we realize that there are way too many places where
> Thunderbird does sync processing on the IU thread. IMAP uses its own thread,
> POP3 does not. It's amazing that there are not more reports like this."

The autosync error may be ignored.

Mike, disable checking for new messages in RSS and NNTP. What are your results?   If that helps, please try beta from http://www.mozilla.org/en-US/thunderbird/channel/ with those still disabled, and then again with them enabled. What are your results?

Comment 32

[Mike]

(In reply to Wayne Mery (:wsmwk, NI for questions) from comment #31)
> (In reply to Mike from comment #26)
> > Have same problem as Tim Johnson (comment 11)  Windows Ver1607 64 bit
> > Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
> > Thunderbird/45.7.1 
> > TB handles mix of POP3, IMAP, RSS and NNTP accounts
> > 
> > Not responding states seem to last a minute or so before system comes back
> > to life and repeat frequently, say every 4 or so minutes. Most often happens
> > whilst composing a new mail message editor, responding to an email
> > downloaded from pop3 server.  UI in complete non-responding state.  Despite
> > so many good features, this is making Thunderbird almost unworkable, just
> > about to throw in the towel.
> > 
> > Concurrent with issue, the error console has reported:
> > Error: uncaught exception: 2147746065
> > Source File: resource:///modules/activity/autosync.js
> > Line: 210
> > 
> > Kent James message 14 seems pertinent:
> > "I think that we realize that there are way too many places where
> > Thunderbird does sync processing on the IU thread. IMAP uses its own thread,
> > POP3 does not. It's amazing that there are not more reports like this."
> 
> The autosync error may be ignored.
> 
> Mike, disable checking for new messages in RSS and NNTP. What are your
> results?   If that helps, please try beta from
> http://www.mozilla.org/en-US/thunderbird/channel/ with those still disabled,
> and then again with them enabled. What are your results?

Hi Wayne,  thanks for suggestion.  I have got further in finding out that the underlying cause of my problem is triggered following a send of email  (usually it is the second email in a session).  If I look in a TCP analyser, I see that a TCP connection from Thunderbird to the smtp server involved in the send remains in a CLOSE_WAIT state and the user interface to Thunderbird UI goes into a "not responding" state until this CLOSE_WAIT connection gets closed which seems to take 2 minutes which possibly relates to TCP standards on timeout time
https://blogs.technet.microsoft.com/janelewis/2010/03/09/explaining-close_wait/
http://www.tcpipguide.com/free/t_TCPConnectionTermination-3.htm

I am not sure what triggers the close wait issue between Thunderbird and the smtp server.  It does not seem to happen with every smtp send but if it is possible to remove the dependency on the Thunderbird UI on this TCP connection closure and or if the 120 sec timeout could be shortened down, that would be brilliant as the user could continue instead of need to stall for 2 minutes.
cheers Mike

Comment 33

[Wayne Mery (:wsmwk)]

Torkil, 

Do you still see this issue with version 52? And if yes, does it also happen with current beta from http://www.mozilla.org/en-US/thunderbird/channel/ ?

Comment 34

[Wayne Mery (:wsmwk)]

Mike, 

Do you still see this issue with version 52? And if yes, does it also happen with current beta from http://www.mozilla.org/en-US/thunderbird/channel/ ?

Comment 35

[Mike]

I haven't seen the problem for a week or so now, being stuck with TCP CLOSE_WAIT state.  
Currently on     
Name: Thunderbird
    Version: 52.6.0
    User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0

Great if it is fixed,

Mike

Comment 36

[Wayne Mery (:wsmwk)]

So, bug 1157256 reported in 2015, prior to this bug, and just fixed may have played a role in reporter's issue - Automatic compacting folders not working in online mode. IMAP users not prompted for compact."

However, compact may have simply removed the conditions - I suspect reporter's issue is more related to large folder particularly those we have seen reported by others who store large numberes of logfile messages in imap, like bug 589310 / bug 588952.  Plus other factors may be at play, such as anything in https://mzl.la/2JKrq86


(In reply to Torkil Svensgaard from comment #21)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #20)
>    
> > > > 2. why would Thunderbird be opening that file?
> > > 
> > > Not sure, as I've un-ticked "When getting new messages for this account,
> > > always check this folder"
> > 
> > Oh. Getting new mail in the folder with TB filters or "always check this
> > folder" explains why it was opening, even though you didn't click on the
> > folder.
> 
> I meant it was un-ticked at the time (and still is), so no idea why the
> folders was checked.

Do you (or did you) have hidden preference  mail.check_all_imap_folders_for_new  set to true?  (either on linux or Windows)

Comment 37

[Wayne Mery (:wsmwk)]

(In reply to Mike from comment #35)
> I haven't seen the problem for a week or so now, being stuck with TCP CLOSE_WAIT state.  

That's good news.  So going from 45 to 52 helped?  

If you see the problem again you might look at open smtp bug reports that mention TCP or CLOSE_WAIT https://mzl.la/2LxodgQ

Comment 38

[Wayne Mery (:wsmwk)]

(In reply to Torkil Svensgaard from comment #18)
> (In reply to Wayne Mery (:wsmwk, NI for questions) from comment #16)
> > (In reply to Torkil Svensgaard from comment #9)
> > >...
> > > That folder has a size of 2.16 GB from 190.283 messages and I'm now
> > > compacting it to see if that makes any difference.
> > 
> > 4 minutes is a long time to load from SSD what is probably a ~300MB .msf
> > index file. However, 190k is a lot of messages. 
> > 
> > Torkil,
> > 1. Do many of the messages in that file have the same subject?
> 
> Yes, lots of emails from root accounts from various machines, with identical
> subjects.

Which are 
Bug 226730 - Slow threading when all messages have the same subject, with mail.strict_threading
Bug 452221 - When many mails have the same subject (e.g. very long thread), time to take Shift+Delete (not sure if this is applicable)
Bug 452944 - TB uses 1.3 GB RAM with sort criteria Junk yes/no and subject and big inbox

Comment 39

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk) from comment #33)
> Torkil, 
> 
> Do you still see this issue with version 52? And if yes, does it also happen
> with current beta from http://www.mozilla.org/en-US/thunderbird/channel/ ?

Sorry for being tardy. Still seeing an issue with the same or worse symptoms. Currently on version 52.8.0.

Mvh.

Torkil

Comment 40

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk) from comment #36)
> Do you (or did you) have hidden preference 
> mail.check_all_imap_folders_for_new  set to true?  (either on linux or
> Windows)

Not at the moment, and I'm certain I've never enabled it myself, but I don't know if it could have been set automagically at some point.

Mvh.

Torkil

Comment 41

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk) from comment #30)
> If you don't use global search against large folders like "root" you can
> save yourself some overhead by disabling global search indexing on those
> high volume folders (n.b. comment 6 item 10 where index size dropped by 2/3)
> by installing glodaquilla addon and set those specific folder properties to
> not index.

Thanks, will try that. That folder currently have about ~130.000 emails, I guess that's probably still the culprit. The issue does not crop up consistently but once in a while.

Mvh.

Torkil

Comment 42

[Wayne Mery (:wsmwk)]

(In reply to Mike from comment #35)
> I haven't seen the problem for a week or so now, being stuck with TCP
> CLOSE_WAIT state.  
> Currently on     
> Name: Thunderbird
>     Version: 52.6.0
>     User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
> Thunderbird/52.6.0
> 
> Great if it is fixed,
> 
> Mike

Yes, that would be great.  Please let us know if the condition returns

Comment 43

[Wayne Mery (:wsmwk)]

(In reply to Torkil Svensgaard from comment #40)
> I'm certain I've never enabled mail.check_all_imap_folders_for_new  myself, but I don't
> know if it could have been set automagically at some point.

unlikely :)


(In reply to Torkil Svensgaard from comment #41)
> (In reply to Wayne Mery (:wsmwk) from comment #30)
> > ...
> > save yourself some overhead by disabling global search indexing on those
> > high volume folders (n.b. comment 6 item 10 where index size dropped by 2/3)
> > by installing glodaquilla addon and set those specific folder properties to
> > not index.
> 
> Thanks, will try that. That folder currently have about ~130.000 emails, I
> guess that's probably still the culprit. The issue does not crop up
> consistently but once in a while.

How is that process going?

Comment 44

[Wayne Mery (:wsmwk)]

Torkil seems to be gone and Mike is good, so => incomplete

Comment 45

[Mike]

Looking good, I confirm I haven't noticed this problem for quite a while so hopefully fixed which is great, many thanks. Currently on V60.4.0 (32-bit)

Comment 46

[Tim Johnson]

I'm not sure I'm seeing the same problem as before (Thunderbird not responding notice on the title bar), but I definitely am seeing hangs after I've had Thunderbird running for 4-5 days, on average. The symptom I see now is the GUI is completely unresponsive. If I try to look at a different folder, or move to a different note, nothing happens. No title bar notice of not responding, just nothing. At this point, navigation with either the mouse or keyboard does not work. The only thing left is to kill Thunderbird, and restart it.

Is there an other bug that fits these symptoms? If so, I'll make further comments there.

Comment 47

[gene smith]

(In reply to Tim Johnson from comment #46)

I'm not sure I'm seeing the same problem as before (Thunderbird not responding notice on the title bar), but I definitely am seeing hangs after I've had Thunderbird running for 4-5 days, on average.

Do you sleep or hibernate the computer during this time or just leave it running normally? Does a screensaver occur during this time?

The symptom I see now is the GUI is completely unresponsive. If I try to look at a different folder, or move to a different note, nothing happens. No title bar notice of not responding, just nothing. At this point, navigation with either the mouse or keyboard does not work. The only thing left is to kill Thunderbird, and restart it.

So TB is just frozen and never responds no matter how long you wait? Kill with task mgr I assume?

Is there an other bug that fits these symptoms? If so, I'll make further comments there.

Note sure, but sounds different since this seems like a permanent hang.

Comment 48

[Tim Johnson]

Gene, to answer your questions:

Yes, I do hibernate the computer over night. No screen saver during hibernation, but during the day a screen saver does activate several times through the day. Most of the time this does not seem to affect Thunderbird's responsiveness.

When TB is frozen like I described, it <<may>> respond after perhaps 5-10 minutes. That's a bit more patience than I think most users would put up with. When I kill it, it is by clicking the X on the upper right of the window frame. I then watch the task manager to make sure TB is gone before restarting it.

Comment 49

[Torkil Svensgaard]

(In reply to Wayne Mery (:wsmwk) from comment #44)

Torkil seems to be gone and Mike is good, so => incomplete

I'm good. Still small freezes but in the less that 10s range.

Thanks,

Torkil

Comment 50

[Wayne Mery (:wsmwk)]

(In reply to Tim Johnson from comment #48)

Gene, to answer your questions:
...
When TB is frozen like I described, it <<may>> respond after perhaps 5-10 minutes. That's a bit more patience than I think most users would put up with. When I kill it, it is by clicking the X on the upper right of the window frame. I then watch the task manager to make sure TB is gone before restarting it.

Tim, do you still see this problem?

Comment 51

[Tim Johnson]

(In reply to Wayne Mery (:wsmwk) from comment #50)

(In reply to Tim Johnson from comment #48)

Gene, to answer your questions:
...
When TB is frozen like I described, it <<may>> respond after perhaps 5-10 minutes. That's a bit more patience than I think most users would put up with. When I kill it, it is by clicking the X on the upper right of the window frame. I then watch the task manager to make sure TB is gone before restarting it.

Tim, do you still see this problem?

Wayne, no I have not seen this for a very long time.

Comment 52

[Wayne Mery (:wsmwk)]

Thanks for the updates.
Based on comment 49, => WORKSFORME

Comment 53

[gene smith]

Just a comment about these issues of slow down or not responding with big folders: At least one user solved it by enabling mail.server.default.use_condstore = true. This avoids having to download and analyze a huge list of message flags when folders are accessed or updated.
When condstore imap feature was added, it caused a problem with gmail so it was defaulted to false. I'm pretty sure the issue no longer exists so condstore set true and then tb restart can be helpful. (Of course the imap server must also support it and most now do.)