781304 - Very slow when large folders don't have a summary/index .msf after manual move outside of Thunderbird

Status: UNCONFIRMED

(MailNews Core :: Database, defect)

Description

[Konstantin Svist]

I accidentally moved some large mail folders around a few days ago and Tb has become really slow. And not just switching from folder to folder, but switching to Tb window, opening right-click menus, even displaying "are you sure you want to delete junk" popup was really slow. Meanwhile, JS console is printing out a lot of these messages on pretty much every action:

Timestamp: 08/08/2012 11:49:13 AM
Error: [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIMsgFolder.getStringProperty]"  nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"  location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2468"  data: no]
Source File: chrome://messenger/content/folderPane.js
Line: 2470


After several false starts, I ran strace and saw that one large folder in particular was being accessed while running a simple test case of trying to empty junk (Junk folder is already empty and compacted). Specifically:

access("[...]stuff.msf", F_OK) = 0
stat64("[...]stuff.msf", {st_mode=S_IFREG|0664, st_size=11345763, ...}) = 0
open("[...]stuff.msf", O_RDWR) = 27
fstat64(27, {st_mode=S_IFREG|0664, st_size=11345763, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb0c15000
_llseek(27, 0, [0], SEEK_SET)           = 0
read(27, "// <!-- <mdb:mork:z v=\"1.4\"/> --"..., 4096) = 4096
fstat64(27, {st_mode=S_IFREG|0664, st_size=11345763, ...}) = 0
_llseek(27, 11341824, [11341824], SEEK_SET) = 0
read(27, ")>[6CB3BD48:^80(^D5^2788E)]\n[6CB"..., 4096) = 3939
_llseek(27, 0, [0], SEEK_SET)           = 0
read(27, "// <!-- <mdb:mork:z v=\"1.4\"/> --"..., 4096) = 4096
fstat64(27, {st_mode=S_IFREG|0664, st_size=11345763, ...}) = 0
_llseek(27, 11341824, [11341824], SEEK_SET) = 0
read(27, ")>[6CB3BD48:^80(^D5^2788E)]\n[6CB"..., 4096) = 3939
_llseek(27, 0, [0], SEEK_SET)           = 0
read(27, "// <!-- <mdb:mork:z v=\"1.4\"/> --"..., 4096) = 4096
_llseek(27, 4096, [4096], SEEK_SET)     = 0
read(27, ")(^91=0)(^93=1)]\n<(19CDC=48da3a3"..., 4096) = 4096
_llseek(27, 8192, [8192], SEEK_SET)     = 0
read(27, "(^91=0)(^93=1)]\n<(19E29=490c6d5b"..., 8192) = 8192

... and so on with _llseek/read for a pretty long time.

At this point I opened up the folder and Tb rebuilt the summary/index by itself (I guess it had become corrupted while I was moving it around?)

My specific problem is fixed now, but I can't help but feel there should've been some warning to the user, saying "Folder so-and-so has a broken index, Tb may be slow until you open/fix it".

Comment 1

[Konstantin Svist]

P.S. The JS console error is still present and appears by the dozen on pretty much every action (e.g. focusing the main Tb window). It doesn't seem to be causing any slowness, but probably needs to be fixed somehow, anyway. Do I need to file another bug for that?

Comment 2

[:aceman]

Try to find out which folder causes that error message and then Right click it -> Properties -> Repair folder.

Comment 3

[Konstantin Svist]

As I said, I already did. The bug is that I had no cue from Thunderbird that it's slow because some folder has a broken index. If you print a message to the Error Console saying something like "Folder NNN has a broken index - Thunderbird may work slower than usual until you repair it", I'll consider the bug successfully resolved.

Comment 4

[:aceman]

In comment 1 you say you still get the JS console error.

Comment 5

[Konstantin Svist]

I had assumed that the error is unrelated since the slowness went away...
Just now I repaired about 20 folders and the errors went away for a minute - but then they came back.
I have >70 folders and don't want to go through them one by one

Comment 6

[Konstantin Svist]

Is there a way to modify folderPane.js without recompiling? Perhaps stick it somewhere into .thunderbird/[profile]/chrome/... ?

Comment 7

[:aceman]

You can find it in the TB install folder, in the file called omni.ja. You can extract/open it as a ZIP file and modify the folderPane.js directly there. Then before starting TB you must remove the XUL/JS cache in .thunderbird/profile/startupCache/ .

Comment 8

[Konstantin Svist]

Thanks, that worked.

I modified folderPane.js as follows:

2470c2470
<     Components.utils.reportError(ex);
---
>     Components.utils.reportError(ex + " *** Offending folder: " + aFolder.name);

The result: error refers to folders "Local Folders" and "kostya@mireille" (which is a Unix Movemail account, can't merge it with "Local Folders"...)
Neither one has an .msf and neither one can be repaired...

Comment 9

[:aceman]

Those seem to be the account names. But the folders under those should have a .msf and should have the repair option.

Comment 10

[Konstantin Svist]

Just now I messed up an index of one folder (deleted a bunch of lines) and watched the console -- the folder's name came up in the error message. After rebuilding that folder's .msf, its name no longer came up in the console.
This means that the error shows up for folders with corrupted .msf files as well as for account folders (which have no corresponding .msf files).

Sounds to me like the error message from account folders is not caused by .msf corruption...

Comment 11

[Wayne Mery (:wsmwk)]

Konstantin, is this in unified folder view?

Comment 12

[Konstantin Svist]

Do you mean delivering everything into "Local Folders"? If so, then yes.

Comment 13

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk) from comment #11)
> Konstantin, is this in unified folder view?

No, I mean that View | Folders  indicates "Unified Folders", not "All Folders".

ALso, is your thunderbird profile or mail storage on a network/folder share?

Comment 14

[Konstantin Svist]

No, I'm using "All folders"
Also, storage is local HDD

Comment 15

[Wayne Mery (:wsmwk)]

(In reply to :aceman from comment #9)
> Those seem to be the account names. But the folders under those should have
> a .msf and should have the repair option.

I read an old bug comment today, bienvenu's I think, that indicated there may be conditions where the repair function does not work, or isnt' offered.

Comment 16

[Wayne Mery (:wsmwk)]

(In reply to Konstantin Svist from comment #0)
> I accidentally moved some large mail folders around a few days ago and Tb
> has become really slow. And not just switching from folder to folder, but
> switching to Tb window, opening right-click menus, even displaying "are you
> sure you want to delete junk" popup was really slow. 

Konstantin
Was this pop or imap?
And was folder moved in the Thunderbird UI? Or did you move using the file system?

Comment 17

[Konstantin Svist]

pop
yes, moved in TB UI

Comment 18

[Konstantin Svist]

I experienced this issue again for a couple of months, confusing it with #733039
I finally run "Repair Folder" for each of my >30 TB folders and the problem went away.
Please consider adding "Repair All Folders" functionality -- and maybe some detection of the problem condition

Comment 19

[Worcester12345]

(In reply to Konstantin Svist from comment #18)
> I experienced this issue again for a couple of months, confusing it with
> #733039
> I finally run "Repair Folder" for each of my >30 TB folders and the problem
> went away.
> Please consider adding "Repair All Folders" functionality -- and maybe some
> detection of the problem condition

I second this. It should not require people to seek out and "repair" things which should just naturally be working. If it were easier to do all at once, this would be a step in the right direction. Not having them go into disrepair would be best.

Comment 20

[Wayne Mery (:wsmwk)]

Aceman, 
What approaches are possible here to mitigate the impact of this, without adding user accessible features?
Or we stuck with saying we don't support users moving folders outside of Thunderbird?

Comment 21

[Wayne Mery (:wsmwk)]

Possibly related?
bug 682725
bug 261419

Comment 22

[Wayne Mery (:wsmwk)]

Konstantin,
Do you still see this when using version 102 from https://www.thunderbird.net/en-US/ ?

If you do, please create a profile using the procedure at https://support.mozilla.org/en-US/kb/profiling-thunderbird-performance