538610 - IMAP mailbox: infinite loop of "compacting folder..." (IMAP folder larger than 4GB limit of offline-store size)

Status: RESOLVED WORKSFORME

(Thunderbird :: General, defect)

Description

[Graham Leggett]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB6
Build Identifier: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0

When an attempt is made to open a large IMAP mailbox with a large amount of folders, soon after the first attempt is made to fetch mail, thunderbird takes it upon itself to start compacting every imap folder in turn.

This continues in an infinite loop, consuming a considerable quantity of bandwidth, and is particularly noticeable on slow links (where bandwidth is typically very expensive).

This compaction occurs despite the fact that the mail folders are for archival purposes, and no changes have been made to the contents of any of the folders.

During this process, if a folder is encountered that does not exist, instead of simply skipping the non existant folder, thunderbird insists on popping up a modal dialog box, forcing the end user to acknowledge that an unspecified folder does not exist. This is then repeated over and over again, in an infinite loop.

This renders thunderbird unusable for normal use.


Reproducible: Always

Steps to Reproduce:
xxx

Comment 1

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #0)
> When an attempt is made to open a large IMAP mailbox (snip)

How large are such mailboxes?
Check file size of offline-store of Tb. It's file named <folder_name>(not <folder_name>.msf) if <folder_name> doesn't have special chras such as \, /, #, ;, ?, ..., and/or total mail data size corresponds to an IMAP mail folder for Tb as an IMAP client.

Comment 2

[Graham Leggett]

Inbox looks like this:

-rw-r--r--@  1 minfrin  501  10715246 Jan  8 20:27 INBOX

One of the subfolders of INBOX is a folder called Archive, I notice this:

-rw-r--r--@  1 minfrin  501         0 Jan  8 19:28 Archive-1
-rw-r--r--@  1 minfrin  501      1911 Jan  8 18:38 Archive-1.msf
drwxr-xr-x@ 38 minfrin  501      1292 Jan  8 19:28 Archive-1.sbd
drwxr-xr-x  12 minfrin  501       408 Oct 16 13:13 Archive.sbd

There is both an Archive-1.sbd and an Archive.sbd folder containing further subfolders.

I'm not sure why the folders have been duplicated, the Archive.sbd seems from the dates to contain files from Thunderbird 2, the Archive-1.sbd directory contains folders that are current and up to date, for example:

-rw-r--r--@  1 minfrin  501    175655 Jan  8 19:28 Sent
-rw-r--r--@  1 minfrin  501   6298710 Jan  8 18:26 Sent.msf

Under what circumstances would a folder called "Archive-1.sbd" appear, when the folder name is "Archive".

Comment 3

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> Inbox looks like this:
> -rw-r--r--@  1 minfrin  501  10715246 Jan  8 20:27 INBOX

Is "10715246"(==10,715,246) file size in bytes of the Inbox file on your system?
If so, see bug 462665, bug 532323, and bug 537498.
Read bug 537498 comment #0 for a posible workaround.

Comment 4

[Graham Leggett]

Yes, INBOX is 10MB in size.

The combined size however of folders and subfolders below inbox amounts to many times greater than 4GB.

Ugly hacks like trying to fragment folders differently (eg by months instead of by years) is not a solution - it's 2010 after all.

This needs to be fixed urgently - thunderbird is completely useless in its current form.

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #2)
> -rw-r--r--@  1 minfrin  501         0 Jan  8 19:28 Archive-1
> -rw-r--r--@  1 minfrin  501      1911 Jan  8 18:38 Archive-1.msf
> drwxr-xr-x@ 38 minfrin  501      1292 Jan  8 19:28 Archive-1.sbd
> drwxr-xr-x  12 minfrin  501       408 Oct 16 13:13 Archive.sbd

Old Tb 3.0x(aN,bN,pre) nightlies added suffix and increased the suffix value upon each change from offline-use=off to offline-use=on of an IMAP mail folder.
I coulnd't see this problem with Tb3.0 offcial release.
I guess the problem was resolved by bug 487992, because old offline store file is deleted upon internal rebuild-index by patch for bug 487992.

If Archive folder size is small, I recomend you to delete all Archive*.* files/directories and re-open Archives folder and force re-sync.

Comment 6

[Graham Leggett]

Having found the one folder (out of many) that was bigger than 4GB, and splitting that folder, the problem was not solved.

It would seem that if the combined total size of your mailbox (sum of all mail in all folders) is greater than 4GB, then thunderbird goes into a loop of compacting folders forever, or the cause could be something else completely.

There has been a long standing problem in thunderbird-2 where on slow networks, thunderbird would download from the imap server continuously, but because of poor UI to the end user, you had no feedback as to what it is doing. TB3 now provides feedback, and like TB2 it downloads from the imap server continuously, which could be the same problem.

I'll be back on a fast network in just over a week, where I'll be able to confirm whether the problem goes away if I am on a fast network.

There is a further problem in that TB3's event loop spins during this process, keeping TB3's CPU usage at close to 100%. It's not clear whether this is caused by this bug, or if it is a separate bug. Setting TB3 to work offline stops the spin, but as soon as you flip to online, the spin returns.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> Having found the one folder (out of many) that was bigger than 4GB,
> and splitting that folder, the problem was not solved.

What was you action for "splitting that folder"? (probably Inbox) Next?
  Move mails in Inbox to new Inbox-A, Inbox-B, Inbox-C.
  IMAP mail folder size of Inbox, Inbox-A, Inbox-B, Inbox-C is less than 4GB.
What is your IMAP delete model? Other than "Mark it as deleted"?
What did you do after the splittig? If above was your spliting, did you execute compact for Inbox? Did you execute "compact" of Inbox with keeping "offline ise=on" for Inbox?

As ofline-store size of Inbox already exceeds 10GB in yout case, bug 537498 occurs even ater you reduced IMAP folder size of Inbox. I recommend you next.
1. Disable Global Search and Indexer to avoid unwanted problem.
   Set IMAP delete model other than "Mark it as deleted".
2. Set "offline use" of Inbox to off.  
3. Move all mails in Inbox to new Inbox-A, Inbox-B, Inbox-C, ,,, Inbox-Z.
   As bug bug 537498 exists, keep less than 2GB to avoid unwanted problem.
4. For null Inbox, enable offline use, and execute rebuild-index.
   (rebuild-index deletes 10GB offline-store for Inbox)
5. Re-enable Global Search and Indexer if required.

Comment 8

[WADA:World Anti-bad-Duping Agency]

Oh, Inbox size was 10MB instead of 10GB. Read "Inbox" "folder bigger than 4GB", please.

Comment 9

[Graham Leggett]

It wasn't inbox that was split, no.

Inbox weighs in at roughly 10MB or so. Below inbox is a folder called "Archive", and below archive are folders for each year, beneath which are folders for specific mail within each year. The >4GB folder was a folder for forwarded mail with attachments that had not been properly split by year. Once it was split by year properly, that particular folder was back below 4GB.

The split was done by moving the mails, causing the original mails to be marked as deleted. The folder was then compacted, causing the mails to be physically deleted, and this was checked on the IMAP server itself.

Moving back to a fast network seemed to fix the problem, in the sense that TB stopped downloading continuously. I suspect at the root of the problem is that if TB is unable to complete a specific task within a certain amount of time, it will attempt to repeat the task over and over, without noticing it is already still busy.

Comment 10

[Graham Leggett]

More behaviour seen.

It seems now that TB3 is on a fast network, it has appeared to finish the endless loop of compacting folders, however it still continues downloading mail over and over again.

My INBOX size on the IMAP server is currently 87MB:

87736	cur/

My INBOX size how according to TB3 is 10GB:

-rw-------@ 1 minfrin  501  10361628701 Feb  8 12:52 INBOX

It seems to be "leaking" mail, not into memory, but rather onto disk.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #10)
> My INBOX size on the IMAP server is currently 87MB:
> My INBOX size how according to TB3 is 10GB:
> -rw-------@ 1 minfrin  501  10361628701 Feb  8 12:52 INBOX

Even if IMAP folder file size is smaller than limit, bug 537498 can occur on Tb without fix of Bug 532323, and this bug may occur.
> Bug 532323 4 GB limit of IMAP offline-store breaks sync of gmail "All Mail" and other systems
> Bug 537498 If IMAP offline-store file size exceeds 4GB, mails downloaded at over 4GB can not be read, and downloaded again & again, even if mail folder size is within 4GB
Bug 532323 is already fixed(trunk, Tb3.2a1pre currently).
Check with trunk nightly, please.
> http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-central-trunk/

Comment 12

[Wayne Mery (:wsmwk)]

Graham have you seen this problem when using version 3.1 or newer?

Comment 13

[Graham Leggett]

I haven't used Thunderbird in a while, so I downloaded v8.0, and tried to download and then compact the folders afterwards, and it all seemed to work correctly for me.

Comment 14

[Ludovic Hirlimann [:Usul]]

Thanks for the update Graham.