280382 - "Compact folders" does not work on inbox causing inbox to grow out of control

Status: RESOLVED FIXED

(Thunderbird :: General, defect)

Description

[john]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0

using thunderbird version 1.0 (20041206) I have a large amount of mail that I
have mostly sorted into subfolders of the "[e-mail]" folder (not
subfolders of the inbox) each of these subfolders have their own mbox file that
lives in "C:\Documents and Settings\%USER%\Application
Data\Thunderbird\Profiles\default\Mail\%HOST%.%DOMAIN%.%TLD%" and they contain
the messages moved from the inbox but the inbox mbox file seems to still contain
every message that I have ever recieved. as a result thunderbird has gotten very
slugish and sometimes features such as thunderbird's filure to display external
images to protect your privacy stop working. also it dosen't update the count of
unread messages in a milbox that I am viewing until I click on another mailbox
and then click back onto the one that I was previously viewing. I am using POP. 

Reproducible: Always

Actual Results:  
inbox is 339025948 bytes and entire mail folder is 675149429 bytes (notice that
inbox is almost exactly half the size of all my mail)

Expected Results:  
expunge messages from the inbox file once they are moved elsewhere, if this had
happened my inbox would be just a few K as it only contains a few messages.

Comment 1

[john]

I have worked around this problem this time by simply renaming the inbox file,
creating a new empty file called inbox, starting thunderbird, copying all of the
messages in the old inbox folder to the new inbox (generating a 1,479KB file)
and then deleting the old inbox folder. I have a backup of everything in the
pre-workaround state so I can go back to it to answer questions or try things.

Comment 2

[WADA:World Anti-bad-Duping Agency]

Use "Compact Folder".
See http://kb.mozillazine.org/Thunderbird_:_Tips_:_Compacting_Folders
Close this bug as INVALID, please.

By the way, I think next is one of the best help for US.
 http://kb.mozillazine.org/Main_Page
  - http://kb.mozillazine.org/Thunderbird_:_Issues
  - http://kb.mozillazine.org/Thunderbird_:_FAQs
  - http://kb.mozillazine.org/Thunderbird_:_Tips
Read them before open bug, please.

Comment 3

[john]

Actually "compact folders" did nothing to the inbox but once I had renamed it
and created a new empty inbox it properly compacted the old inbox.

Comment 4

[john]

I have changed this bug to '"compact folders" does not work on inbox causing
inbox to grow out of control' as a result of the message sent me by WADA and
further investigation.

To duplicate perform the following steps:

1. remove a message from the inbox (I moved the first one out because it was the
easiest to check)

2. right click on inbox and select "compact this folder" from the context menu
OR
2. select "compact folders" from the file menu

3. close thunderbird

4. open "C:\Documents and Settings\%USER%\Application
Data\Thunderbird\Profiles\default\Mail\%HOST%.%DOMAIN%.%TLD%\inbox" in your
favorite text editor

5. confirm that the message that you removed from your inbox is still there

Comment 5

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #4)
What is your directry/file names for the "inbox" folder which "compact folder"
won't run?
(Q1) Where the "inbox" folder is defined? (under which account?)
(Q2) What is "local directry:" setting for the account?
     See "Server Settings"/"Local directry:" of Account Settings
     where the "inbox" id defined.
(Q3) What is "folder location" of the "inbox" folder?
     See "folder location of the "inbox"
      - "Copy folder location" from context menu of the "inbox" folder
      - Open text editor (notepad.exe is sufficient)
      - Paste(CTRL+V) on text editor

Comment 6

[john]

A1: the account "[e-mail]" which is a pop3 account for user "jonny5"
on server "shell.lightlink.com"

A2: C:\Documents and Settings\john\Application
Data\Thunderbird\Profiles\s0mt5204.default\Mail\shell.lightlink.com

A3: mailbox:/C|/Documents and Settings/john/Application
Data/Thunderbird/Profiles/s0mt5204.default/Mail/shell.lightlink.com/Inbox

please note that "compact folders" works with all of the other folders it is
only inbox that it doesn't work for.

Comment 7

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
Proper mail folder file seems to be checked...
John, I have some questions.

(Q1) How long did you wait for "Compact folder" completion when compacting "inbox"?
Since your old "inbox" file size is 339,025KB bytes, compact folder usualy takes
long time.
(If only one mail which is placed at the top in "inbox" file is deleted, compact
folder have to physicaly move or copy almost all 339,025KB bytes.) 

(Q2) What is your evidence of "Compact folder didn't work for 'inbox'"?
     Please note that mail displayed top(as oldest) in mail folder is usually 
     NOT the mail data placed at the top of mail folder file('inbox' file).

Comment 8

[john]

(A1) on my old inbox I waited until the progress bars had finished and it had
showed completion which took a long time, on my new inbox I waited only a few
seconds as I only removed one message and the inbox is only about 10 messages
altogether. it seemed to indicate completion and all hard drive access had stopped.

(A2) I moved a specific message out of the inbox and then "compacted folder",
then closed thunderbird and opened the inbox with a text editor then verified
the existence of that specific message in the inbox even though it does not show
up in the message display and I have compacted the folder (several times at this
point)

please note that at present my inbox is only a few messages long and it is
rather small.

here is the header of the deleted message copied out of the inbox MBOX file:

From - Sun Jan 02 00:26:50 2005
X-Account-Key: account1
X-UIDL: T*("!E"5!!,/H!!mgQ!!
X-Mozilla-Status: 0009
X-Mozilla-Status2: 00000000
Return-Path: <[e-mail]>
Received: from mx.lightlink.com (pop.lightlink.com [205.232.34.15])
	by adore.lightlink.com (8.9.3p2/8.8.8) with ESMTP id AAA26493
	for <[e-mail]>; Sat, 1 Jan 2005 00:15:13 -0500 (EST)
Received: from spam2.lightlink.com (spam2.lightlink.com [205.232.89.54])
	by mx.lightlink.com (8.9.3p2/8.8.8) with ESMTP id AAA20436
	for <[e-mail]>; Sat, 1 Jan 2005 00:15:09 -0500
X-ASG-Debug-ID: 1104556508-32320-32-0
X-Barracuda-URL: http://spam2.lightlink.com:80/cgi-bin/mark.cgi
Received: from smtp0.lightlink.com (mail0.lightlink.com [205.232.89.50])
	by spam2.lightlink.com (Spam Firewall) with ESMTP id 679202004128
	for <[e-mail]>; Sat,  1 Jan 2005 00:15:08 -0500 (EST)
Received: from lightlink.com (root@localhost)
	by smtp0.lightlink.com (8.9.3p2/8.8.8) id AAA03339
	for [e-mail]; Sat, 1 Jan 2005 00:15:08 -0500
Received: from nnytech.net (mail.converselabs.com [64.19.75.36])
	by smtp0.lightlink.com (8.9.3p2/8.8.8) with SMTP id AAA03323
	for <[e-mail]>; Sat, 1 Jan 2005 00:15:07 -0500
Received: (qmail 23178 invoked from network); 1 Jan 2005 05:09:58 -0000
Received: from localhost (HELO ?127.0.0.1?) (127.0.0.1)
  by 0 with SMTP; 1 Jan 2005 05:09:58 -0000
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-ASG-Orig-Subj: wwia.org mailing list memberships reminder
Subject: wwia.org mailing list memberships reminder
From: [e-mail]
To: [e-mail]
X-No-Archive: yes
Message-ID: <[e-mail]>
Date: Sat, 01 Jan 2005 05:05:19 +0000
Precedence: bulk
X-BeenThere: [e-mail]
X-Mailman-Version: 2.1.5
List-Id: mailman.wwia.org
X-List-Administrivia: yes
Sender: [e-mail]
Errors-To: [e-mail]
X-Virus-Scanned: by Barracuda Spam Firewall at lightlink.com
X-Barracuda-Spam-Score: 0.64
X-Barracuda-Spam-Status: No, SCORE=0.64 using global scores of TAG_LEVEL=3.5
QUARANTINE_LEVEL=9.0 KILL_LEVEL=1000.0 tests=FORGED_RCVD_HELO, NO_REAL_NAME
X-Barracuda-Spam-Report: Code version 2.64, rules version 2.1.754
	Rule breakdown below	pts        rule name                      description
	---- ---------------------- -------------------------------------------
	0.64 NO_REAL_NAME           From: does not include a real name
	0.00 FORGED_RCVD_HELO       Received: contains a forged HELO
X-UIDL: T*("!E"5!!,/H!!mgQ!!

Comment 9

[john]

Attachment: corrupted msf file causing inox not to compact properly

Comment 10

[john]

I have tracked the problem down to being a corrupted MSF file (I can only assume
that it is corrupted as I do not know what the format is supposed to be) simply
removing this MSF file caused the problem (along with several others) to go away.

Comment 11

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #10)
> simply removing this MSF file caused the problem (along with several others)
to go away.
As you see, automatic ".msf" (re)creation works in some occasions. For expamle,
 - No ".msf" is found on restart(your workaround procedure)
 - Inconsistency between "inbox.msf" and "inbox" file, eg. different timestamp.
But this "automatic .msf recreation" was not invoked in your case.
This indicates that Thunderbird didn't find inconsistency between them.
What's wrong in your ".msf"?
What in your ".msf" caused "compact folder doesn't work"?

Comment 12

[john]

I have no idea what it was about this .msf file that was causing my problems,
only that removing it solved them, it also solved the problem of the inbox not
updating it's unread messages count in the folders pane until I clicked away and
then clicked back. I was hoping that someone who knew something about the format
of the .msf file could find mine useful as it seemed to be causing my problem.

I used the following logic to solve my problem:

1. based on your response in comment #7 I assumed that you were unable to
duplicate the problem even given the information that you seemed to need to
duplicate it

2. other folders seemed to compact properly so it was something specific to the
inbox

3. the inbox seems to have three components: and empty directory called
'inbox.sbd", the file "inbox" and the file "inbox.msf"

4. I had already deleted the file "inbox" and replaced it with a new empty file

5. the "inbox.sbd" directory was empty anyway so it was unlikely the problem

6. the "inbox.msf" file was the only remaining component that could be suspect

I had noticed that another troubleshooting tip (having to do with an error
message that claimed that the folder was being processed) was to delete the .msf
file so I already knew that it would regenerate one for me if it didn't exist so
I decided that it would be safe to delete it but I also wanted to preserve the
evidence so that hopefully something could be learned from all of this so I
saved the file instead.

I am curious if using my inbox.msf file in another configuration (perhaps on a
test machine or test install of thunderbird or test account) would cause simular
problems. I can assure you that there were different modification times between
the .msf and the inbox itself and it wasn't regenerated as I deleted the inbox
and replaced it with a new empty file and the problems persisted.

Comment 13

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #12)

Your logic 1. to 6. is right.

> it also solved the problem of the inbox not updating it's unread messages
> count in the folders pane until I clicked away and then clicked back. ion

Do you mean other problems already existed before this bug's problem?
If yes, your case sounds next.
 (1) "inbox.msf" has already some problems.
 (2) Then some process is running on the folder, or a kind of hung exists.
 (3) While (2), you tried "compact folder", then compact folder cannot run.
If right, this bug never has relation to "compact folder".
"Compact folder" is only a victim of other problem.

> I can assure you that there were different modification times between
> the .msf and the inbox itself and it wasn't regenerated as I deleted the inbox
> and replaced it with a new empty file and the problems persisted.

This indicates that problem is above (2).
To validate timestamp of "Inbox" file, contents of ".msf" is required.
Then ".msf" file processing steps exist before .msf recreation steps.
Since ".msf" was same one, problem such as wait or loop and so on occured due to
damaged ".msf" content, then ".msf recreation logic" couldn't get control.

> I am curious if using my inbox.msf file in another configuration
> (perhaps on a test machine or test install of thunderbird or test account)
> would cause simular problems.

If you curious, use safer way - copy "Inobx" file(and "Inbox.sbd" if required) only.
As you know, ".msf" is receated on restart if deleted.
So, even if garbage exists in ".msf", it'll be cleared by new Thunderbird.
But please note that next.
   If folder name contains special character(illegal filename char, eg. "\"),
   file name for the folder becomes hashed name(hexa-string).
   In this case, original folder name is kept in ".msf" file.
   Therefore, deleting ".msf" causes loss of original folder name.
   Folder name becomes the hased file name(hexa-string) after resatrt.

Comment 14

[David :Bienvenu]

Attachment: handle .msf file that's out of sync with mailbox

when compacting a folder, and detecting a .msf file that's out of sync with the
folder, we were trying to mark the db invalid so that we'd reparse the folder
the next time you opened it. But, we weren't committing the db, so that change
was never getting flushed. This should help, though I'm not sure it's the same
problem.

Comment 15

[Mike Cowperthwaite]

David Bienvenu, what's the status with your (sr'd) patch here?
Was it checked in?

Comment 16

[David :Bienvenu]

yes, it was checked in.