Closed Bug 493222 Opened 15 years ago Closed 13 years ago

Inbox does not recreate itself from scratch when it loses the mail index or index is corrupt (Mac)

Categories

(MailNews Core :: Database, defect)

1.8 Branch
x86
macOS
defect
Not set
major

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: judith, Unassigned)

References

Details

(Whiteboard: [needs protocol log if repopend])

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.10) Gecko/2009042315 Firefox/3.0.10
Build Identifier: 2.0.0.14

Using Imap.  If outside mail server has corruption or other activities in the mail index file this causes Thunderbird to recreate its own index file (inbox.msf) and download all email (rebuild/recreate the Inbox). Setting are set to download all mail while offline. The bug is that instead of recreating the Inbox and starting fresh, Thunderbird just adds to the Inbox. Although this does not produce duplicate emails, it does double the size of the Inbox file.  So if the Inbox is 60Mb than each time the Inbox rebuilds it will add 60mb of data to the file. So First rebuild is 120mb, next one is 120mb and so on. Once the Thunderbird Inbox file reaches over 2Gb Thunderbird starts to slow down and fail. Over 3gb mail no longer downloads correctly or can be viewed off line. The only correction to this problem is to manually delete the Inbox file and have Thunderbird rebuild it. 

Reproducible: Always

Steps to Reproduce:
1.Mail server has corruption or other things going on in the Index files
2.Thunderbird notices this corruption and rebuilds the mail index and downloads the mail again into the Inbox. Inbox file grows exponentially so if Inbox is 60Mb,it grows by a factor of 60 each time.
3.When file reaches over 2gb Thunderbird slows down and over 3Gb Thunderbird stops downloading mail
Actual Results:  
1.Inbox file grows exponentially so if Inbox is 60Mb,it grows by a factor of 60 each time the inbox.msf gets rebuilt.
2.When Inbox reaches over 2gb Thunderbird slows down and over 3Gb Thunderbird stops downloading mail

Expected Results:  
If the mail index file, inbox.msf, is rebuilt so should the actual Inbox. It should be rebuilt from scratch. The old one deleted and a new one created. This would keep the integrity of the files. So each time the mail inbox is created and mail is downloaded again from the server it should be in a new Inbox file and not just added to the older file.
Same phenomenon/problem as bug 487992?
Similar, but in my situation I was not trying to compact the Inbox nor sought to delete the .msf file, as the other person was .  The index.msf file gets rebuilt because it notices a corruption in the IMAP mail index on carrier's mail server, where my mail is stored. I can tell when Thunderbird will rebuild the mail index since it almost always happens after seeing the following error message in Pine message "x (numerical message number)" UID is less than 13,346, or some figure like this. (I read my mail via Pine before downloading it to my machine using Thunderbird).
(In reply to comment #2)
> The index.msf file gets rebuilt because it notices a corruption in the IMAP mail index on carrier's mail server, where my mail is stored.
> I can tell when Thunderbird will rebuild the mail index since it almost always
> happens after seeing the following error message in Pine message
> "x (numerical message number)" UID is less than 13,346, or some figure like this.
> (I read my mail via Pine before downloading it to my machine using Thunderbird).

In bug 487992's case(manual rebuild-index without deletion of Inbox.msf), some data in Inbox.msf(e.g. local only Tag due to unsupported user defined keyword) is reused to avoid loss of such data which doesn't exist on IMAP server.
I guess Tb discards "Inbox.msf" in your case(same as "manual deletion of Inbox.msf"). But Tb's behaviour is possibly same as "manual rebuild-index without deletion of Inbox.msf" case even in your case.

Which do you mean by "doesn't create from scratch"? Both?
 (a) Old data in Inbox.msf such as "local only Tag" remained.
     If yes, what kind of problem occurrs after that?
 (b) File size of Offline-store("Inbox", no file extension) is not reduced.
     (bug 487992 is for (b) only)
1) If Thunderbird running under IMAP senses its Index file for the Inbox-Inbox.msf file is corrupt it will rebuild and recreate it.  Are you saying your Inbox.msf file is never recreated unless it is manually deleted.  In that case this is a totally different bug. The bug occurs only when Thunderbird running under IMAP thinks its current index is corrupt and rebuilds it-rebuilds theinbox.msf file

The bug here is that the Inbox file is not rebuilt/recreated each time the Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.  I now delete it every week and start over. Creation date of Inbox is does not change with each rebuild/recreation of Inbox.msf. There are no duplicate emails, file just gets doubled. 

2) Not looking to compact file. So reference to compact and mail delete button do not apply to this bug

3)auto-sync/"Download Now" will work correctly if not for the bug. As long as inbox.msf is not rebuilt/recreated Download now or download mail offline will work, ie, correctly download the new mail and incrementally increase the size of the inbox by the size of the new mail that is downloaded.
(In reply to comment #4)
> 1) (snip)
> Are you saying your Inbox.msf file is never recreated unless it is manually deleted.

No.
I wanted to say that (a) "rebuild-index" doesn't not always discard all data in previous "Inbox.msf", (b) If "Inbox.msf" is deleted manually, "rebuild-index" re-creates "inbox.msf" from absolute scratch.
See bug 392704, please. Before bug 392704, rebuild-index was same as "manual deletion of Inbox.msf", but it was changed(to keep local only data).
As for mail data, all mails are re-fetched from server when rebuild-index of IMAP folder. Mail of UID which doesn't exist in "fetch 1:*" response is removed, and mail of UID which Tb knew is replaced by data downloaded from server, and mail of UID which Tb didn't know is newly added. So (a) can be said "from scratch".

> The bug here is that the Inbox file is not rebuilt/recreated each time the
> Inbox.msf is recreated/regenerated. The file, Inbox, grows almost exponentially
> so 76.7 mb becomes 159 and 159 becomes 236.5 mb. Until it gets too large.  I

Bug 487992 is for this problem. Bug 487992 can occur upon any of (1) manually forced rebuild-index(test case of Bug 487992), (2) rebuild-index due to server side change such as reconfiguraion(your case), (3) manual deletion of msf, (4) internal rebuild-index due to corrupted msf(who corrupted==Tb, usually).
Component: General → Database
Product: Thunderbird → MailNews Core
QA Contact: general → database
Judith, can you reproduce with v3.0 or latest v2.0.0.24 due out next week?
Whiteboard: closeme 2010-04-12
Version: unspecified → 1.8 Branch
Let me try it under version 3.  Currently I was using an earlier build of version 2.  Under this build, each week or every two weeks I manually delete the inbox and the inbox.msf file as the Inbox file has grown to almost 2Gb.  Once the inbox.msf is regenerated and a new inbox file is created the size of the inbox falls down to 80mb. As the bug explains as the inbox.msf file is regenerated the inbox file doubles in size but there are no duplicate emails.
Yes. I am using version 3.0 and am still having the same problem.  When the inbox.msf file was regenerated the inbox grew from 1.6gb to 2.2 gb but no additional or duplicate emails were contained.  If I delete the inbox.msf file the inbox will shrink to under 100mb and with each regeneration of the inbox.msf file the inbox size will continue to grow exponentially.
(In reply to comment #8)
> Yes. I am using version 3.0 and am still having the same problem.
> When the inbox.msf file was regenerated, the inbox grew from 1.6gb to 2.2 gb
> but no additional or duplicate emails were contained.
> If I delete the inbox.msf file the inbox will shrink to under 100mb
> and with each regeneration of the inbox.msf file the inbox size will continue to grow exponentially.

Next bug is already fixed by Tb 3.0rc1. 
> Bug 487992 File size of offline store of IMAP folder increases upon each "Rebuild Index"
Judith Hellerstein, are you confusing "rebuild-index" with "compact folder"?
If you use "offline use=on" for the IMAP folder, did you confirm message of "<imap_account> is up to date" at Activity Manager window? (indicator of "all mails are downloaded to offline-store file").
Perhaps I used the wrong term as I am not a technical person.  All this happens online and not offline.  I never use compact folder so if I used the wrong terminology and you thought I was referring to compact folders I am sorry about that.  

I have not used the Activity Manager, I can see all the messages and know they are upto date. I can read my messages in Alpine and so see that the two message indices match If that is what you were referring to.

All I can tell you if that twice today IMAP recreated its index and so had to reload all the message headers and then the messages. If I look at the size of the inbox within Thunderbird I can tell the size has almost doubled from the last time the index got rebuilt. It does not matter whether Thunderbird is offline or online.
(In reply to comment #10)
> It does not matter whether Thunderbird is offline or online.

Please distinguish "offline use=on/off"(Folder Properties/Synchronization) and "offline/online mode"(Work Offline/Work Online).
Auto-sync uses local file for "download upon Work Offline"(offline-store) as "file for local copy of whole mail data to utilize Global Search and Indexer(body search)", if "offline use=on" is set for an IMAP folder. 

> All I can tell you if that twice today IMAP recreated its index and so had to
> reload all the message headers and then the messages.
> If I look at the size of the inbox within Thunderbird I can tell the size has almost doubled from the last time the index got rebuilt. 

As I wrote before, if rebuild-index occurs, offline-store file is deleted before re-synching of all mail data, because Bug 487992 is already fixed.
"almost doubled from the last time the index got rebuilt" is possibly one of next phenomena;
 (i) Nearly all mails were deleted, and mails were newly added,
     after last rebuild-index. Compact of folder is not executed.
     Total size of deleted mails == Total size of newly added mails.
 (ii) Re-fetching of all mail data happened due to something wrong.
     Compact of folder is not executed, or fails to reduce file size.
In any case, offline-store file is not deleted. mail data is appended to offline-store file, and old data in offline-store is merely ignored(no one will access the old mail data).
To remove old data from offline-store and reduce offline-store file size, "Compact folder" is required.

> When the inbox.msf file was regenerated the inbox grew from 1.6gb to 2.2 gb

Problem of bug 537498? (fixd by bug 532323, but fix for Tb 3.1 only)
If (i) occurs, bug 537498 can happen. Once bug 537498 occurs, (ii) can occur, and "Compact" fails to execute compact operation correctly, because offset value is wrapped and corrupted.
Ok So I think I understand.  Yes I have selected, save mail while offline, but not sure why that matters when the same problem occurs whether I am online or offline.  I am using IMAP.

So what you are saying is that the file "Inbox" gets larger since it no longer can recognize that the data that is there are the old mails so it just appends the mail contained in the inbox to what is already in the file, which is why I see no duplicate emails. Since I do not choose compact the Inbox file the size keeps growing exponentially larger since it just adds the current mail onto of the old mail which is taking up the space in the file, but can not be recognized by the system.  According to what you are telling me, if I compacted the file, the size would shrink to the correct size.

However, I have a problem with that solution. To me, either way you look at it a problem remains. I either have to toss out the old inbox file into the trash and have it rebuilt from scratch or compact the Inbox after it is rebuilt. Either way to me that is still a bug. The goal would be to totally eliminate that step.

Now if I could figure out how to stop Thunderbird from rebuilding the inbox that would be an amazing feat.
(In reply to comment #12)
> So what you are saying is that the file "Inbox" gets larger since it no longer
> can recognize that the data that is there are the old mails so it just appends
> the mail contained in the inbox to what is already in the file, (snip)

No.
Because Unix Mbox format is used for offline-store file too, similar "compact folder" to "compact of local mail folder"(Unix Mbox format is used too) is required to remove old data. As file for offline-store and file for local mail folder usually contains many mail data, "compact" is usually expensive job because copy of all alive data is required for compaction.
It's reason why "control of frequency of compaction" is entrusted to user. 

> However, I have a problem with that solution. To me, either way you look at it
> a problem remains. I either have to toss out the old inbox file into the trash
> and have it rebuilt from scratch or compact the Inbox after it is rebuilt.
> Either way to me that is still a bug. The goal would be to totally eliminate
> that step.

See Bug 58308(support qmail's maildir format) for request of "one file per mail"(no need of "compact" like current), please.

> Bug summary : Inbox does not recreate itself from scratch when it loses the mail index or index is corrupt

Does it still problem for you?
Rebuild-Index of Tb3 recreates .msf from scratch except local only data(e.g. tag data if IMAP server doesn't support user defined flag=keyword), and deletes offline-store file and re-synchronize all mail data if "offline use=on".
What is problem?
Are you claiming that such local only data should be discarded upon rebuild-index?
No. the .msf folder does not bother me. It is just that every time the inbox.msf is rebuilt then I have new data added to the the old data and the off line inbox file gets larger. So, according to what you have told me I need to compact the local inbox file after each rebuild to get rid of the old data. Or, if I do not like to compact then I can toss into the trash the local offline inbox file each week and rebuild the offline inbox file.
Whiteboard: closeme 2010-04-12
Judith, how is version 3.1.6? And can bug 562456 be fold back into this issue? It seems like they must have the same cause.
Blocks: 562456
also, please give more details about "outside mail server has corruption" and why this isn't getting fixed by your provider.
I am using version 3.1.7 and the problem is still there. I am using IMAP. Often when I take the mail offline and then go online again, it either downloads the entire inbox headers and messages or only downloads the messages that were most recent as stated in bug 562456.  I then have to go into Edit, Folder properties and hit repair.  So yes, bug 562456 can be folded into to this bug.  I think they were only separate since they happened on two different versions of Thunderbird and was told they were two separate issues.

I am unsure why some times Thunderbird downloads all the mail and headers again and why sometimes it only downloads the newest messages. Repairing the folder often solves this issue, but sometimes not and I have to hit repair folder a second time for Thunderbird to redownload all messages in the Inbox.

I am unsure why my provider is not fixing this.  Perhaps they do not see it as an issue.  The newer versions of Thunderbird seem to automatically compact the Inbox so I no longer have a problem with a growing Inbox that I have had before.
It was mentioned to me to post a an imap and autosync log as that would show what is happening between TB and the server, but I am unclear how to do that.
(In reply to comment #18)
> It was mentioned to me to post a an imap and autosync log as that would show
> what is happening between TB and the server, but I am unclear how to do that.

Documentation is at https://wiki.mozilla.org/MailNews:Logging
I have tried using the documentation you listed and it does not seem to generate a log.  Keeps saying event not found. I have a MacBook, 2.4 GHz Intel Core 2Duo running version 10.5.8.
Judith, please post details about what you tried. Did you use
 imap:5,autosync:5 ?

I don't understand where you would see a message or log entry like "event not found"
Whiteboard: [needs protocol log]
Wayne, I use terminal to write the script and for some reason could not generate a log.  It worked last time, but now with an Intel based laptop the same script does not work.  After several tries I gave up.  I then tried compacting twice a week and some how after two weeks the problem went away. I do not seem to be having the corruption problems that caused the problem. Hopefully my mail will continue to remain stable and not corrupt.  I have noticed other quirky issues (cannot copy text from one email to another without first copying the text to word and then recopying it into the email) but these happen intermittently and so have not reported them.
> I then tried compacting twice a week and some how after two weeks the problem went away. I do not seem to be having the corruption problems that caused the problem.

thanks for that update.

is the problem with Bug 562456 - Mail disappearing from Inbox - thus gone as well?
Status: UNCONFIRMED → RESOLVED
Closed: 13 years ago
Resolution: --- → WORKSFORME
Summary: Inbox does not recreate itself from scratch when it loses the mail index or index is corrupt → Inbox does not recreate itself from scratch when it loses the mail index or index is corrupt (Mac)
Whiteboard: [needs protocol log] → [needs protocol log if repopend]
Yes. It is.  I went to that bug's comment page and checked resolved for the current version of Thunderbird I am using 3.1.8
You need to log in before you can comment on or make changes to this bug.