232047 - Frequent need to rebuild summary file(.msf) with folder on a different OS's sharing hard disk or network drive (timestamp inconsistency), often when share goes "off-line". Causes lost folder column settings.

Status: NEW

(MailNews Core :: Database, defect)

Description

[Viktors]

User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.6) Gecko/20040113
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.6) Gecko/20040113

In past versions of Mozilla (before 1.6) I noticed that occasionally it would
spend a lot of time rebuilding the summary file for my inbox upon opening the
email windows.  Now with 1.6, it is happening almost every time even though I
shut down cleanly.  I have about 4803 messages in my inbox so this takes many
minutes to rebuild.  Why is this happening so often?

Reproducible: Sometimes

Steps to Reproduce:
1. Shutdown system (with or without closing Mozilla first)
2. Boot system
3. Start Mozilla with -mail parameter

Actual Results:  
It spends a lot of time (several minutes) rebuilding the summary file for my
inbox and also forgets that I used to have it sorted by date with the newest on top.

Expected Results:  
Why does it think the summary file needs rebuilding so often?  Make it not
require this unless there is a system crash or something more serious.

I have 4803 messages in my inbox currently and it is this big:

INBOX          215,467,800  01-24-04 10:32a Inbox
INBOX    MSF     2,138,693  01-24-04 10:32a Inbox.msf

Comment 1

[David :Bienvenu]

First of all, you should compact your INBOX, if you haven't done so recently (or
ever). Just use the folder pane context menu and pick "Compact this Folder".

Second, I don't know why the summary file would get regenerated so often. It
also shouldn't forget your sort order; I thought I had fixed that.

If you can figure out some pattern to what causes this to happen, that would be
useful. What's the last thing you do in the inbox before you shutdown the app?

Comment 2

[Viktors]

I compacted the file and it has not happened again so far.  I have restarted
Mozilla twice and rebooted once.

I usually visit each folder that is marked with bold and either delete unwanted
messages, view them, or ctrl-shift-C to mark them all as read.  I usually empty
the trash last.  If emptying the trash does not unbold the trash bin, I visit
the trash bin and this unbolds it.  Then I am done and shutdown the system or
close Mozilla.  I am not using the fast start feature.

Comment 3

[Viktors]

I wanted to add that I am still getting rebuilds from time to time even though I
have compacted the folders.

Comment 4

[Christian Eyrich]

I do get rebuilds a couple of times each day too. Compacting doesn't help.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Automatic mail check task keeps alives after Mozilla Mail&News window termination.
This may cause ".msf" file contention between next mail check on Mail&News start up.
If the contention occurs, or if start up of Mail&News cancels old mail check
task while the old mail check task is trying to down load mail, it possibly
causes  ".msf" file inconsistency.
Then ".msf" recreation possibly occurs.

David Bienvenu and Christian Eyrich, can ".msf" inconsistency and ".msf"
recreation occur in this sisuation?

I do not know this is the real cause of ".msf" recreation problem.
But I experienced ".msf" recreation problem a few times.
If this type of ".msf" recreation can not be avoided, workaround I can imagine
is "to reduce Inbox size".
So I reduced Inbox folder size.

I have "Inbox2" folder for this puropse.
I move mails from my friends or known copmanies to categorized folders
automaticaly by message filter.
I also move almost all unfilterd mails to "Inbox2" manually.
And I use "Compact folder when it will save over nnn KB" option.
These keep Inbox size very small and ".msf" recreation time will become very
short even when ".msf" inconsistency problem occurs.

Viktors, try to reduce Inbox size.

Comment 6

[David :Bienvenu]

no, we have profile locking, which means that you can't start up mozilla on a
profile until the previous instance of mozilla has finished.

Christian, what's your setup? Linux...local file system or networked file system?

Comment 7

[Viktors]

For me, it happens occasionally on other mail folders as well, not just the
inbox. Also, that folder loses its sort order preference when this happens. 
Maybe it has something to do with mail filters moving incoming mail to them.

This reminds me of another annoyance where sometimes I don't seem to be able to
do the "run now" function when diddling with my message filters.  I cannot tell
if it the filter(s) are already running or are very very slow or if it is doing
nothing.  I found that when there are multiple "Received:" header fields, I
don't seem to be able to filter on their content.  Also, if there is a character
in them (or whatever does, length?) to cause them to display on more than one
line, I also cannot filter on that content.  When I do the Run Now function, I
was not ever sure if the filters were not running or if they simply didn't see
certain fields.  It would be nice to see some more positive notification about
the running filters.

Comment 8

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #6)
> no, we have profile locking, which means that you can't start up mozilla on a
> profile until the previous instance of mozilla has finished.

How about Mail&News only restart?
If Browser instance is still active, Mail&News starts up under same profile again. 

Comment 9

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #7)
> This reminds me of another annoyance
Bugzilla is not a BBS to complain of your annoyance.
This bug is to find the cause of your frequent ".msf" recreation problem and to
fix it if the casue is Mozilla's bug.
Search Bugzilla for other problems and open new bug if new problem.

> I found that when there are multiple "Received:" header fields, I don't seem
to be able to filter on their content.
See Bug 233484 and/or Bug 124641.

Comment 10

[David :Bienvenu]

Wada, no. Mail&News only restart is not a restart as far as the mailnews code is
concerned. The mailnews code keeps track of open db's and a subsequent request
to open a db returns the open db pointer. There's really no such thing as
"Mail&News restart."

Comment 11

[Christian Eyrich]

This very frequent rebuilds occur on my Linux installation. But the mailboxes
and .msf reside on a Windows FAT32 partition.

It's quite sure a rebuild will occur after I completely shut down Mozilla, then
start it again and get messages. Subfolders and Sent folder are often rebuilt
but don't follow this obvious pattern.

Automatic mail check isn't activated.

Comment 12

[David :Bienvenu]

Christian, that's probably because the time stamps aren't consistent between
linux and the FAT file system, if I've understood you correctly.

You could try setting the mail.db_timestamp_leeway to a few seconds and see if
it helps.

Comment 13

[WADA:World Anti-bad-Duping Agency]

David, same situation may occur when Mozilla on MS Windows and "Samba" on Unix
combination?

Comment 14

[David :Bienvenu]

Yes, any time you have two different OS's sharing a hard disk, or networked file
access.

Comment 15

[WADA:World Anti-bad-Duping Agency]

Different OS case only when networked file access?
Or even when same OS?

Comment 16

[David :Bienvenu]

any time the file system reports inconsistent time stamps. It's very simple:

we get the last modified timestamp of the mailbox file using nspr, in seconds,
and store that value in the .msf file.

If, the next time we open the mailbox, the last modified time stamp has changed
from what we stored in the .msf file, then we consider the .msf file to be
invalid because the mailbox has changed out from under us.

Comment 17

[Christian Eyrich]

David, I'll try this pref. But if I access the mailbox dor days only from Linux
can the timestamp really differ? Sorry if I didn't understood.

Comment 18

[David :Bienvenu]

I don't profess to understand why the time stamp differs, but if setting that
pref to a small number of seconds fixes the problem, then we'll know that's the
cause.  With these network file systems, there's a translation layer that I
assume is responsible for the deltas. The timestamp is stored on the native file
system, and translated when retrieved - my guess is that the initial time stamp
we get back after closing the mailbox file and asking for the last modified time
isn't correct, because some delayed write happens later after we close the file.

Comment 19

[Benedikt Kantus]

I am not sure whether this is of any help, but I experienced exactly the same
problem. The circumstances were, however, a little different:

Thunderbird nightly 20040524
Win2k
local file system

I do NOT see the problem with Thunderbird nightly 20040528.

Comment 20

[David :Bienvenu]

Benedikt, I believe that was just a one day regression in the db handling, in
the 05/23 build, and not related to this.

Comment 21

[Benedikt Kantus]

Maybe, I wasn't sure about that. Just wanted to let you know in case it might
help ;-)

Comment 22

[Stephen Bosch]

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040708

Either I am describing a variant of this bug or the shared filesystem is a red
herring.

I have this problem on Mozilla 1.7 on Linux. The profile is only ever used on
Linux. My mail folders tend to be large (500 - 8000 messages). (I suspect that
this is related to bug 180604).

Is there anything I can do to help fix this problem? It has been getting worse
(i.e., it happens more and more often) not better, with new releases of Mozilla.

Comment 23

[WADA:World Anti-bad-Duping Agency]

Change summary(add "different OS's sharing a hard disk, or networked file
access") according to comment #14 for ease of search.
(Old)
Frequent need to rebuild summary file for inbox
(New)
Frequent need to rebuild summary file(.msf) when mail folder is located on a
different OS's sharing hard disk or network drive(timestamp inconsistency).

Comment 24

[will smith]

I see this when using WinXP to share a samba drive, on Thunderbird 1.0 (do we
need a separate bug for Thunderbird?).  Moving the files locally is a
workaround, but this is not suitable in my case.

To try to eliminate the slightly out filetimes issue, I have set the user pref
mail.db_timestamp_leeway to 5 seconds (does this apply to Tbird?).  I have also
tried setting 'dos filetime resolution = yes' in my samba config file (smb.conf)
to try to relax the file timestamps issue at the samba end.   Neither works. 
The 'rebuilding summary file' at startup happens for any folder that was changed
during the previous time Thunderbird ran.

Comment 25

[will smith]

I see this when using WinXP to share a samba drive, on Thunderbird 1.0 (do we
need a separate bug for Thunderbird?).  Moving the files locally is a
workaround, but this is not suitable in my case.

To try to eliminate the slightly out filetimes issue, I have set the user pref
mail.db_timestamp_leeway to 5 seconds (does this apply to Tbird?).  I have also
tried setting 'dos filetime resolution = yes' in my samba config file (smb.conf)
to try to relax the file timestamps issue at the samba end.   Neither works. 
The 'rebuilding summary file' at startup happens for any folder that was changed
during the previous time Thunderbird ran.

Comment 27

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #25)
> Thunderbird 1.0
> I have set the user pref mail.db_timestamp_leeway to 5 seconds

As far as I know, mail.db_timestamp_leeway(default=4000) is available on 1.8.1 branch only (Tb 2.0 and Seamonkey 1.1 series only, not trunk yet).
I guess this is for testing of summer time case(and/or network drive case) by many users. 
Is mail.db_timestamp_leeway=4000 of Tb2.0/Sm1.1 effective when different OS case?

Comment 28

[David :Bienvenu]

I landed the leeway on the trunk a few days ago

Comment 29

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #28)
> I landed the leeway on the trunk
Thanks. I could see mail.db_timestamp_leeway=4000 with Tb trunk 2007/6/23 build. 

Comment 30

[Marc Rechté]

Hi,

I am using Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11.

My mail directory is located on a Samba share that is in synchronized with "My documents".

Since I am currently experimenting many Linux system crashes, it often happens that the share becomes "off-line" while TB is opened. In such case, the next synchronization will report that all the msf files have been modified from both side (share vs. local cache) and I know this is not the case.

In many instances, keeping the local version makes me loose all my Thunderbird folder setting (like retention rules).

This may confirm a bug in the timestamp usage on Windows platform (unless a bug on Samba ?).

Regards

Comment 31

[Wayne Mery (:wsmwk)]

Christian Eyrich, did you ever learn why comment 17 was happening for you?

Marc Rechté, in your case then, the time stamps have not changed, the only trigger you found is the share connection was lost?

http://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders?topic_tools=open

Comment 32

[Matthew Jurgens]

I have a similar sounding problem in TB3. I have described the problem in detail at http://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders
It possible that the leeway workaround/fix mentioned in this bug previously fixed the problem and maybe that's why I never had or have this problem with TB2.

However, a problem sounding very similar to this one appeared in TB3 as I have described. The leeway setting does not resolve it.

Comment 33

[Marc Rechté]

Yes. It is easy to reproduce (TB3.1):

1) Close thunderbird cleanly
2) Shutdown Samba (sahre is now offline)
3) Re-open TB
4) Close TB
5) Startup Samba
6) Synchronize: you are asked what to do for TB files which appear to have changed on both sides.

Comment 34

[crapo2]

I am running Thunderbird 3.1.8 on Ubuntu 10.04. I get email from a POP server at earthlink.net. I recently purchased a network drive hoping to be able to access email and documents from any computer on the home network. I mounted the network drive under my home folder. Initially I moved just my email account folders to the network drive and immediately saw the behaviors described here--very slow access, repeatedly rebuilding indexes. In fact, based on the progress bar, an index would often start to rebuild multiple times before finishing. In addition, any settings such as sort order of messages were not preserved on clean shutdown. After a little reading, I also moved my profiles to the network drive with no improvement. At this point I have moved everything back to my home folder. Thunderbird is entirely unusable with data on a network drive.

What I'm missing in this thread is any clear indication of whether allowing greater time leeway fixed anything for anyone and if so, exactly how to set that parameter.

Comment 35

[Wayne Mery (:wsmwk)]

samba users should refer to the details in Bug 624806 - Repeated rebuild of summary files. profile on Samba share - rather than this bug.

Comment 36

[John]

I have found that mounting a samba share with the noserverino option makes this problem go away. I have put the details about this under Bug 624806.

Comment 37

[Wayne Mery (:wsmwk)]

FWIW, Viktors (the reporter) no longer sees this problem)

Comment 38

[Matthew Jurgens]

I still see the problem with Thunderbird 17.
Even with very small folders. The leeway setting is at 4000 (but the client and host are NTP synced anyway). My data on Linux Samba share with Windows XP client.

Comment 40

[Wayne Mery (:wsmwk)]

(too many bug reports mention "sort", so removing "sort" from summary to make certain bug queries simpler)