Frequent need to rebuild summary file(.msf) with folder on a different OS's sharing hard disk or network drive (timestamp inconsistency), often when share goes "off-line". Causes lost folder column/sort settings.

NEW
Unassigned

Status

--
critical
15 years ago
a year ago

People

(Reporter: j011112mz, Unassigned)

Tracking

({dataloss, perf})

Trunk
x86
All
dataloss, perf

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [TB3 and newer: see bug 624806][preceeds TB3][steps comment 33])

(Reporter)

Description

15 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.6) Gecko/20040113
Build Identifier: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.6) Gecko/20040113

In past versions of Mozilla (before 1.6) I noticed that occasionally it would
spend a lot of time rebuilding the summary file for my inbox upon opening the
email windows.  Now with 1.6, it is happening almost every time even though I
shut down cleanly.  I have about 4803 messages in my inbox so this takes many
minutes to rebuild.  Why is this happening so often?

Reproducible: Sometimes

Steps to Reproduce:
1. Shutdown system (with or without closing Mozilla first)
2. Boot system
3. Start Mozilla with -mail parameter

Actual Results:  
It spends a lot of time (several minutes) rebuilding the summary file for my
inbox and also forgets that I used to have it sorted by date with the newest on top.

Expected Results:  
Why does it think the summary file needs rebuilding so often?  Make it not
require this unless there is a system crash or something more serious.

I have 4803 messages in my inbox currently and it is this big:

INBOX          215,467,800  01-24-04 10:32a Inbox
INBOX    MSF     2,138,693  01-24-04 10:32a Inbox.msf

Comment 1

15 years ago
First of all, you should compact your INBOX, if you haven't done so recently (or
ever). Just use the folder pane context menu and pick "Compact this Folder".

Second, I don't know why the summary file would get regenerated so often. It
also shouldn't forget your sort order; I thought I had fixed that.

If you can figure out some pattern to what causes this to happen, that would be
useful. What's the last thing you do in the inbox before you shutdown the app?
(Reporter)

Comment 2

15 years ago
I compacted the file and it has not happened again so far.  I have restarted
Mozilla twice and rebooted once.

I usually visit each folder that is marked with bold and either delete unwanted
messages, view them, or ctrl-shift-C to mark them all as read.  I usually empty
the trash last.  If emptying the trash does not unbold the trash bin, I visit
the trash bin and this unbolds it.  Then I am done and shutdown the system or
close Mozilla.  I am not using the fast start feature.
(Reporter)

Comment 3

15 years ago
I wanted to add that I am still getting rebuilds from time to time even though I
have compacted the folders.

Comment 4

15 years ago
I do get rebuilds a couple of times each day too. Compacting doesn't help.
Automatic mail check task keeps alives after Mozilla Mail&News window termination.
This may cause ".msf" file contention between next mail check on Mail&News start up.
If the contention occurs, or if start up of Mail&News cancels old mail check
task while the old mail check task is trying to down load mail, it possibly
causes  ".msf" file inconsistency.
Then ".msf" recreation possibly occurs.

David Bienvenu and Christian Eyrich, can ".msf" inconsistency and ".msf"
recreation occur in this sisuation?

I do not know this is the real cause of ".msf" recreation problem.
But I experienced ".msf" recreation problem a few times.
If this type of ".msf" recreation can not be avoided, workaround I can imagine
is "to reduce Inbox size".
So I reduced Inbox folder size.

I have "Inbox2" folder for this puropse.
I move mails from my friends or known copmanies to categorized folders
automaticaly by message filter.
I also move almost all unfilterd mails to "Inbox2" manually.
And I use "Compact folder when it will save over nnn KB" option.
These keep Inbox size very small and ".msf" recreation time will become very
short even when ".msf" inconsistency problem occurs.

Viktors, try to reduce Inbox size.

Comment 6

15 years ago
no, we have profile locking, which means that you can't start up mozilla on a
profile until the previous instance of mozilla has finished.

Christian, what's your setup? Linux...local file system or networked file system?
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Reporter)

Comment 7

15 years ago
For me, it happens occasionally on other mail folders as well, not just the
inbox. Also, that folder loses its sort order preference when this happens. 
Maybe it has something to do with mail filters moving incoming mail to them.

This reminds me of another annoyance where sometimes I don't seem to be able to
do the "run now" function when diddling with my message filters.  I cannot tell
if it the filter(s) are already running or are very very slow or if it is doing
nothing.  I found that when there are multiple "Received:" header fields, I
don't seem to be able to filter on their content.  Also, if there is a character
in them (or whatever does, length?) to cause them to display on more than one
line, I also cannot filter on that content.  When I do the Run Now function, I
was not ever sure if the filters were not running or if they simply didn't see
certain fields.  It would be nice to see some more positive notification about
the running filters.
(In reply to comment #6)
> no, we have profile locking, which means that you can't start up mozilla on a
> profile until the previous instance of mozilla has finished.

How about Mail&News only restart?
If Browser instance is still active, Mail&News starts up under same profile again. 
(In reply to comment #7)
> This reminds me of another annoyance
Bugzilla is not a BBS to complain of your annoyance.
This bug is to find the cause of your frequent ".msf" recreation problem and to
fix it if the casue is Mozilla's bug.
Search Bugzilla for other problems and open new bug if new problem.

> I found that when there are multiple "Received:" header fields, I don't seem
to be able to filter on their content.
See Bug 233484 and/or Bug 124641.

Comment 10

15 years ago
Wada, no. Mail&News only restart is not a restart as far as the mailnews code is
concerned. The mailnews code keeps track of open db's and a subsequent request
to open a db returns the open db pointer. There's really no such thing as
"Mail&News restart."

Comment 11

15 years ago
This very frequent rebuilds occur on my Linux installation. But the mailboxes
and .msf reside on a Windows FAT32 partition.

It's quite sure a rebuild will occur after I completely shut down Mozilla, then
start it again and get messages. Subfolders and Sent folder are often rebuilt
but don't follow this obvious pattern.

Automatic mail check isn't activated.

Comment 12

15 years ago
Christian, that's probably because the time stamps aren't consistent between
linux and the FAT file system, if I've understood you correctly.

You could try setting the mail.db_timestamp_leeway to a few seconds and see if
it helps.
David, same situation may occur when Mozilla on MS Windows and "Samba" on Unix
combination?

Comment 14

15 years ago
Yes, any time you have two different OS's sharing a hard disk, or networked file
access.
Different OS case only when networked file access?
Or even when same OS?

Comment 16

15 years ago
any time the file system reports inconsistent time stamps. It's very simple:

we get the last modified timestamp of the mailbox file using nspr, in seconds,
and store that value in the .msf file.

If, the next time we open the mailbox, the last modified time stamp has changed
from what we stored in the .msf file, then we consider the .msf file to be
invalid because the mailbox has changed out from under us.

Comment 17

15 years ago
David, I'll try this pref. But if I access the mailbox dor days only from Linux
can the timestamp really differ? Sorry if I didn't understood.

Comment 18

15 years ago
I don't profess to understand why the time stamp differs, but if setting that
pref to a small number of seconds fixes the problem, then we'll know that's the
cause.  With these network file systems, there's a translation layer that I
assume is responsible for the deltas. The timestamp is stored on the native file
system, and translated when retrieved - my guess is that the initial time stamp
we get back after closing the mailbox file and asking for the last modified time
isn't correct, because some delayed write happens later after we close the file.

Comment 19

15 years ago
I am not sure whether this is of any help, but I experienced exactly the same
problem. The circumstances were, however, a little different:

Thunderbird nightly 20040524
Win2k
local file system

I do NOT see the problem with Thunderbird nightly 20040528.

Comment 20

15 years ago
Benedikt, I believe that was just a one day regression in the db handling, in
the 05/23 build, and not related to this.

Comment 21

15 years ago
Maybe, I wasn't sure about that. Just wanted to let you know in case it might
help ;-)
Product: MailNews → Core

Comment 22

14 years ago
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7) Gecko/20040708

Either I am describing a variant of this bug or the shared filesystem is a red
herring.

I have this problem on Mozilla 1.7 on Linux. The profile is only ever used on
Linux. My mail folders tend to be large (500 - 8000 messages). (I suspect that
this is related to bug 180604).

Is there anything I can do to help fix this problem? It has been getting worse
(i.e., it happens more and more often) not better, with new releases of Mozilla.
Change summary(add "different OS's sharing a hard disk, or networked file
access") according to comment #14 for ease of search.
(Old)
Frequent need to rebuild summary file for inbox
(New)
Frequent need to rebuild summary file(.msf) when mail folder is located on a
different OS's sharing hard disk or network drive(timestamp inconsistency).
OS: Windows 98 → All
Summary: Frequent need to rebuild summary file for inbox → Frequent need to rebuild summary file(.msf) when mail folder is located on a different OS's sharing hard disk or network drive(timestamp inconsistency)

Comment 24

14 years ago
I see this when using WinXP to share a samba drive, on Thunderbird 1.0 (do we
need a separate bug for Thunderbird?).  Moving the files locally is a
workaround, but this is not suitable in my case.

To try to eliminate the slightly out filetimes issue, I have set the user pref
mail.db_timestamp_leeway to 5 seconds (does this apply to Tbird?).  I have also
tried setting 'dos filetime resolution = yes' in my samba config file (smb.conf)
to try to relax the file timestamps issue at the samba end.   Neither works. 
The 'rebuilding summary file' at startup happens for any folder that was changed
during the previous time Thunderbird ran.

Comment 25

14 years ago
I see this when using WinXP to share a samba drive, on Thunderbird 1.0 (do we
need a separate bug for Thunderbird?).  Moving the files locally is a
workaround, but this is not suitable in my case.

To try to eliminate the slightly out filetimes issue, I have set the user pref
mail.db_timestamp_leeway to 5 seconds (does this apply to Tbird?).  I have also
tried setting 'dos filetime resolution = yes' in my samba config file (smb.conf)
to try to relax the file timestamps issue at the samba end.   Neither works. 
The 'rebuilding summary file' at startup happens for any folder that was changed
during the previous time Thunderbird ran.
Duplicate of this bug: 283490
(In reply to comment #25)
> Thunderbird 1.0
> I have set the user pref mail.db_timestamp_leeway to 5 seconds

As far as I know, mail.db_timestamp_leeway(default=4000) is available on 1.8.1 branch only (Tb 2.0 and Seamonkey 1.1 series only, not trunk yet).
I guess this is for testing of summer time case(and/or network drive case) by many users. 
Is mail.db_timestamp_leeway=4000 of Tb2.0/Sm1.1 effective when different OS case?

Comment 28

12 years ago
I landed the leeway on the trunk a few days ago
(In reply to comment #28)
> I landed the leeway on the trunk
Thanks. I could see mail.db_timestamp_leeway=4000 with Tb trunk 2007/6/23 build. 

Comment 30

11 years ago
Hi,

I am using Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11.

My mail directory is located on a Samba share that is in synchronized with "My documents".

Since I am currently experimenting many Linux system crashes, it often happens that the share becomes "off-line" while TB is opened. In such case, the next synchronization will report that all the msf files have been modified from both side (share vs. local cache) and I know this is not the case.

In many instances, keeping the local version makes me loose all my Thunderbird folder setting (like retention rules).

This may confirm a bug in the timestamp usage on Windows platform (unless a bug on Samba ?).

Regards
QA Contact: esther → database
(Assignee)

Updated

11 years ago
Product: Core → MailNews Core

Updated

9 years ago
Severity: normal → major
Keywords: perf
Christian Eyrich, did you ever learn why comment 17 was happening for you?

Marc Rechté, in your case then, the time stamps have not changed, the only trigger you found is the share connection was lost?

http://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders?topic_tools=open
Whiteboard: [gs]

Comment 32

8 years ago
I have a similar sounding problem in TB3. I have described the problem in detail at http://getsatisfaction.com/mozilla_messaging/topics/tb3_repeatedly_rebuilding_summary_files_for_many_folders
It possible that the leeway workaround/fix mentioned in this bug previously fixed the problem and maybe that's why I never had or have this problem with TB2.

However, a problem sounding very similar to this one appeared in TB3 as I have described. The leeway setting does not resolve it.

Comment 33

8 years ago
Yes. It is easy to reproduce (TB3.1):

1) Close thunderbird cleanly
2) Shutdown Samba (sahre is now offline)
3) Re-open TB
4) Close TB
5) Startup Samba
6) Synchronize: you are asked what to do for TB files which appear to have changed on both sides.

Comment 34

8 years ago
I am running Thunderbird 3.1.8 on Ubuntu 10.04. I get email from a POP server at earthlink.net. I recently purchased a network drive hoping to be able to access email and documents from any computer on the home network. I mounted the network drive under my home folder. Initially I moved just my email account folders to the network drive and immediately saw the behaviors described here--very slow access, repeatedly rebuilding indexes. In fact, based on the progress bar, an index would often start to rebuild multiple times before finishing. In addition, any settings such as sort order of messages were not preserved on clean shutdown. After a little reading, I also moved my profiles to the network drive with no improvement. At this point I have moved everything back to my home folder. Thunderbird is entirely unusable with data on a network drive.

What I'm missing in this thread is any clear indication of whether allowing greater time leeway fixed anything for anyone and if so, exactly how to set that parameter.
samba users should refer to the details in Bug 624806 - Repeated rebuild of summary files. profile on Samba share - rather than this bug.

Comment 36

7 years ago
I have found that mounting a samba share with the noserverino option makes this problem go away. I have put the details about this under Bug 624806.

Updated

7 years ago
Assignee: dbienvenu → nobody
FWIW, Viktors (the reporter) no longer sees this problem)

Comment 38

6 years ago
I still see the problem with Thunderbird 17.
Even with very small folders. The leeway setting is at 4000 (but the client and host are NTP synced anyway). My data on Linux Samba share with Windows XP client.

Updated

6 years ago
See Also: → bug 624806

Updated

5 years ago
Severity: major → critical
Keywords: dataloss
Summary: Frequent need to rebuild summary file(.msf) when mail folder is located on a different OS's sharing hard disk or network drive(timestamp inconsistency) → Frequent need to rebuild summary file(.msf) with folder on a different OS's sharing hard disk or network drive (timestamp inconsistency), often when share goes "off-line". Causes lost folder column/sort settings.
Whiteboard: [gs] → [TB3 and newer: see bug 624806][preceeds TB3][steps comment 33]
Duplicate of this bug: 499689
You need to log in before you can comment on or make changes to this bug.