Closed Bug 178641 Opened 22 years ago Closed 22 years ago

Problems with large mailboxes: 1) summary file is often rebuild, resulting is a large portion of the messages being marked us unread; 2) moved or deleted messages reappear, only 'compacting folder' seems to help here

Categories

(MailNews Core :: Database, defect)

Product:
MailNews Core
Component:
Database
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Chris, Assigned: Bienvenu)

References

Details

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021016
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2b) Gecko/20021016

I only have this problems on large mailboxes (> 5.000 mails, > 40 MB).

1) summary files are often rebuild, which results in the status of many of the
message in the folder being changed from read to unread.

2) when moving messages from one folder to another, I noticed that message
reappeared in the source folder after some time.  For example, I moved all
messages from folder A to B, A appears empty, but when I move the messages back
from B to A, all messages are twice in A!  The can easily be reproduced. 
Compacting the folder between the two moves 'solves' the problem.

Reproducible: Always

Steps to Reproduce:
(problem 2)
1. move a large number of mails from A to B
2. move them back from B to A
3. see the result...

Actual Results:  
mails are twice in folder A

Expected Results:  
:-)
are you putting messages in these mailboxes with another client? That would
explain the messages coming back as unread (because those messages would not
have an x-mozilla-status line, which is where we remember the read state), and
would explain why the summary files get rebuilt as well.

You may also have been running into problems caused by the auto-compaction code,
and the compact code in general, running at the same time as new messages arrive
into the folder. I believe Navin has fixed all of those now, though I thought
the 10/16 build had all the fixes.
No, all actions where done by this mail client only...

The automatic rebuilding of the summary files happens from time to time, without
any specific action from my side.  Most of the messages will be marked unread
afterwards, although they were already read and marked so by the same mozilla
mail client.  It's mostly the older messages that are marked unread, the newer
ones seem to retain their read status.  The problem happens about once a week
since I'm using the 1.2 builds, also still in this build.  Yesterday, after
opening the Local Folders Inbox, mozilla told me that it's rebuilding the
summary file and afterwards most messages were marked unread...

The deleted/moved messages that reappear could already have been solved in this
(1.2b) build, since I don't seem able to reproduce it myself now.
My guess is that the older messages don't have an x-mozilla-status line (perhaps
they were imported?). The summary file regeneration can be caused by having the
mail folders on a network file system, which wrecks havoc with our timestamp
code because of file date latency. If you could look at the messages in question
(e.g., with an editor, or view | msg source, you should be able to see if they
have an x-mozilla-status line or not.
True, I checked the message source and the older messages don't seem to have the
x-mozilla-status line in the header (these messages were idd imported by Mozilla
1.0).  Is there a way to ask mozilla to add status info to all messages (also
the imported ones)?

The mailboxes, however, are stored safely on my own hd (no network drive or
similar device).  So this can't be the reason for the regeneration of the
summary file...
QA Contact: gayatri → laurel
I'll have this issues as well and confirm all that is mentioned below.... just
with a fewer number of messages. If you need any additional input just let me
know. Tobias
Chris, there's no way I can think of to add the x-mozilla-status lines after the
fact, other than by hand. It used to be that compacting folders would do that,
but it doesn't anymore. I could look at making move/copying messages to other
folders add the x-mozilla-status line if it was not present in the source, which
would be a workaround. Some of the old import code did not add the
x-mozilla-status lines, which has since been fixed, but early adopters lost out :-(

So, I believe we're left with the summary file regeneration problem. I don't
know what's causing this. It would be helpful if you noticed some pattern of
what happened before the regeneration, like mozilla crashing, or linux crashing,
or some sort of user operation - for example, there was a time when mark thread
read could lead to invalid time stamps in the db, and subsequent regeneration
the next time mozilla was opened.
Status: UNCONFIRMED → NEW
Ever confirmed: true
does this happen after move/copying messages into a folder? E.g., startup, open
your inbox, drag/drop a few messages into a large folder, then open the large
folder? If so, I think I have a fix for this in  bug 169870.
I've checked in a couple fixes for what look like related bugs. Please let me
know if this still happens (actually, I haven't turned on the fix for linux, but
that one should only affect users with their profile on networked drives).
Status: NEW → RESOLVED
Closed: 22 years ago
Resolution: --- → FIXED
*** Bug 179695 has been marked as a duplicate of this bug. ***
Could you please enable the fix for Linux, too? I would be able to test. I've
seen this problem on my machine for quite some time, even though my mailbox is
on a local disk. Thanx.

I did check it in for linux as well last night.
I tried the version of this morning: The problem was still in. 

could you be a little more specific? What's the build version #? I ask because
people have been downloading the 1.2 builds and expecting them to have the fix,
but my fix is oly in 1.3 builds. And do you have any steps for reproducing the
problem?
It was the 1.3 version of about 7:00 or 8:00am. I remember the junk flags. Sorry
to be that unprecise, but I had to deinstall it again, cause composing EMails
did not work. Currently I am back to the latest 1.2 version. 

It was possible to reproduce the problem using version 1.3 2002111321: 

After exiting Mozilla I have manually removed a Mailfolder.msf file. Next I
called 'mozilla -mail' and selected Mailfolder. Some mails were marked read,
others were not. On the previous session all mails were marked read.
Harold, removing the .msf file will always cause it to be regenerated, by
definition. Please see http://bugzilla.mozilla.org/show_bug.cgi?id=178641#c3 as
to why the messages show up as unread again, after the summary files get
regenerated.  I'm just trying to fix it so that the summary files don't get
regenerated, barring the user deleting them, of course. So I'm looking for some
steps to reproduce the problem of the summary files getting regenerated, other
than the user deleting them.

The compose failure has been fixed from what I understand, and the fix should be
in today's 1.3 build.
Of course removing the SMF file is just the way to reproduce my problem. I get
the same effect without removing the SMF file, but then the problem occurs just
sometimes. People migrating from Netscape to Mozilla don't have any SMF files.
Probably they will stumble over the same problem for each of their mail folders,
so I would assume that this must be seen as a bug.
Sorry for the late reply.

bienvenu@netscape.com> does this happen after move/copying messages into a
folder? E.g., startup, open your inbox, drag/drop a few messages into a large
folder, then open the large folder?

Yes, I think it always happens after one of the described actions.

I have upgraded my build to 2002111422, and I haven't had the problem today
after a lot of moving messages around...

Chris.
Great, thx, Chris.

Harold, yes, it's a bug when summary files get regenerated (usually). But what
we need to know is how to cause the summary files to need to be regenerated in
the first place. 

(btw, People migrating now should have their old mail folders imported
correctly, because the bug with the mozilla-status lines not getting added has
been fixed, I believe. I realize that's no consolation for you :-( )

Also, this fix made it into the most recent 1.2 build.
*** Bug 180687 has been marked as a duplicate of this bug. ***
*** Bug 181032 has been marked as a duplicate of this bug. ***
qa contact change
QA Contact: laurel → esther
*** Bug 207038 has been marked as a duplicate of this bug. ***
On the basis of new bugs being duped against a supposedly fixed bug, shouldn't
this bug be reopened?
In particular I'm referring to bug Bug 207038 which I've been experiencing.
As I already wrote to Jo Hermanns I have this behavior with Mozilla 1.3 and not
with 1.2, so I think this is a new bug
btw: there are 2 other Bugs that may relate to this: 105816 and 93271, but there
nobody seems to have interest in solving this issues :(
Product: MailNews → Core
Product: Core → MailNews Core
You need to log in before you can comment on or make changes to this bug.