641583 - Mail index gets corrupted when an external process randomly delets mbox files.

Status: RESOLVED INVALID

(SeaMonkey :: MailNews: Message Display, defect)

Description

[Rob Janssen]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; nl-NL; rv:1.9.1.17) Gecko/20110123 SeaMonkey/2.0.12
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; nl-NL; rv:1.9.1.17) Gecko/20110123 SeaMonkey/2.0.12

We are using Mozilla Seamonkey in our organization for a long time, with about 400 users.
Recently we upgraded from version 1.1.19 to version 2.0.11.
From that moment we get several reports a day from users complaining that they click on a message line in the thread pane, and get a different message displayed.
When we instruct them to rebuild the index, the situation is OK again.
But it often happens again for the same user later.
This weekend we updated from 2.0.11 to 2.0.12, but if anything changed the problem became worse!
HELP!  What can be going on here.  This problem almost never occurred in 1.1.19 and now it is very frequent.  The fact that a handy feature to rebuild the index has been added to the 2.0 series is convenient, but at the same time worrying.  Is the indexing now so unreliable that index files need to be rebuilt all the time?

The users all have IMAP accounts.  The server is UW IMAPD.  It was stable for 10 years.


Reproducible: Sometimes

Steps to Reproduce:
Cannot be reproduced on demand.  Happens several times a day in a 400 user population.
Actual Results:  
When clicking on a message, another message in the list is displayed, or a number of messages appear in one view (with headers in between).

Expected Results:  
Of course, the message clicked on should appear.

Default theme.
May be tied to pecularties tied to user or user's mailbox, as I have not seen it occur on my own system and it looks like affected users get the problem over and over again.

Comment 1

[Rob Janssen]

I think I know why it happens...
In version 1.1.19, for an IMAP account, the tree under ImapMail in the profile contained only .msf files and the message filter rules.
After changing to 2.0.11 we see files appear there that contain (part of) the contents of the mail folders on the server.  A kind of message cache?
They do not always appear, but only for some users (apparently only under certain circumstances).
In the past, I have seen them when folders were marked for offline use.  But now it seems they get created for other uses as well?

We don't want those files.  They contain message data that must not be kept on the workstation, only on the server.  They increase the size of the roaming profile, for no good purpose.
So I have written a logoff script that deletes the data files (e.g. INBOX) while keeping the corresponding .msf files (e.g. INBOX.msf).
It looks like this deletions, although they seemed to cause no immediate problems, are the root cause of the corruptions we see later.  Maybe the .msf file contains offsets in the data files where it is assumed a certain message starts.

So the real question is: how can we permanently disable the creation of these "cache" files that contain messages retrieved from an IMAP server?
When the files are not created by Seamonkey, there probably will be no problems anymore.  But I have not yet found a setting in about:config that could likely switch this "feature" on/off.

Again, those files were never created in Seamonkey 1.1 unless offline folders were selected, but it seems now they are.

Comment 2

[Philip Chee]

The default in current versions of SeaMonkey is to mark all IMAP folders as offline folders. You could go go through all those folders and remove the offline setting.

Also if you delete the data file, you should also delete the index (.msf) files.
> Maybe the .msf file contains offsets in the data files where it is assumed a
> certain message starts.
This is correct. the .msf files also contain some metadata such as whether a received/read notification has been sent back to the sender.

Comment 3

[Rob Janssen]

Please add some preference to define that default to be turned around...
We want SeaMonkey to do no caching.  All messages are to be retrieved from the mail server when read, not from the local disk.  And are not to be stored there.
Storing mail in the profile is a security and privacy risk.  Maybe it has a performance advantage, but users should be enabled to favor security/privacy over performance.

(with a preference value I can centrally enable this feature in the organization.  it is not reasonable to require the users to arrange this via the GUI)

Comment 4

[Philip Chee]

> Please add some preference to define that default to be turned around...
Preferences exist. Please see:
<http://kb.mozillazine.org/Minimize_the_size_of_a_profile>

This knowledgebase article is for Thunderbird but most of it applies to SeaMonkey as well. Except:
1. Our menu item is Edit->Mail and Newsgroups account settings.
2. I was wrong, you should Leave the *.msf files alone.
3. Global searching/indexing: This is turned off in SeaMonkey and can't be turned on so you can ignore that paragraph.

Comment 5

[Rob Janssen]

Thank you.
Well, I have locked many preferences in the past (including those that are described in this article) to attempt to completely turn off the local caching of mail, but it looks like there is still the possibility that a user switches it on for an individual folder via the GUI, and it gets stored somewhere where it cannot be manipulated by the admin (i.e. not in prefs.js or another plain text file)
It would be nice if there was a preference to completely disable all caching.

Comment 6

[Wayne Mery (:wsmwk)]

> click on a message line in the thread pane, and get a different message displayed.

I doubt that offline storage/autosync plays a role here. (but it might)

Has your testing proven this theory?

Comment 7

[Rob Janssen]

As I explained, the root cause of the problem is the unfortunate decision to cache all IMAP
mail data without a lockable preference to disable this.
We DON'T WANT to cache the mail.  So I tried all kinds of approaches to disable it and it
caused the problem mentioned at the start of this bug report.

I still think a pref should be available (lockable) to disable IMAP caching.

Some time ago I discovered that I can delete a mailfolder file, its corresponding .msf file,
and copy a pristine panacea.dat file over the current one, to combat the problem of mail
folders that get cached due to some user action.  Apparently the "being cached" state of
a folder is stored in panacea.dat.

The default is locked using:

lockPref("mail.server.default.autosync_offline_stores", false);
lockPref("mail.server.default.offline_download", false);
lockPref("offline.autoDetect", false);
lockPref("offline.download.download_messages", 0);
lockPref("offline.startup_state", 2);

but still sometimes users manage to click somewhere and cause a mailfolder to be stored.
it gets thrown away from the profile during a nightly job (as described above) and the
next day the user copies the cleaned profile and usually the problem is gone.

Comment 8

[Wayne Mery (:wsmwk)]

I understand your desire. But it's NOT at all clear to me that that disabling sync will avoid your  "corruption" issue.

Comment 9

[Rob Janssen]

Please read the original report and comment 1.
When the caching could be permanently disabled, no cache files would have existed, there
would have been no reason for me to delete them, and the corruption would not have occurred.
The corruption is caused by deleting INBOX while not deleting INBOX.msf.
When both files are deleted or both are left, there is no corruption.

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Rob Janssen from comment #9)
> Please read the original report and comment 1.
> When the caching could be permanently disabled, 

This was actually tested, that you don't experience the problem when sync/offline storage is disabled?
If so, this suggests compact operation might be causing the problem.

Do you still get complaints of users get a different message displayed.

Comment 11

[Rob Janssen]

This ug report is from 5 years ago.
We have stopped using the program in the company so I cannot comment on that anymore.

Comment 12

[Wayne Mery (:wsmwk)]

thanks for the feedback.
perhaps then close incomplete because we lack the testcase. But I'll set ratty make the call

Comment 13

[Rob Janssen]

The testcase is described in comment 1.
The solution is a preference to make it stop caching mail when the admin does not want
local e-mail caching.
With the new legislation as it is here starting 1-1-2016, caching e-mail on a workstation
in a company is an ABSOLUTE NO-NO.
Does not matter if a programmer thinks it will improve efficiency, it just SHOULD NOT HAPPEN.
So it has to be configurable like that, or else out it goes.
(which in fact has already happened for us, but that was for other reasons)

Comment 14

[Wayne Mery (:wsmwk)]

I was commenting only on the dataloss.

I'm not in a position to morph this to providing UI in Seamonkey to address the caching you are complaining about - someone else will need to do it. And in that case, the bugzilla component would also need to change

Comment 15

[Philip Chee]

I'm on Windows 7. I don't see any cached IMAP data in Local and Local Low. The profile IMAP data is all in Roaming. In a networked environment Roaming should be mapped to a server. However the Original reporter says he has stopped using SeaMonkey so I'll close this as won't fix if nobody objects.

Comment 16

[Wayne Mery (:wsmwk)]

Yeah, he's not complaining about the traditional store, but what's in roaming next to the .msf files
Does SM download all the imap mail bodies by default?

Comment 17

[Philip Chee]

> Does SM download all the imap mail bodies by default?
IMAP bodies are downloaded if the IMAP folder is marked for offline use. Just like Thunderbird. In a networked environment the sysadmin should map Roaming to a networked share (Why did you think Microsoft invent Roaming?). It's true that Windows XP didn't have this but the OP should have migrated to a newer OS by now.

External processes randomly deleting mbox files is definitely not a supported environment. If Rob Janssen doesn't want to do that he should set up an internal webmail server. Zimbra comes to mind.