Closed Bug 267991 Opened 20 years ago Closed 15 years ago

mail message filter rules suddenly disappeared. No entries are found on startup after disk shortage

Categories

(MailNews Core :: Filters, defect)

Product:
MailNews Core
Component:
Filters
Version:
1.7 Branch
Platform:
x86
Linux
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ishikawa, Unassigned)

Details

(Keywords: dataloss)

Attachments

(1 file)

Dubious processing of write when ENOSPC is encountred.
7.55 KB, text/plain
Details 
User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20040913
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20040913

A couple of days ago, I modified filter rules.
I probably had about 20 rules for classifying messages based on
various conditions.

Today, I found that the rules were gone. When I checked with "Message Filters"
menu, I found NO entry at all!

The only reason I could think why mozilla could lose the rules is:
- I may have experienced a low disk space state for a partition where
  the mail folders reside. I ran compact folders and one of the folders
  was quite large and the disk remaining space became low although not zero.

- Temporary directory (/tmp) became full during I was playing with mozilla.
  This was due to a large update of Linux debian package update.
  I purged the old packages and then the partition including /tmp had
   free space.

 I suspect that during the time /tmp was empty ( I didn't notice it
until mozilla failed to send out a message. Then I checked around
and noticed that root partition where my /tmp was located was empty.
And then realized apt-get under linux to update Debian linux packages
had run out of space.

Tentative conclusion.

The writing of filter rule to external data file is not
very cautions about file I/O errors.
I think the zero space condition could have been
caught with careful check of return values of
 fwrite(),
 write()
 fclose()
 close()
 fsync() or fdatasync().

This is a critical show stopper bug for me...

I am trying to salvage old filter rules from a backup tape, but now
I don't know where the rules are stored...

 

Reproducible: Always
Steps to Reproduce:
1.
2.
3.
Product: MailNews → Core
Reporter - do you use imap or pop for your email (or even both?). I have seen
something that may be similiar, but I can't confirm it yet.
See Release Notes for file name.
 http://www.mozilla-japan.org/releases/mozilla1.7/installation.html#files
msgFilterRules.dat in each directry for account under mail directry.

By the way, real msgFilterRules.dat problem?
What will happen if you resize message filter panel which displays no rule.
I experienced "no rule in filter definition panel" some days ago, and I was in
panic.
But when I accidentaly moved and resized the panel, filter rules were displayed
again.
(This happened on latest trunk nightly of Win-2K)
Hi,

Thank you for the comments.

Answer to comment 1
   I use POP on the PC where the problem occurred.

Answer to comment 2

   Yes, the problem was real.
   The file msgFilterRules.dat was completely gone.

   I had to re-suurrect the file from my backup tape.

   Since initially I didn't know the name, I ran the following command
   first to figure out what the file could be.

   find my-mozilla-preference-directory my-mozilla-mail-directory \
   -print | grep -i filter

   ( msgFilterRules.dat for my POP account was missing.
     There was one msgFilterRules.dat for my news account which
     was empty.)

   I then knew probably msgFilterRules.dat was looking for.

   I also did
   tar tjf my-tape-drive-name | egrep -i filter

   After learning that there was msgFilterRules.dat in my backup file,
   I retrieved it from the tape.
   I copied it to the appropriate place. (Again, the file was
   completely gone previously.)

   Then I started mozilla, and voila the old filter rules were
   there again.

   So the problem was real.

I should have stated "my /tmp stage was full and there was no
free space" instead of saying "/tmp was empty" in my original post.
I can see that file save / update code might fail under such conditions
if not written *VERY* carefully.

(My home directory and my mozilla mail directory are on different partition 
from that of /tmp, / which became full. So I suspect that the creation of
temporary copy of msgFilterRules.dat to update it failed. But
this is my pure guess.)
(In reply to comment #3)
>    So the problem was real.
> 
> I should have stated "my /tmp stage was full and there was no
> free space" instead of saying "/tmp was empty" in my original post.

Probably, "safety file writing" like Bug 250092(prefs.js case) is also required
for msgFilterRules.dat.
Thanks for the pointer to bug 250092.

Yes, I think something similar to the handling of message filter file
msgFilterRules.dat
is necessary if it is not done yet.

(Hmm. I didn't realize that the safe user configuration data update/save
is so messy. In most programs, it is simply a matter of
making a backup of copy of the user configuration data [if this fails,
then warn and quit.] if necessary. [Usually it is not.]

create the new configuration file based on user data under a new file name
[if this fails, then warn and quite.],

then rename the original under backup file name [if this fails, then
warn and quit],
then rename the new file under the original file's name [ if this fails,
then warn and quit. We have to warn the user that the original may
be found under backup filename.],
if the last step is successful, the whole operation is a success.

Conceptually, quite simple. We DO need to check write(), close(), fsync(), and
other function values along the way, but such care is well understood.

I am a little afraid that the thread 250092 didn't mention fsync() and friends
at all. This may bite us in the long run ;-(
fclose(), close() simply writes the file contents to the
internal page/buffer cache.
Later when the system buffer is flushed, we may run out of space. Too late :-(
This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

The BUG *IS* still there!

Firebird 1.0.6

The problem returns:
when free space in /tmp partition is very low (almost nil), then
this problem of losing the previous msgFilterRules.dat 
is still there.
I suddenly lost the file again yesterday when
an eager installation process of one free software package
used up remaining free space in /tmp.
df showed 0 K bytes left in /tmp.

The firebird program at least should be careful
NOT to lose the original msgFilterRules.dat file when
a rewrite incorporating a modification to the rules
failed ( write() system call or similar fails and/or
the resulting file size is 0 for some reason or the other!).

There have been editor programs in the past with similar problems
when /tmp, /var/tmp, or partitions that contains directories
pointed by the enviornment variables TMP/TMPDIR have close to zero 
free space. They have been fixed by not to lose the original file
and warn the user to save the contents of the edit buffer to 
a partition that had space left in it.




>The BUG *IS* still there!

>Firebird 1.0.6

I meant to say THUNDERBIRD 1.0.6.

Sorry for the confusion!


I created a simple test case and attach an excerpt from a log file.
(I edited the log file in a few places.)


How I re-created the problem:

Since I figure that the overflowing of temporary file directory (i.e.,
/tmp) and thunderbird is not robust enough, I created
a small loopback device and used it as temporary directory.
This is under Linux, but I assume this would manifest under
other UNIX-like OSs, Solaris, *BSD, etc.

I created a small loop device (100KB) and mounted it to be used
as the loopback device.

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/loop0                  93        90         0 100% /mnt

I defined environment variable TMP, TMPDIR
to be
TMP=/mnt
TMPDIR=/mnt
export TMP TMPDIR

I copied a large file to /mnt so that there is no space left on it.
(Well, from the ordinary user's viewpoint, that is. Probably
superuser, root, still can write a few more KBs to it.)

/dev/loop0                  93        90         0 100% /mnt

Then I started firebird as an ordinary user  and modified filter definition.

I edited an existing filter and added one condition "Subject contains
something" and then removed it and closed thunderbird.
I repeated this a few times. 
And sure enough, after a few tries, I could re-create the "msgFilterRules.dat file
disappearing" problem nicely.
When I restarted thunderbird, I found there is no filter definition(!).
And I found that msgFilterRules.dat is completely gone!

(To recover from this, I stopped thunderbird. I had saved a copy under
my home directory and copied it back to where it should be.  When I
restarted thunderbird, the filter was back. I learned this the hard
way when I first reported this problem.)

Now, I captured the system calls during the operation and
exerpted a few suspicious operations. The attachment is this
excerpt.

It seems that when ENOSPC error code is returned from a write system
call (to temporary rule defintion files /tmp/tmprules.dat) and that
the temporary copy of filter rules is incomplete, or for that matter
not created at all(?), thunderbird still seems to try to copy the
temporary to the msgFilterRules.dat file by renaming it.  (I believe
the programmers who know in and out of the source code can confirm the
general outline of the operation, and find where the appropriate error
handling is missing.)


I still don't get why msgFilterRules.dat is gone (that is it doesn't
exist any more.) I can understand if it remains as a 0-byte file.
(I think when thunderbird reads in empty msgFilterRules.dat file on the
next restart, and the user tries to fiddle with it, probably it
unlinks the file? I think that is why I found the msgFilterRules.dat
file missing from below ~/.thunderbird/blahblah/Mail/domainname/ directory.
(Sorry, I didn't capture that part of system calls, i.e, the system
calls after the invocation of Thunderbird: this was when I saw empty
rules. I checked the directory afterward and found msgFilterRules.dat
file is no where to be found.)


In any case, I think the attached log files is very clear on the problem.
If ENOSPC occurs, then NO copy should be attempted from temporary
files, that is /mnt/tmprules.dat in this case,
and instead ThunderBird should try to preserve the orignal.
I have no idea how to attract attention to this problem.
The bug now is 
 - reproducible under test condition.

 - has now a system-call log that shows how the mozilla
   mis-behaves under low termporary file space conditions
   and trash msgFilterRules.dat file.

So I am trying to change this to "confirm/reassign" although it may not work for a
mere luser like me.


Version 1.5.0.4

I hate to report this, but the strange disapperance of msgFilterRules.dat is
still observed. I think the proper attention to take care of this file
when we make changes to the filter rule set
is in order. 

I have already outlined the
problems, possible culprit under linux at least (error code ENOSPC is not
handled properly), and reported the continueing intermittant
problems when the disk space is low.

I wonder why ENOSPC is not properly handled back in the older version,
and I am afraid the bug is still there.

I am afraid that the patch proposed for similar bugs in 250092 in comment #4 is
not correct.
At least, on the linux kernel which I tested,
write() returned ENOSPC when the filesystem was full, but
close() returned 0, i.e., succeeded despite the write failure(!).

The patch proposed in 250092 didn't look at the write() value [it seems to me]
and ONLY CHECKED the CLOSE CALL  VALUE at the end. This is totally bogus on
such a system!!

Can anyone point to the source file name which handles the saving of msgFilterRules.dat?
I may take a look at it and point to the problematic code there.
Then someone can produce the patch (it could be me) and then the patch
can be included in the code base.

I am tired of this and similar problems of ThunderBird which occur when the filesystem on my  note PC is close to overflow. (Yeah, I lost a few e-mail archives in the last 12 months, but the backup saved my day.)

Anyone for help?

AGAIN, MY POINT: when the filesystem is full (for temp files), write() returns error, but close() doesn't [at least on some linux kernel versions]. Any code
that ignores the intermediate write() return values and only depends on the
final return value of close() is BOGUS and FATALLY BUGGY...



perhaps Ben or dwitte can offer some advice

perhaps this not is the best component to assign to. what about RDF or file handling?

there is a boatload of disk space-related bugs to consider:
 bug 169220, bug 202912, bug 169220, bug 226970, bug 327610, bug 346003 	
and bug 226970, bug 102030, bug 353462, bug 75440, bug 174818 and others listed in http://tinyurl.com/ymozhz
Keywords: dataloss
Summary: mail message filter rules suddenly disappeared. No entries are found on startup. → mail message filter rules suddenly disappeared. No entries are found on startup after disk shortage
Version: Trunk → 1.7 Branch
sorry for the spam.  making bugzilla reflect reality as I'm not working on these bugs.  filter on FOOBARCHEESE to remove these in bulk.
Assignee: sspitzer → nobody
based on my brief reading of the bug, if anyone wants to patch this, nsISafeOutputStream is the way to go. see

interface: http://mxr.mozilla.org/mozilla/source/netwerk/base/public/nsISafeOutputStream.idl
example usage: http://mxr.mozilla.org/mozilla/source/extensions/cookie/nsPermissionManager.cpp#865
Dear Dan Witte,

Thank you for the post. I am so glad that someone finally comes up and
paid enough attention to suggest a possible solution.

Is there anyone who is knowledgeable enough to fix msgFilterRules.dat file
writing by using the nsISafeOutpuStream ?

As a matter of fact, I am afraid that the compaction code (the one that is invoked when we "compact folder") had the same bug at least in the version of thunderbird last summer. I lost a large chunk (the latter part of a large
archive). This lossage prompted me to investigate the system calls and
led me to the discovery that ENOSPC is not handled correctly.

I can't believe this type of bugs that can cause the lossage of important files
survived in a mail reader program for so long...

Too bad, I don't have much time to tinker with the source code nowadays.

Any takers to fix this bug?

Bug 413680 changed the code in question to use nsISafeOutputStream.  The change was made on the trunk and will be present in all 3.0-series releases.
I am glad that finally this problem is being looked into seriously
by someone who is knowledgeable enough to tinker
with the trunk source.

However, if I have a chance (and figure out how to access the source code quickly),
I will look into the modified code and see if I can still spot a problem.
The way ENOSPC was ignored made me suspicious of simple ignorance of
some esoteric error conditions.

Thanks!

I am afraid that the current production version, 2.0.0.14, still
suffers from the same problem. I lost the message filter rules
last week due to low TMPDIR space again. (An eager simulation program
produced a large log which ate all user accessible space...)

Any chance that this change makes into the current 2.0.0.14+ series?

(In reply to comment #19)
> Any chance that this change makes into the current 2.0.0.14+ series?

Not at the current time.  Since it sounds like this is a serious problem for you, you may want to try using Thunderbird 3.0 alpha 1 or even a nightly.
QA Contact: filters
Product: Core → MailNews Core
I have had this problem now for a while.  I just found in my error log this:

Error: uncaught exception: [Exception... "Component returned failure code: 0x8052ffff [nsIMsgFilterList.saveToDefaultFile]"  nsresult: "0x8052ffff (<unknown>)"  location: "JS frame :: chrome://messenger/content/FilterListDialog.js :: onFilterClose :: line 396"  data: no]

Everytime I close out Thunderbird my new filters disappear.  I have not seen the problem with space.
(In reply to comment #21)
> Error: uncaught exception: [Exception... "Component returned failure code:
> 0x8052ffff [nsIMsgFilterList.saveToDefaultFile]" (snip)

Same error code as Bug 448374 Comment #1.
To Kathy Hickerson:  If tmprules.dat(or tmprules-N.dat) remains in mail directory, add comment to Bug 448374, please.
Reporter - Is this still manifesting for you with current builds and new 3.x Beta / pre-release builds?
I have not had it happen with the Beta release.   Kathy Hickerson
(In reply to comment #24)
> I have not had it happen with the Beta release.   Kathy Hickerson

But you probably haven't had a disk shortage?  In which case, the bug may be/is like to be still valid.
I am the original reporter of this bug. Tomorrow, when I have time,
I will try this on 3vb4 under linux and see if the msgfilter survives.
(That is creating a low space condition for $TMP, $TMPDIR directories
and add a rule/save, delete the rule/save the filters a few  times
while recording the system calls and their return code like I did in comment #9
(boy, 5 years ago?!).
I tested it.
I have not confirmed whether the bug disappears completely or not due to
the minor change the way temporary file seems to be created.

But the change may be an improvement.

Instead of using /tmp directory for creating a temporary copy, it seems that
TBv3b4 creates a copy under the one's 
$HOME/.thunderbird/uniquestring/Mail/mailserverdomainname directory, i.e., the
original copy of msgFilterRules.dat resides, but under a different name
msgFilterRules-01.dat (or something to that effect), and
then rename it when the message filter management dialog/console is closed.
(Merely OK'ing one filter creation didn't seem to flush the internal data
structure to the disk. We need to close the filter management dialog for
the whole filter set to flush the internal data structure to the disk, it
seems.)

So my testing of creating a very low situation for $TMP and $TMPDIR directory
is not effective any more. 

At least, the low-space condition of the default $TMP and $TMPDIR, namely
/tmp is not easy to control due to other programs running behind your back,
and at least, you control your home directory better in terms of space
limit, this may be an improvement.

However, the question still remains (unless somebody dives into the code
and verify that TBv3b4 no longer trash msgFilterRules.dat even under
the low space condition of your Mail/mailserverdomainname directory.)


I need to create a low directory situation for whole
Mail/mailserverdomainnamedirectory before testing the bug disappeared
completely, 
and this requires a little more set up.
Maybe next week.

I will welcome someone else to chime in to check it.

BTW, the procedure to create a smallish virtual disk image and
then mount it under linux is as follows.

  Create a mount point (or you can use an existing directory.)

  mkdir /mnt/disk-image


  Create a one mega byte image

  dd if=/dev/zero of=./disk-image bs=1M count=1

  Create a file system and mount

  mkfs.ext2 disk-image   (and answer YES when a question appears.
                          You can use mkfs.ext3, of course)

  mount -o loop -t ext2 ./disk-image /mnt/disk-image
                         (You should match the fs type, i.e., ext2 or ext3).

  Create a temporary directory

  mkdir /mnt/disk-image/tmp
  chmod a+trwx /mnt/disk-image/tmp

  Setting up the mounted directory as the temporary directory

  As the mail user, do
  TMP=/mnt/disk-image/tmp
  TMPDIR=/mnt/disk-image/tmp
  export TMP      (bash)
  export TMPDIR

  run tbv3b4

  tbv3b4 should run using /mnt/disk-image/tmp as the temporary directory.

  Now, to run it under low disk condition for temporary file,
  copy a reasonably large size to $TMP so that the remaining
  space is very low.

  If you make the space zero, however, you can't start tbv3b4 any more.
  It seems a socket or something can't be created there anymore, and the
  program will not start up. (see below.)

Hope this helps.


[chiaki@dell-w2k-note ~]$ df $TMPDIR
Filesystem           1K-blocks      Used Available Use% Mounted on
/u2/ishikawa/disk-image
                          1003       988         0 100% /mnt/disk-image
[chiaki@dell-w2k-note ~]$ !/home
/home/ishikawa/THUNDERBIRD-DIR/thunderbird-v3-beta/thunderbird/thunderbird

** ERROR **: Resource problem creating '/mnt/disk-image/tmp/orbit-chiaki'
aborting...
[chiaki@dell-w2k-note ~]$ 
** ERROR **: Resource problem creating '/mnt/disk-image/tmp/orbit-chiaki'
aborting...

[chiaki@dell-w2k-note ~]$ ls -l $TMP
I switched my mail on another computer which is low on disk space, but even before it got there it would not save a new filter.  I am using version version 2.0.0.23 (20090812)  So, whatever is out there for the latest Thunderbird release, it is still happening.  I will try going to the Beta again.

Kathy
Kathy,

2.0.0.23 has many bugs IMHO.
(Creating a 4GB+ folder silently and
user not being able to delete/modify the folder after the  fact is one such a bug.)

Try v3b4 bug with caution. You may not want to enable deleting 
junk e-mails immediately, for example.
(Some suggest try nightly, but that is too much for a production use...)

Otherwise,  you may want to stick to 2.0.0.23 with the  known bugs.
(At least, if the bug is  known and if you know the work around, then
we can somehow manage. The above 4GB+ folder bit my colleagues so badly a few times just  before a large exhibition happens annually [so many last-minute
flier PDFs got  exchanged in just a  week or two and increased the folder
above 4GB limit easiliy.] and that is why I am biting  the bullet,
so to speak, to test this v3b4. 

YMMV.

PS: I am intending to backport the fix for 4GB+ bug [in v3b4] to 2.0.0.23
sometime in the future and if I can re-produce  this filter disappearing bug
again in 2.0.0.23, I will alert this, but given that the development community
is so in love with the latest and greatest  v3b4, I am not  sure
if I can persuade to incorporate the bug fix for  4gb+ and
have someone look into this filter disappearing issue.
this should be fixed based on comment 17, and bug 413680 comment 14

(FWIW bug 427865 is unrelated)
Status: UNCONFIRMED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: