Closed Bug 260226 Opened 20 years ago Closed 20 years ago

Email marked as spam disappears - not in trash folder or junk folder

Categories

(Thunderbird :: General, defect)

x86
Windows XP
defect
Not set
critical

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 260981

People

(Reporter: mskorman, Assigned: Bienvenu)

References

Details

(Keywords: dataloss, regression)

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20040913 Firefox/0.10
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20040913 Firefox/0.10

Email marked as spam disappears. It does not appear in trash folder or junk folder. 

Reproducible: Sometimes
Steps to Reproduce:
1.Can't figure it out. There is an optition in the buttons extension that allows
junk mail to be completely deleted but I never checked that. To be sure, I
uninstalled buttons.
2.
3.

Actual Results:  
The frequency has increased.

Expected Results:  
Placed email marked as spam in the trash folder and shown the messages as unread.
Which version of TB?
If you're using 0.8, are you also using the Global Inbox feature?

See bug 257421.
Confirming on version 7.0 (20040909) - CVS build

I've seen this a few times over the last few weeks. I wasn't sure at first as I
have several mailboxes and it's not easy to waych them all as it's d/ling.

I've now turned off auto fetching and d/l one mailbox at a time. I've just watch
it get 72 mails for an account, the junk filter ran, and I ended up with 2 in
the Inbox and two in Junk - 68 missing!!

This is serious, even though my junk filter is very accurate (months since I had
a false positive).

I am NOT using Global Inbox but I've noticed something odd. I don't have a
folder named Junk - I have the junk filter send junk straight to Trash - but a
Junk.msf file exists in every mail directory (but no Junk file) with a current
timestamp.
I'm going to confirm this because I think bug 260216 and bug 260370 might be the 
same problem.  I hope this gets some attention soon from people who can figure 
it out better than I can.

(In reply to comment #2)
> I've just watch it get 72 mails for an account, the junk filter ran, and I
> ended up with 2 in the Inbox and two in Junk - 68 missing!!

You mean "two in Trash," right?

You might try turning off the setting to automatically move junk mail.  You'll 
get a lot more pointless notifications, but it would be good to know whether 
this deletion is taking place as a function of the "move junk mail" or the 
"detect junk mail" portion of the problem.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Keywords: dataloss, regression
> You mean "two in Trash," right?

Duh! yes, I do

Does the Junk Filter move/delete messages in the same way as doing it manually,
i.e. it doesn't physically delete them from Inbox, but flags them deleted, to be
removed when the Inbox is Compacted?

I've just opened the Inbox in question in a text editor and the 68 "missing"
mails are there, all with the status flags set to

X-Mozilla-Status: 0008
X-Mozilla-Status2: 00010000
OK, I've just seen this happen on 3 mailboxes:

Go to Inbox
Click the Get New Messages button
Watched several mails arrive in the Inbox, automatically marked as Junk
Open the "junk" folder (Trash in my case) - messages not there
Switch back to Inbox
In Exporer, go to the mail folder and delete Trash.msf
Switch to Trash - and there are the messages (checked timestamp to confirm)

So, it seems that the Junk filter is DTRT, but the .msf file of the "junk"
folder isn't being updated.

Also, I have seen a variation on this where the mails disappear from Inbox but
"Trash (xx)" in the folder tree doesn't chamge to bold and the unread count
doesn't change, but when I switch to Trash it does, and the deleted messages are
there.
X-Mozilla-Status: 0008

that means the message is deleted. I believe the junk mail plugin moves messages
with the same code as filters and ui move/copies. I wonder if what's happening
is that the target folder db gets out of date, so the summary file is invalid,
and we copy the message to the dest folder w/o updating the db, but then some
code comes along and erroneously marks the db as valid.
Assignee: mscott → bienvenu
I can't be 100% sure of this yet, but it appears that it may only happne when
the "junk" folder is empty, but  maybe that's because I'm using Trash as my
"junk" folder and it is a special folder - the Empty Trash function?
If it happens when the destination folder is empty, I wonder if it is more
widespread than junk mail handling.  For example, I created a bunch of empty
folders to hold mail from each of my friends...
Re comment 8

It seems to only affect Junk. I have a lot of filters set up and they seem to 
be working OK.
In bug 260370 comment 2 (a probable dupe), reporter says that the setting for 
"Move detected junk to..." is turned off.  She also is getting notifications 
when all the incoming mail has disappeared.
As far as I could tell this only happened with mail marked as junk on my
computer. It didn't happen every time but often enough. I never had this problem
 at all until I upgraded to TB .8. I have been using Mozilla 1.x mail for years
and switched to Thunderbird/Firefox 4 months ago. Two nights ago, I uninstalled
TB .8 and FF pr1.0 then saved mail, abook, and bookmarks only, then deleted all
Mozilla/Thunderbird profiles, then cleaned up Mozilla/Firefox/Thunderbird in the
registry, then reinstalled Firefox .9.3 and Thunderbird .7.3 . So far everything
is fine.
My concern is that I will lose some mail that is real mail, but that Thunderbird
might think is junk, such as newsletters, plus whatever bug this is may affect
other parts of mail handling, if not now then in the future.

I haven't run Mozilla mail or Thunderbird previous to installing Thunderbird a
few days ago.  I did have a previous version of Firefox installed.

I assume, perhaps incorrectly, that there is a way to tell Thunderbird don't
check for spam, let everything thru to the inbox, but I can't deduce this from
the Junk Mail Controls settings.  I was thinking turning off the adaptive filter
might do this, but I caught thunderbird losing mail even with that done (a spam
email was on the server but nowhere on my system.)
(In reply to comment #12)

In reference to to the above comment, I have turned off Junk Mail Controls and
all Junk mail now hits my Inbox, and I delete each one manually.

Unfortunately, I have just had the experience (in the last few minutes) of
deleting an unread spam message from my Inbox (not marked as Junk, btw), then
going to the Trash folder to check the contents of that message (was it really
spam?), and finding it gone. 

WTF?

I am using a nightly build from 2/3 days ago.
Is the message really in the Trash folder, but just not showing up? Is the Trash
folder threaded or sorted flat (e.g., by date)? If you delete the .msf file, and
then look in the Trash, does the message appear? Or is it in the trash but
marked as deleted with an x-mozilla-status line of 0008 or 9? I can't reproduce
this problem myself?
(In reply to comment #14)
> Is the message really in the Trash folder, but just not showing up? Is the Trash
> folder threaded or sorted flat (e.g., by date)? If you delete the .msf file, and
> then look in the Trash, does the message appear? Or is it in the trash but
> marked as deleted with an x-mozilla-status line of 0008 or 9? I can't reproduce
> this problem myself?

I did a WinHex (Hex Editor) check on files Trash (10Kb) and Trash.msf (2Kb). the
.msf file does not reference the missing email, but I found the missing email in
the Trash file.

Inbox and Trash Folders are both sorted by date, but one is ascending, one
descending.
what does the mozilla status line of the missing e-mail in the trash look like?
If you shutdown tbird, delete the .msf file, launch tbird, and click on the
trash, does the message reappear?
(In reply to comment #16)
> what does the mozilla status line of the missing e-mail in the trash look like?
> If you shutdown tbird, delete the .msf file, launch tbird, and click on the
> trash, does the message reappear?

I shut down Mozilla completely (not using TB, using Moz mailnews) and deleted
the msf file, restarted Moz, and voila: the missing email (status unread) is
back in the Trash!

[It's important to tell people to get right out of Moz when doing this,
including the quick launch memory resident module, or else the msf file gets
instantly recreated, and you can even get into a loop in M$ Explorer Search
function doing this, as happened to me last time I tried this.]

Do you still need the status line from the Trash file? I got his from the Trash
file: X-Mozilla-Status: 0200 X-Mozilla-Status2: 00000000
David, I have stopped doing Empty Trash after checking the messages are actually
spam, just marking the Trash folder as Read and I have not had any disappearing
mail since (although I have had the count not updated, as per the last paragraph
of comment 5). It seems as though the problem only manifests itself if the
"junk" folder is empty - at least for me using Trash as the "junk" folder.

If you can point me to the general area of the code where this is handled I can
spin a debug build and see if I can work out what is happening.
Parish, my guess is that somehow your Trash .msf file is getting its timestamp
out of sync with the Trash folder on disk. If we hit this line of code, that
means we had a db with a bad timestamp:

http://lxr.mozilla.org/mozilla/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#1116

I've tried forcing this (by Touch'ing the Trash folder) but haven't seen any
problems. I've also tried emptying trash w/o a problem.

Is you local mail directory in the standard profile dir?
(In reply to comment #19)
I was wondering about timestamps, but couldn't see how it could be assuming that
Trash is updated (on disk) and then the summary file is updated.

I'll build a debug version and see what is happening around that part of the
code since I can reproduce the problem.

> Is you local mail directory in the standard profile dir?

Yes.
Here's where we add the hdr to the db:

http://lxr.mozilla.org/mozilla/source/mailnews/db/msgdb/src/nsMsgDatabase.cpp#2935

all evidence is that we're not getting here. When running a debug build, you
should look at the console window and see if anything interesting gets dumped there.

Do you have it set so that when you delete or move a message locally from the
inbox, it's removed from the pop3 server? Does this bug happen when you delete
messages from folders other than the inbox?
I did Empty Trash, then Get New Messages; it got one message which was marked as
Junk (automatically) and moved to Trash - it didn't appear when I looked in
Trash. This is the console output generated for getting that message:

Begin mail message delivery.
Abort mail message delivery.
Begin mail message delivery.
GetDiskSpaceAvailable returned: -1371414528 bytes
Incorporate message begin:
uidl string: <1095800990-21088@excite.com>
Incorporate message complete.
End mail message delivery.
WARNING: NS_ENSURE_TRUE(NS_SUCCEEDED(rv)) failed, file
c:/mozilla_src/mozilla/mailnews/db/msgdb/src/nsMsgDatabase.cpp, line 160
Begin mail message delivery.
Abort mail message delivery.
Begin mail message delivery.
Abort mail message delivery.

Without doing Empty Trash I waited until I was notified of new nessages, did Get
New Messages which got two messages, both of which were automatically marked as
junk and moved to Trash. These two DO appear when I look in Trash but the
previous message still does not appear - opening Trash in an editor confirms
that all 3 messages are there. Here is the console output for getting these two
message:

Begin mail message delivery.
GetDiskSpaceAvailable returned: -1371426816 bytes
Incorporate message begin:
uidl string: <1095801996-28240@excite.com>
Incorporate message complete.
Incorporate message begin:
uidl string: <wkv4_mhn4-1823697035-2697354367-78-47049.7262474792@jimenez6>
Incorporate message complete.
End mail message delivery.
Begin mail message delivery.
Abort mail message delivery.
Begin mail message delivery.
Abort mail message delivery.

Notice that NS_ENSURE_TRUE(NS_SUCCEEDED(rv)) doesn't fail at line 160 this time.
I have reproduced the above scenario several times and each time
NS_ENSURE_TRUE(NS_SUCCEEDED(rv)) only fails after I've done Empty Trash.

Does this tell you anything?

Ref comment 21, it does get here, but only when Trash is not empty AFAICT.

BTW, LXR is out of date; the two line numbers you mention are 1113 and 2932 in
the file pulled by cvs.

yes, it tells me that after the empty trash, there was an error opening the
message db, presumably for the trash:

WARNING: NS_ENSURE_TRUE(NS_SUCCEEDED(rv)) failed, file
c:/mozilla_src/mozilla/mailnews/db/msgdb/src/nsMsgDatabase.cpp, line 160

I see the same warning, but the message ends up visible in the trash...I'll try
to think why it might be different for you.
Just checked something about timestamps.

After it has moved messages to Trash, Trash and Trash.msf have the same modified
timestamp (to the nearest second) but after doing Empty Trash then Trash.msf has
a modification time 3 seconds later than Trash. If the code is checking for
Trash being later than Trash.msf _before_ messages have been moved to Trash then
it will get it wrong and assume that the .msf file is up to date. Is this likely?

it doesn't compare the file timestamps; it compares a time stamp stored *in* the
.msf file vs the file time stamp of the folder. But emptying the trash should
create a new, empty, valid msf file, with the correct timestamp stored in it.
I'll try to figure out why it doesn't...
Re #24

Peeps, don't you have the ability to lock the "update data file update msf file"
operation, so you can be sure the files are synchronized and not have to check
time stamps?  I have no idea what language you're writing in...
*** Bug 259149 has been marked as a duplicate of this bug. ***
This happens to me on the 0.8 milestone.

I'm using global inbox with three mail accounts.
When I receive new mail and decide to delete it with or without reading, the
first message disappears into the global thrash folder but the next messages
will show up in the folder.

Same thing happens with mail automatically marked as junk and moved into the
junk folder, the first one will dissapear and the rest will be visible.

If I then delete all messages from junk and thrash folders and compact both
folders and then go into my profile directory and delete the junk.msf and
thrash.msf files for global inbox the lost messages will reappear in the
appropriate folders.
re timestamps, the issue is other programs altering our mail store, while we're
running or not running...though they cause a lot more grief than they're
probably worth...

The warning I'm seeing happens when we empty the trash, but we create the
trash.msf immediately after, and then it's valid. After you empty the trash, is
the Trash.msf file 0 bytes, or 2k (1207 bytes, actually)?
I posted already on mz, I've also send my two trash files to you David (hope you
got them)...

I came across another interesting behaviour... After I moved my mails to trash
(with 'del' key), the disappeared message reappeared - the reappeared message
had the same subject as the just deleted messages (generated mail from
bugzilla). Could be related to the threading bug - Delete bug in thread view;
#231846?

My Trash.msf file has 1.220 bytes after I empty the trash.
I've reproduced this a couple times by changing the timestamp of the trash
folder with "touch" after emptying the trash, but even that's not 100% reliable.

select the trash folder
empty trash
touch Trash
select another folder and delete a message. The db doesn't get updated, but the
Trash folder does.

 I believe the problem is something like this, however:

1. Trash.msf gets out of date (not sure how that happens, but it seems that
empty trash can leave it out of date for some users/scenarios)
2. We move a message into the trash.
3. That causes us to get the db for the trash. Since it's out of date, we get a
new db and (perhaps) start to reparse the trash folder.
4. In the meantime, the move finishes, we get the empty db, and we mark the new
db as valid, before the reparse finishes. So it's valid w/o any headers.
Status: NEW → ASSIGNED
Re #31:

I'm not sure how many processes(threads, whatever) are involved here, but why is
step 3 starting a reparse of the trash file before the move of the message into
the trash file has completed?  (Or is the data file being parsed in what we
oldies used to call core and by move you mean write it to disk?)

Why is a reparse of the entire data file necessary on a move in order to update
the db(index file?) instead of just inserting the new data in the db?

Why is the db being marked valid if you're depending on timestamps to determine
validity?  Having two mechanisms to indicate validity would seem to be a source
of problems.  Is it because (not sure if this is possible) some process can
start reading the db while you're still modifying it?

Why in step 4 is that process marking the db as valid when it has no idea if the
db has completed updating?
Trying to figure this out some more.  Is this a buffer management problem, where
you need flags for a buffer for valid, read in progress, write in progress, etc.
and need to manage the id of the buffer so it is recognizable even while it is
being worked on, and a process that wants it has to queue up if there is another
process working on the buffer or a read in progress?
If we change the mail folder (e.g., add a message to it), we have to mark the
corresponding db as valid, or the time stamps won't match. Obviously, we should
never have a db that was previously invalid, make a change to the folder, and
then mark the invalid db as valid. But I suspect that's what's happening. It's
probably the case that the folder object is caching a pointer to the open db,
and the db becomes invalid, and we don't realize that cached db pointer points
to a no longer valid db. How the db is becoming invalid in the first place is a
mystery to me...

Also, I think my previous theory about the reparsing is probably wrong - I can't
catch it doing that.
This doesn't affect just spam emails.

check this bug for other details and other related bugs:
"message count and file size is wrong for folders on pop3 account"
https://bugzilla.mozilla.org/show_bug.cgi?id=259185
Flags: blocking-aviary1.0?
do any of you have virus checkers running that might be holding the trash folder
open or modifying it?
(In reply to comment #36)
> do any of you have virus checkers running that might be holding the trash folder
> open or modifying it?
FWIW I duplicated the problem on a test PC with a fresh image of XP, no virus
checker installed, and in fact no other software running. 
I also don't have a virus checker and I'm running win2k (SP4 + latest "fixes").
do you have empty trash on exit set? I'm seeing odd things if I empty trash and
then exit, or turn on empty trash on exit.
I know your question wasn't addessed to me, but I just checked and I don't have
empty trash on exit set.
Hum, I don't use the "empty trash on exit" option. However, I've emptied the
trash, then went to my inbox, deleted one mail and it didn't show up in trash -
but I can't reproduce this. 

Update, I've reproduced the bug:
1. compact the trash folder and empty it
2. restart tb
3. delete a message

(like mentioned before, to get back the disappearred messages, go to your
profile and delete the Trash.msf file)
please try tomorrow's trunk or aviary 1.0 branch build - I've checked in a fix
for bug 260981, which I suspect is the same bug, or at least has the same fix.
Just spun a build with this patch in and it fixes the problem :-) Thanks David.

BTW, for anyone else testing this, it may not work the first time if Trash has
an "invisible" message in already, but do Empty Trash and it wil then work
(obvious really I guess).
I can confirm that the problem with deleting is fixed in the latest nightly.
However, weird things can happen (maybe because of the old Trash.msf file?).
Will there be a bugfix release? 

Anyway, thanks for fixing this David!
I think Scott spins up weekly .8 builds, and of course, there are the nightly
tbird .8 builds, as well as the nightly trunk builds - and this fix will be in
1.8a4, for seamonkey users.

marking dup of 260981 as the cause and fix seem to be the same.

*** This bug has been marked as a duplicate of 260981 ***
Status: ASSIGNED → RESOLVED
Closed: 20 years ago
Resolution: --- → DUPLICATE
> *** This bug has been marked as a duplicate of 260981 ***

Bug 260216 is also a dup, AFAICT. As reporter of that bug, I'd say it is.
*** Bug 260216 has been marked as a duplicate of this bug. ***
*** Bug 260370 has been marked as a duplicate of this bug. ***
(In reply to comment #16)
> what does the mozilla status line of the missing e-mail in the trash look like?
> If you shutdown tbird, delete the .msf file, launch tbird, and click on the
> trash, does the message reappear?

Not for me (at least a few days ago).
(In reply to comment #37)
> (In reply to comment #36)
> > do any of you have virus checkers running that might be holding the trash folder
> > open or modifying it?
> FWIW I duplicated the problem on a test PC with a fresh image of XP, no virus
> checker installed, and in fact no other software running. 
> 

I have Symantec Antivirus 9.0.0.338, but don't know what it is holding open or not.
(In reply to comment #41)
> Hum, I don't use the "empty trash on exit" option. However, I've emptied the
> trash, then went to my inbox, deleted one mail and it didn't show up in trash -
> but I can't reproduce this. 
> 
> Update, I've reproduced the bug:
> 1. compact the trash folder and empty it
> 2. restart tb
> 3. delete a message
> 
> (like mentioned before, to get back the disappearred messages, go to your
> profile and delete the Trash.msf file)


Like mentioned before, that didn't work for me.
Flags: blocking-aviary1.0?
You need to log in before you can comment on or make changes to this bug.