273778 - If target folder for filter is deleted/moved received mail is not visible in INBOX

Status: VERIFIED FIXED

(MailNews Core :: Filters, defect)

Description

[Thomas]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.7.5) Gecko/20041108 Firefox/1.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE; rv:1.7.5) Gecko/20041108 Firefox/1.0

I came across this bug when reorganizing my folders. During this process, I
deleted some of them. As a result, Thunderbird told me the filters referencing
these folders would be inactivated -- which is wrong. Thus, mails that meet the
filter criteria and can't be moved into the target folder (since it was deleted)
are fetched from the server but are not displayed anywhere.

Reproducible: Always
Steps to Reproduce:
1. Create a new folder and a filter rule that moves messages which meet some
kind of criteria (like a particular subject) into this folder.
2. Now delete this newly created folder. You receive the info that the
corresponding filter has been disabled. Click OK. Check this filter, you'll see
that it is still active.
3.Clear the trash bin where the deleted folder still resides.
4.Write a test mail that meets the criteria of the filter in question.
5.When you receive the test mail, a warning windows pops up telling you the
missing folder couldn't be found and referencing filters are disabled. Confirm
this window with the OK button.
6.Thunderbird says: "You have one new mail". However it is invisible. You'll not
find it unless you locate your inbox file and open it with an editor.
Alternatively, delete the file inbox.msf, restart Thunderbird, and the mails
(re-)appear.
Actual Results:  
Filter was still active after deleting the folder.

Expected Results:  
The filter rule should have been deactivated immediately after the folder was
deleted.

Comment 1

[sascha.mrowka]

I can confirm this bug! It is always reproducible.

Comment 2

[WADA:World Anti-bad-Duping Agency]

As far as I know, mails are saved in account's Inbox(hidden) first, then message
filter is applied to it, then Junk filter is applied, then moved to Global
Inbox, if Global Inbox is used.

First, see message filer log of the account's(not Local Folder).
What log says on your "lost" mails? 

Second, see Inbox of the account(hidden in folder pain) by next procedure.
 (1) Create a folder "TEST" in "Local Folders".
 (2) See Account settings of the account and see "Local Directry" setting.
 (3) See Account settings of "Local Folders" and see "Local Directry" setting.
 (4) Shutdown Thunderbird.
 (5) Copy "Inbox" file (not Inbox.msf file) under directry (2)
     to "TEST" file(not TEST.msf file nor TEST.sbd directry) under directry (3),
     then delete "TEST.msf" file.
 (6) Restart Thunderbird and check contents of "TEST" folder.

Comment 3

[Jan Puchelt]

This is serious, I think!
Many Users won't know the described workaround, an regard their mails as "lost".

Comment 4

[WADA:World Anti-bad-Duping Agency]

(In reply to comment #3)
> Many Users won't know the described workaround, an regard their mails as "lost".
Steps in my comment #2 is not workaround.
That is searching of your kidnapped mail according to your application to the
police for a search.

> This is serious
Your mail was kidnapped but is not murdered yet.
Isn't it a good news? ;-)

Problem is :
 - Mail in the account's Inbox does not moved to Global Inbox
   when "move to folder" action in message filter fails.
Same problem is repored to Bug 274330, although the reason why "move mail
failure" is different from this bug.

Comment 5

[WADA:World Anti-bad-Duping Agency]

Change Product/Component.

Comment 6

[WADA:World Anti-bad-Duping Agency]

Same situation has been reported to Bug 275467, although the reason why "move mail
failure" is different from this bug.
Bug 275467's trigger is rename of mail folder of "Global Inbox" account.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

*** Bug 275467 has been marked as a duplicate of this bug. ***

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

*** Bug 288633 has been marked as a duplicate of this bug. ***

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

*** Bug 279500 has been marked as a duplicate of this bug. ***

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I can confirm with Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6)
Gecko/20050331 Thunderbird/1.0.2

Active message filters with a non existing target folder (renamed, deleted)
leaves incoming mail invisible inside the INBOX. Don't compact the INBOX in that
state because any hidden hidden mail will permanently deleted! A recreation of
the associated msf-file will show the mail within the INBOX.

When the mail is hidden the msf doesn't have an entry for the mail. I don't know
the list of processes which run on an incoming message. After showing the error
dialog for the not working filter it seems that the rest of the processes aren't
run and the msf file isn't updated with the new entry.

If such a situation appears the rest of the processes should be executed and the
msf file for the INBOX has to be updated or the mail has to be copied to the
global inbox (when it is used).

A testcase with the broken msf file and MBOX follows.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: MBOX and msf file with hidden message

Comment 12

[David :Bienvenu]

I'm assuming this is local/pop3 mail...

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #12)
> I'm assuming this is local/pop3 mail...

Yes, this testcase is from my local POP3 inbox. Sorry for not mention it
explicitelly.

Comment 14

[David :Bienvenu]

Attachment: proposed fix

a long time ago, we used to save the filter file in the destructor for the
server, but that code was removed, since it was unreliable. But, we never added
code to flush the filter file when we disabled invalid filters. So this patch
adds that.

And the real fix is that if the move failed, don't set the m_msgMovedByFilter
boolean, and don't delete the header from the db.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

David, your patch is working fine in TB 20050405. Two things I noticed: We lost
the warning dialog if a filter can't be applied!

Further the file msgFilterRules.dat contains the wrong (not existing) folder.
But when editing the Filter within the UI the destination is set to the default
account? What's happening here? 

IMO we have to warn the user that the filter (with name) is corrupted and will
be disabled.

Comment 16

[David :Bienvenu]

>We lost the warning dialog if a filter can't be applied!

that was lost before my patch. The reason is that the msgwindow is null, so we
don't put up an alert. msgWindow is only non-null when you do a get new mail, I
think...

>Further the file msgFilterRules.dat contains the wrong (not existing) folder.
>But when editing the Filter within the UI the destination is set to the default
>account? What's happening here? 

We only disable the filter; we never removed the folder name from the filter
file. The UI doesn't show the folder name because it can only show folders that
exist, and the folder doesn't exist. That hasn't changed.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #16)

> that was lost before my patch. The reason is that the msgwindow is null, so we
> don't put up an alert. msgWindow is only non-null when you do a get new mail, 
> think...

Users will see that one of the filters doesn't work. But when you have a bunch
of that feel happy to search for the broken one. :( IMHO we really should open a
warning dialog with the name of the filter. Any chance to get it back?

> We only disable the filter; we never removed the folder name from the filter
> file. The UI doesn't show the folder name because it can only show folders hat
> exist, and the folder doesn't exist. That hasn't changed.

Ok, but we shouldn't show the Inbox of the first account. That's confusing
because the not filtered message doesn't land there. Should I open a new bug for
that issue?

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 179622 [details] [diff] [review]
proposed fix

David, shouldn't we provide the patch for the next aviary thunderbird release? 

It's a dataloss bug and if only one percent of the users have a corrupted
filter, so over 10.000 people live with loosing messages! Already one person is
enough. If it's too late for 1.0.3 do it for 1.0.4 please!

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Reopening until it will be checked in or the wish of a lot of people is not
fulfilled.

Comment 20

[David :Bienvenu]

it's fixed on the trunk - so re-closing...we don't re-open bugs to check fixes
into branches.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #20)
> it's fixed on the trunk - so re-closing...we don't re-open bugs to check fixes
> into branches.

Ok, but is there a chance to get this into the branch?

Comment 22

[David :Bienvenu]

it's not up to me, but I'd say that yes, there's a chance, assuming we do
another branch build before we ship 1.1

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Verified with Thunderbird version 1.0+ (20050407)

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Because it was too late for aviary1.0.3 asking for blocking-aviary1.0.4.

Comment 25

[Daniel Veditz [:dveditz]]

Comment on attachment 179622 [details] [diff] [review]
proposed fix

a=dveditz

Comment 26

[David :Bienvenu]

1.7.x doesn't have the dataloss part of this bug, but it doesn't flush the
filters to disk after they're disabled.  I'll check in that part to 1.7.x, and
the full fix to the aviary branch.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Fixed checked in. Thanks David.