413680 - Message filter rules vanish with nightly builds since 2008-01-18

Status: RESOLVED FIXED

(MailNews Core :: Filters, defect)

Description

[callow386]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b3pre) Gecko/2008012204 Minefield/3.0b3pre
Build Identifier: Thunderbird 3pre1 (20080122)

I am running your bleeding edge Thunderbird 3pre1 nightly builds.  I had set up some filter rules and enjoyed them working to auto file/mark read emails that i didn't need to see.  This morning I came into work and started Thunderbird.  It installed the nightly build update 20080122.  After it started all my email went into the inbox instead of the folders that they were suppose to go into.  When I checked the message filters all of my rules were gone.  I can remake them but wanted y'all to know they disappeared on me.  I then downloaded and installed the nightly build 20080123 however, the rules were still gone.

VMWare's Virtual Server 1.0.4 client
2.8GHz/512MB Ram
Windows XP Sp3 (beta)
Thunderbird 3 pre1 20080123 (nightly builds)
  using IMAP connection to mail server


Reproducible: Didn't try

Steps to Reproduce:
1. Running Thunderbird 3pre1 nightly 20080121 and enjoying my message filter rules.
2. Autoupdate feature installed nightly 20080122 and message filter rules vanished.
3. 
Actual Results:  
After the installation my message filter rules vanished.  I remade the filter rules.

Expected Results:  
I was hoping to keep my message filter rules so that my email would be auto filed and marked read so i didn't have to worry about them.

The only work around that I could think about is to backup one's Thunderbird profile before doing the upgrade.

Comment 1

[Eyal Rozenberg]

Seeing the same thing with SM trunk build 2008-01-22 02, after an autoupdate (although it could just be the build and have nothing to do with the update; that's even likely, actually)

If I try to create new filter rules, they soon disappear as well.

Comment 2

[callow386]

I recreated my Message Filters.  After reading Eyal's comment I attempted to
look for the file msgFilterRules.dat.  I did not find msgFilterRules.dat file. 
I closed Thunderbird and reopened it and all my filters have vanished again. 
:(

Comment 3

[Wayne Mery (:wsmwk)]

perhaps a dupe of bug 389058

Comment 4

[callow386]

Bug number 389058 is the same issue on a different version of Thunderbird (2.0.0.4).  I did not see that bug when I did my search for the issue so I created a new ticket.  I have added a comment to that ticket referencing this bug ticket.

Comment 5

[Doug Wright]

dataloss->critical

Comment 6

[Eyal Rozenberg]

It just happened to me with build 2008-01-22 02, after several hours of not 'forgetting' the filter rules with this build. I'm not sure what the trigger is, though, but it's not merely getting new messages or the application of filters.

Comment 7

[Eyal Rozenberg]

Umm, I meant 2008-01-21 02.

Comment 8

[Phil Ringnalda (:philor)]

Eyal (or anyone else, for that matter): if you're losing them quickly enough to tell if a build is okay easily, would you mind jumping back to 01-17/18? I'm mildly suspicious about bug 375292 (even though it doesn't look wrong). It landed just after the SM trunk builds for 01-17, but before the Tb builds, so if that was it, I'd expect failure in Seamonkey 01-18 and Thunderbird 01-17 (the first things to try), and success in Seamonkey 01-17 and Thunderbird 01-16 (the next thing to try, once you lose rules in the later builds).

I'd try, but I'm only losing them around once every other day for some reason.

Comment 9

[Eyal Rozenberg]

Filters disappear with SM trunk 2008-01-18 05.

Comment 10

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log when Message Filters window is closed

Test scenario.
(1) Create message filer(for "Local Folders") with 2008/01/02 build
(2) Start Tb trunk 2008/01/24 build(MS Win-XP) => msgFilterRules.dat was read
(3) Open Tools/Message Filters => msgFilterRules.dat was read
(4) Edit a filter rule, and close edit window => msgFilterRules.dat existed
(5) Close Message Filters window => msgFilterRules.dat vanished
    Attached process monitor log is for Tb's access to msgFilterRules.dat
    only while this step (5). 

Following looks to be culprit.
Change by Bug 375292(delete of tmpFile) was executed even when original msgFilterRules.dat? 
> "18232","10:43:47.4943260","thunderbird.exe","2448","IRP_MJ_SET_INFORMATION",
>         "C:\Local Folders\msgFilterRules.dat","SUCCESS",
>         "Type: SetDispositionInformationFile, Delete: True"

Comment 11

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log(2), Create&Save Filter with Tb trunk 2008-01-17-03,

> Tb trunk 2008-01-16-03(MS Win XP) : No problem
> Tb trunk 2008-01-17-03(MS Win XP) : deleted msgFilterRules.dat

Attached Process Monitor log is for step (1) to (3), with next filtering.
  - Process name contains "thunderbird.exe" & Path contains "rules"
(0) No msgFilterRules.dat for "Local Folders"
(1) Start Tb trunk 2008-01-17-03
(2) Tools/Messages Filters("Local Folders"), create a rule, end Message Filters
(3) Terminate Thunderbird

(Off-Topic?)
Query for "...\Local Folders\rules.dat" is seen in log. What is "rules.dat"?

Comment 12

[WADA:World Anti-bad-Duping Agency]

Attachment: Process Monitor log(3), Create&Save Filter with Tb trunk 2008-01-16-03 (no problem)

Log for final delete of "Local Folders\msgFilterRules.dat" is not seen when Tb trunk 2008-01-16-03.

After write to \tmprules.dat in \Temp directory, it is moved to "Local Folders\msgFilterRules.dat", because same drive in my case.
And, this "file move" looks to be a kind of "file re-name" of file accessed by variable of "tmpFile".
I guess that delete request for "tmpFile" by patch for Bug 375292 will delete the updated(moved/renamed) "Local Folders\msgFilterRules.dat" in this case.

Comment 13

[Eyal Rozenberg]

Filters do not seem to disappear after half-a-day's use of SM trunk 2008-01-17 02.

Comment 14

[Phil Ringnalda (:philor)]

Attachment: Fix v.1

I think this is what we want to do: stick around until we delete the temp file once we hit the point where we create it, and don't delete it when we've successfully moved it, since that means deleting the real file, which makes Hulk smash!, but every time I touch C++ I do something foolish, so, really review, plzkthx.

Comment 15

[Rich Gray (:rbgray)]

I'm seeing this under OSX 10.5.1 in both TB & SM nightlies, so OS->All.   Asrail suggests this should block 1.9.  Seems like a good idea.  Version ->trunk.

I was starting from clean profiles, so it looked as if the filter definitions were simply not saved.   Curiously, TB was able to execute the filter while SM did not.  But in either case, the definitions did not show up in the News directories (at least long enough to see in finder :/ ) and the filters were gone if one quit and restarted.

Comment 16

[Rich Gray (:rbgray)]

Just spun a build of Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b3pre) Gecko/2008012621 SeaMonkey/2.0a1pre with fix v.1 and that seemed to do the trick.   Defined a news filter which worked and didn't go *poof*. I see the msgFilterRules.dat on disk.

Comment 17

[Magnus Melin [:mkmelin]]

Though I think the patch is fine, I wonder if it might be worth checking if we can write even before we create the temp file. Something like 

filterFile->IsWritable(&isWritable);
if (!isWritable)
  return NS_ERROR_FILE_ACCESS_DENIED;

Comment 18

[[e-mail]]

Comment on attachment 299484 [details] [diff] [review]
Fix v.1

So, anyone want to file a bug on documenting that moveTo modifies the nsIFile that you call it on to point to the new location?

>+  if (NS_FAILED(ret))
>+  {
>+    PRBool tmpExists;
>+    nsresult rv = tmpFile->Exists(&tmpExists);
>+    if (NS_SUCCEEDED(rv) && tmpExists)
>+      tmpFile->Remove(PR_FALSE);
>+  }
I was going to point out that since you don't care about the return value of Remove you don't actually care if the file exists before you try to remove it. However I'd really prefer this simplified to use a safe file output stream ;-)

Comment 19

[Nelson Bolyard (seldom reads bugmail)]

This is now hitting me daily with nightly trunk builds of SeaMonkey.
I have lost two separate msgFilterRules.dat files in less than 24 hours.
(Fortunately, I have backups.)
AFAIK, I have not edited any message filters at any time for several days 
now, so editing doesn't seem to be the cause of this problem.

Comment 20

[Phil Ringnalda (:philor)]

Attachment: Fix v.2

Mmm, safe file output stream's much saner. And I can't have gone (far) wrong, since every impl in the tree appears to be a copy-paste from one original source (though I filed off a few of the serial numbers, like the "failed to save foo file! possible dataloss" warning text).

Comment 21

[[e-mail]]

Comment on attachment 299937 [details] [diff] [review]
Fix v.2

Sorry if these are errors that you copied ;-)

>+  rv = filterList->SaveToFile(strm);
I don't see you checking rv here.

>+  if (NS_FAILED(rv))
>   {
>+    NS_WARNING("Failed to save filter list");
>+    return rv;
>   }
>-  NS_ASSERTION(NS_SUCCEEDED(ret), "error opening/saving filter list");
Why not keep the assertion? Or use NS_WARN_IF_FALSE?

>+  return NS_OK;
Could return rv; here too.

Comment 22

[Nelson Bolyard (seldom reads bugmail)]

Given that I'm not editing filter rules, 
why is the file being rewritten at all?
Also, why did this start just ~10 days ago?

Comment 23

[Phil Ringnalda (:philor)]

Attachment: Fix v.3

The really sad part is that I did realize partway through that I wanted to copy "rv=, assert about safeStream, if(rv && safeStream)" from bookmarks instead of just copying from wallet which doesn't do anything but ->Write to write, but then I apparently just forgot.

Comment 24

[[e-mail]]

Comment on attachment 300314 [details] [diff] [review]
Fix v.3

>+  rv = filterList->SaveToFile(strm);
You're still stomping on this rv...

Comment 25

[Nelson Bolyard (seldom reads bugmail)]

I'd still like to know why msgFilterRules are being modified by the program
when the user does not modify any rules.  Seems like a bad idea.  Has now had 
obvious bad consequences.  How about if we stop doing it? 
Is there any defensible justification for continuing to do it?

Does this problem also explain the loss of .dat files in the News group server
folders containing rules for Newsgroup filtering?

Comment 26

[Magnus Melin [:mkmelin]]

Possibly since SaveFilterList /can/ be called at least also in OpenFilterList and SaveToDefaultFile - both are called on a number of occasions.

Comment 27

[Phil Ringnalda (:philor)]

Attachment: Fix v.4

Gah. Sleep may be for the weak, but the weak certainly do seem to be able to concentrate better.

Comment 28

[Phil Ringnalda (:philor)]

mailnews/base/search/src/nsMsgFilterService.cpp 1.64

Comment 29

[Phil Ringnalda (:philor)]

(In reply to comment #25)
> I'd still like to know why msgFilterRules are being modified

Me too. I've been running a debug build for five days now with a breakpoint in SaveFilterList, and haven't hit it once. "Some unknown code for some unknown reason saves the filter list without changes only in opt builds" isn't exactly going to be an obvious and easy thing to fix.

Comment 30

[Magnus Melin [:mkmelin]]

Many of the calling points seem to involve "upgrading" the filter file, so perhaps you have to mix-use the profile with branch builds. Just a guess, but I haven't hit this bug at least for my main profile, which have probably been trunk only for some months.

Comment 31

[Phil Ringnalda (:philor)]

kFileVersion last changed in April 2000, so it would take a pretty branchy branch build to trigger that (properly - improperly is another matter, but either way I wasn't using branch builds during the period when I was hitting it at least every other day).

Comment 32

[Nelson Bolyard (seldom reads bugmail)]

The profile I'm using that is afflicted has only ever been used by trunk SM
builds.

Comment 33

[Worcester12345]

(In reply to comment #22)
> Given that I'm not editing filter rules, 
> why is the file being rewritten at all?
> Also, why did this start just ~10 days ago?
> 

(In reply to comment #29)
> (In reply to comment #25)
> > I'd still like to know why msgFilterRules are being modified
> 
> Me too. I've been running a debug build for five days now with a breakpoint in
> SaveFilterList, and haven't hit it once. "Some unknown code for some unknown
> reason saves the filter list without changes only in opt builds" isn't exactly
> going to be an obvious and easy thing to fix.
> 

(In reply to comment #30)
> Many of the calling points seem to involve "upgrading" the filter file, so
> perhaps you have to mix-use the profile with branch builds. Just a guess, but I
> haven't hit this bug at least for my main profile, which have probably been
> trunk only for some months.
> 

Any relation to this one?
Bug 391463 – Thunderbird crashes when working with filters [@ iid] [@ nsHashtable::Reset]

Comment 34

[Phil Ringnalda (:philor)]

(In reply to comment #33)
> Any relation to this one?

Absolutely none whatsoever: you're crashing while changing filters, we want to know why we're saving the filter file while doing nothing to change them.