485176 - Backslash quotes in filter prevents msgFilterRules.dat from being read

Status: NEW

(MailNews Core :: Filters, defect)

Description

[MediaDog]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7
Build Identifier: 2.0.0.21 (20090302)

If a filter is created that contains backslash-quotes (\"\"), the filter will work correctly, but after closing Thunderbird and trying to reopen it, you will get the message that msgFilterRules.dat could not be read, and that the backup is being used.  Manually removing the filter definition with the backslash-quotes allows msgFilterRules.dat to be read OK.

Example:

I get a lot of spam with an empty name in the "To:" field, such as:

To: "" [e-mail]

so I created a filter to match the To: field to:

\"\" foo

which works perfectly (quotes without preceding backslash escapes will NOT filter properly).

The problem comes after quitting Thunderbird and restarting it, when the msgFilterRules.dat file will now be seen as corrupt (may have to click "Tools->Message Filters" if it doesn't fail on startup).



Reproducible: Always

Steps to Reproduce:
Example msgFilterRules.dat file that fails:

name="Spam - To \"\" foo"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/Junk"
action="JunkScore"
actionValue="100"
condition="OR (to,contains,\\\"\\\" foo)"

Making the "To:" rule without the escapes:

"" foo

will not match the target string, but does load OK:

version="8"
logging="no"
name="Spam - To \"\" foo"
enabled="yes"
type="1"
action="Move to folder"
actionValue="mailbox://nobody@Local%20Folders/Junk"
action="JunkScore"
actionValue="100"
condition="OR (to,contains,\"\\"\\" foo\")"

The above condition will display properly in the filters edit dialog, but again will NOT correctly match the target field.  (Hmmm, too many \" in there?)


Actual Results:  
See steps


This is a major bug, as it causes the loss of all defined filters.  The backup file in my case was empty, even though I have been defining filters for years.  When is the backup filter file updated, anyway?

Comment 1

[Russ Archer]

I have the same problem. If you try to filter out Toys"R"Us in the from field, the msgFilterRules.dat files is wiped out when you reopen Thunderbird. I entered the rule by backslashing each quote (\"). The rule shows up with a triple backslash.

I finally was able to create a rule by looking for Toys AND R AND Us in the from field. Of course, this will filter out a few messages besides Toys"R"Us, like Rusty Toys, but its better than nothing.

This problem is repeatable. It took me several days of experimenting to figure out what the problem was, but once I found the " problem and removed the rule, Thunderbird once again filters properly.

Comment 2

[Kent James (:rkent)]

With a couple of detailed reports, let's at least confirm this.

Comment 4

[:aceman]

What happens if you edit the file and rewrite the last line in the filter definition to:
condition="OR (to,contains,\"\\\"\\\" foo\")"

Comment 5

[Ludovic Hirlimann [:Usul]]

(In reply to :aceman from comment #4)
> What happens if you edit the file and rewrite the last line in the filter
> definition to:
> condition="OR (to,contains,\"\\\"\\\" foo\")"

ping ?

Comment 7

[:aceman]

I tested it and it really kills all the filters (if the backup is empty).

My proposed solution does not work, but this one does:
condition="AND (to,contains,\"\\\\"\\\\"\")"

When TB is started with this file, it shows the filter definition correctly. But after closing, it mangles the string again to:
condition="AND (to,contains,\\\"\\\")"

At next start the filters are killed again.

So there is some escaping problem when reading OR writing the definition.

Normally backslashes are not escaped in any way. Both of these definitions work:
condition="AND (to,contains,\"asdasd asda \ dkasdj k () asds\")"
condition="AND (to,contains,xx\xx)"

There could be a problem when there are more backslashes in succession.

In all these texts, "works" means the texts are preserved after restart as the user typed them in the filter editor. I have not tested if they match messages correctly.

Tested on TB12.

Comment 8

[:aceman]

See also bug 779438.

Comment 9

[:aceman]

I try to play with this a bit.
http://mxr.mozilla.org/comm-central/source/mailnews/base/search/src/nsMsgSearchTerm.cpp#398
http://hg.mozilla.org/comm-central/file/7d005aa5bd49/mailnews/base/search/src/nsMsgFilterList.cpp#l723
http://hg.mozilla.org/comm-central/file/7d005aa5bd49/mailnews/base/search/src/nsMsgFilterList.cpp#l451

Comment 10

[Binarus]

Is anybody working on this? It's 12 years old, has severity "critical" and no progress?

I just have run into the related bug https://bugzilla.mozilla.org/show_bug.cgi?id=779438 and would highly appreciate any solution. I wouldn't have a problem with entering complex filter criteria with all sorts of weird escaping, but there seems to be no workaround to the problem.

Comment 11

[Wayne Mery (:wsmwk)]

Binarus, does this still reproduce for you?

Comment 12

[Binarus]

Dear Wayne,

thanks for caring about the problem.

The problem I referred to (bug #779438) still exists. I have just tested and have created a filter with "Subject" "contains" "foo bar \SERVER bar foo". I saved the filter, closed TB and restarted it. The filter criterion then had been changed to "foo bar \SERVER bar foo". That is, one of the backlashes had been lost.

However, the situation in that other bug I referred to is not exactly the situation the thread we are currently reading is about. Here, the filters could not be read any more. I never encountered this; in my case, it just loses backslashes upon restart, as described in #779438.

Anyway, we have given up on filters in TB, because there were other issues with them that were more severe; we have switched to server-side filters for that reason.

Best regards,

Binarus

Comment 13

[Binarus]

The forum software got me. Of course, my filter was foo bar \\SERVER bar foo (When writing this as normal text, the forum software removes the second backslash). Sorry for the worry ...

To be clear: Have foo bar \\SERVER bar foo as the filter criterion, restart TB, and this becomes foo bar \SERVER bar foo; one backslash gets lost.

Comment 14

[Binarus]

Tested with 102.3.1 x64 under Windows 10 x64 Enterprise.