647790 - Allow filter to search on the entire message

Status: RESOLVED DUPLICATE of bug 453385

(MailNews Core :: Filters, defect)

Description

[Laurent Bauvens]

User-Agent:       Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9

I need to move messages into special folders on the fact they have a .PDF or a .DOC file attachment.

But Thunderbird filtering capability doesn't permit to create filter which allows to search:
 a) on the entire message. Even with customized keywords.
 b) on keywords of attachment header (like Content-Type). The search is restricted to 
    the message header.

Reproducible: Always

Steps to Reproduce:
1. Send to yourself a mail with a PDF attachment
2 [review]. Create a new filter
    Filter name: PDF
    Apply filter when: Checking mail or run manually
    Match all the following
3. Create a new search keyword
    Customize...
    Content-Type
4. "Content-Type" "contains" "application/pdf;"
5. Perform these actions
    Add Star 
6. Select the new filter and click on the button "Run now"
Actual Results:  
Nothing happens

Expected Results:  
Thunderbird should check all Content-Type keywords existing in the entire message, find the string "application/pdf;" and perform the Add-Star action.

Comment 1

[Ludovic Hirlimann [:Usul]]

When you run this anything coming out of the filter log ?

Comment 2

[Wayne Mery (:wsmwk)]

almost certainly a duplicate

Comment 3

[Wayne Mery (:wsmwk)]

dupe of bug 453385 or bug 559789 ?
actually, probably not far afield of bug 224392

Comment 4

[Coenraad de Beer]

(In reply to Wayne Mery (:wsmwk) from comment #3)
> dupe of bug 453385 or bug 559789 ?
> actually, probably not far afield of bug 224392

As a matter of fact, I won't say this is a dupe of those bugs you mentioned. It is not really a bug either, the filtering functionality lacks the ability to filter by attachment type, period. As far as I can recall, people have been requesting this feature since TB 3. We are at TB 11 and this feature is still not present. What are the chances of this feature being added soon?

Comment 5

[Coenraad de Beer]

I would like to correct myself: According to http://kb.mozillazine.org/Message_Filters#Filtering_the_message_body, you wee able to search attachments as well. I can understand that they redefined the meaning of Body (limiting the search to the actual body of the e-mail only), but why did they remove the ability to filter by attachment?

Comment 6

[Derek Shaw]

@ Coenraad (comment 5) a VERY good question.  And we are now at TB17.

Clearly there are numerous inter-related issues that are not really bugs, but examples of design failure.  Sadly, there is no mechanism to bring attention and external resources to bear upon such issues.  In other words, there appears to be no way to make it more attractive to for us to get the problem solved. 

https://bugzilla.mozilla.org/show_bug.cgi?id=453385#c8

"Yes, I had been watching this.  What I would like to do is to look at the body filter code, and see how difficult it would be to add the requested option.  Perhaps with a small change to the core code, I could then add this "filter raw body" as a custom search term."

https://bugzilla.mozilla.org/show_bug.cgi?id=224392#c25
"2012-07-16 08:51:52 PDT

This is difficult because with the current filter design, the message body is not available to filters at the time they are evaluated, and to accurately determine attachment name, type, and even existence, you need to do MIME processing over the entire message body. There is however the body search, that DOES read the body though is a complex way. Perhaps that could be extended to also do MIME processing as parts of its actions.

"This is an old request, and still very valid"

I believe that those of us who have worked on the core filter technology in the past are aware of this issue, and would love to be able to do the internal rework needed to make this happen. But it is not trivial."

Comment 7

[Thomas D. (:thomas8)]

(In reply to Coenraad de Beer from comment #5)
> I would like to correct myself: According to
> http://kb.mozillazine.org/Message_Filters#Filtering_the_message_body, you
> wee able to search attachments as well. I can understand that they redefined
> the meaning of Body (limiting the search to the actual body of the e-mail
> only), but why did they remove the ability to filter by attachment?

That documentation is very fuzzy, and I don't think we ever had a true ability to filter for names, content-type, or content of attachments. So I don't think any functionality was deliberately removed. I recall we once had problems (now fixed, iirc) that the msg body filter would sometimes match the raw mime source of attachments, even if encoded, so there was a random chance to search plaintext content of attached txt files, but otherwise a good chance for undesired mime garbage matches.

(In reply to Coenraad de Beer from comment #4)
> (In reply to Wayne Mery (:wsmwk) from comment #3)
> the filtering functionality lacks the ability
> to filter by attachment type, period. As far as I can recall, people have
> been requesting this feature since TB 3. We are at TB 11 and this feature is
> still not present. What are the chances of this feature being added soon?

Unfortunately very slim, unless you can fix it yourself. Mozilla is removing their paid staff from TB project, so we'll depend on volunteer contributors to do the work.

One thing we can do is to make it easier for those volunteer coders to find an entry point with clear-cut bugs which they can pick up. So we need to eliminate bugs that are duplicates, non-reduced, fuzzy, lacking direction, covering multiple issues etc. Unfortunately, this bug currently is not clear-cut and hence not actionable. So to support the request of this bug, it'll actually be best to close it because it's not clear and things are covered in other bugs.

Comment 8

[Thomas D. (:thomas8)]

(In reply to Thomas D. from comment #7)
> Unfortunately, this bug currently is not
> clear-cut and hence not actionable. So to support the request of this bug,
> it'll actually be best to close it because it's not clear and things are
> covered in other bugs.

(In reply to Laurent Bauvens from comment #0)
> I need to move messages into special folders on the fact they have a .PDF or
> a .DOC file attachment.

As Wayne said in comment 4, that's covered by bug 224392.
The problem of comment 0 is that problem and solution are mingled, and they differ in scope.

> But Thunderbird filtering capability doesn't permit to create filter which
> allows to search:
>  a) on the entire message. Even with customized keywords.

If that refers to the "raw source" of the entire message, then that seems covered by Bug 453385.

>  b) on keywords of attachment header (like Content-Type). The search is
> restricted to 
>     the message header.

Imo, that's a subset of a), so it's also covered by Bug 453385. If we can search "raw source" of entire message (bug 453385), then we can also search for those Conotent-Type headers of attached files.

> Reproducible: Always
> 
> Steps to Reproduce:
> 1. Send to yourself a mail with a PDF attachment
> 2 [©] [details] [diff] [review]. Create a new filter
>     Filter name: PDF
>     Apply filter when: Checking mail or run manually
>     Match all the following
> 3. Create a new search keyword
>     Customize...
>     Content-Type
> 4. "Content-Type" "contains" "application/pdf;"
> 5. Perform these actions
>     Add Star 
> 6. Select the new filter and click on the button "Run now"
> Actual Results:  
> Nothing happens
> 
> Expected Results:  
> Thunderbird should check all Content-Type keywords existing in the entire
> message, find the string "application/pdf;" and perform the Add-Star action.

Those expectations are flawed, because they misunderstand the current design.
Currently, by design, if you add a customized filter criterion using the "Customize Headers" dialogue, you are adding a "New message header" (as seen in that dialogue), i.e. TB will only search the *message header* for the message header field name that you enter. So by design, adding "Content-Type" (without quotes, without colon) as a custom header will search the header section of the msg for a header field named "Content-Type:" and match against the value of that field, which works correctly as expected. E.g., you can search Content-Type of the message for "multipart/mixed" (without quotes) to find messages with attached files.

Anyway, the scenario of searching "Content-Type" "contains" "application/pdf;" is also covered by bug 453385. Even if it weren't covered there, it's probably covered elsewhere, or we'd have to file a new, clearer bug that avoids the flawed notion of description in comment 0.

-> Dupe of bug 453385.

Finally, for the majority of users (probably including reporter of this bug), it will be sufficient if we address the main problem of Bug 224392, namely to just search for attachments based on attachment name and/or extension ([filename].pdf). Searching for the content-type of attachments ("Content-Type" "contains" "application/pdf;") is pretty technical and advanced.