224392 - Add ability to filter by attachment name (extension)

Status: NEW

(MailNews Core :: Filters, enhancement)

Description

[nick nelson]

User-Agent:       Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1) Gecko/20031009
Build Identifier: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1) Gecko/20031009

With the massive flow of trojans/spam going through, it would make sense to add
the ability to filter by attachment name (extension, ie *.exe) 

Reproducible: Always

Steps to Reproduce:
1.
2.
3.



Expected Results:  
Add this feature please? :)

Comment 1

[Kevin Christman]

(In reply to comment #0)

Thunderbird should by default, check to see if an attachment ends with .vbs,
.vbe, .js, .jse, .hta, .wsf, .wsh, .shs, .shb, .exe, .com.  If found,
Thunderbird should give a SERIOUS warning to the user, or even automatically
delete the mail/attachment.  

The "dangerous" file extensions list and Thunderbird's response (warn, delete,
etc.) could be configurable in either Tools > Options > Attachments or Advanced.

This will significantly reduce virus susceptibility for newbie users and help
fill the in the window for the fast-spreading viruses that slip through
un-updated antivirus software.

Comment 2

[Colin Ogilvie [:cso]]

(In reply to comment #1)
> Thunderbird should by default, check to see if an attachment ends with .vbs,
> .vbe, .js, .jse, .hta, .wsf, .wsh, .shs, .shb, .exe, .com.  If found,
> Thunderbird should give a SERIOUS warning to the user, or even automatically
> delete the mail/attachment.  

I would most certainly hope it would NOT delete the mail/attachment - if I wish
an email to be deleted, then it is my choice, not up to the whims and coding of
my email client.

Comment 3

[Nils]

I can only agree with the previous comments, a feature to control what to do
with messages containing certain attachments would be great. I really wouldn't
want Thunderbird to take away that responsibility from me.

Currently I get this filtering done by applying a filter on the message body
which searches for ".pif" or ".exe" or ".vbs" etc. The drawback here is that
this also detects messages mentioning these extensions in the message itself so
I have to check my virii folder once a week before allowing all filtered
messages to be deleted.

Comment 4

[Ryan Hoegg]

I'd love to see this feature in 1.0.

Kevin's idea for a virus warning is good.  Instead of building this into the
mail client, we could add some kind of "warn" actions to the mail filters and
include a default filter for these extensions when generating a new profile.

Also, this bug applies to all Hardware and OS, not just PC/Linux.

Comment 5

[will smith]

Attached, sample patch for this for Tbird 0.7.2.  

Currently there's no default warning, all that is added is that the user can now
manually filter for attachment filename however they want.  E.g. they could
create a filter to auto-delete 'attachment filename ends with .pif'.

This is my first attempt at coding for Mozilla, so although it works for me, I
think it needs a serious review, in particular string functions and character
encoding issues may be wrong.

I'm also getting the people from 
http://bugzilla.mozilla.org/show_bug.cgi?id=105169
involved, since that is a similar request (but larger scope), and for Mozilla
mailnews, not Thunderbird.

Could I also request that owner changes OS to 'All'?

Comment 6

[will smith]

Attachment: Adds ability to filter by attachment filename.

Also added a number of comments to the filtering code since I found it hard to
understand at first.

Comment 7

[Mike Cowperthwaite]

Comment on attachment 155632 [details] [diff] [review]
Adds ability to filter by attachment filename.

Clearing review+ on patch.
Nick, to get a review, you need to ask for one.  Set the '?' for the review,
and enter the requestee's email addr.

Comment 8

[Mike Cowperthwaite]

*** Bug 219889 has been marked as a duplicate of this bug. ***

Comment 9

[David :Bienvenu]

Thx for working on this. We can't break existing custom filters, so we need to
check if your change does that (I'm not convinced it does break them, since I
don't think we file out the numbers). Also, this only works for local mail, not
IMAP...and your diff is full of tabs and/or 8 space indent - we just use 2 space
indent.

So, can you just verify that you're not breaking custom filters, and if so,
attach a patch with the standard indent style, and I'll look at it more closely...

Comment 10

[will smith]

Attachment: Filter by attachment - updated to tbird 0.8

Working fine in 0.8 for me.

Haven't had time to check custom filters of fix indenting / tabstops level, as
requested.

Unfortunately I am moving from Asia (Malaysia) to UK, my computers are getting
shipped by boat (no laptop) and I won't get net access back ~until 6 November.

Will Smith

Comment 11

[will smith]

Comment on attachment 160039 [details] [diff] [review]
Filter by attachment - updated to tbird 0.8

I'm pretty sure the string searching code is inefficient - I'm new to Mozilla
string classes.

Comment 12

[will smith]

Comment on attachment 160039 [details] [diff] [review]
Filter by attachment - updated to tbird 0.8

I'm pretty sure the string searching code is inefficient - I'm new to Mozilla
string classes.

Comment 13

[Mark Banner (:standard8)]

For what it's worth, I've just tried appling the updated patch to the Mozilla
truck & running with the latest Mozilla/Mail news.

I copied some filter & custom set-ups from a 1.7 version (straight copy of
msgFilterRules.dat) and the appropriate pref, mailnews.customHeaders, and they
seemed to work file.

Couldn't find any reference to the numbers in the files, so it looks like it
should work ok. Suggest though someone else tries this just to make sure.

It'll be a good feature once it's in.

Comment 14

[Mike Cowperthwaite]

Comment on attachment 160039 [details] [diff] [review]
Filter by attachment - updated to tbird 0.8

Unsetting the duplicate request flag.

Comment 15

[Mike Cowperthwaite]

Will Smith: does your patch still build OK with the trunk, or has it bitrotted?

David, any chance you could give this a look?

Comment 16

[Mukunda]

I would like to see something like this make it into Thunderbird 2.0...

I recently saw a short tutorial / tip about a similar feature in apple's Mail.app. The tip was for filtering spam that uses images for the body of the message and then packs the end of the message with random text to bypass spam filters. On top of getting past the filters these messages also pollute the training data with random words not related to the message content. The common feature of these spam messages seems to be inline image attachments. If thunderbird could filter based on attachment content-type / file extension then we could easily set filter rules for these spam messages in addition to the other worthy uses for this capability.

This suggestion has been around for years now, what's the chance of giving it some attention?

Comment 17

[Phil Hannent]

(In reply to comment #16)
> I would like to see something like this make it into Thunderbird 2.0...
> 
> I recently saw a short tutorial / tip about a similar feature in apple's
> Mail.app. The tip was for filtering spam that uses images for the body of the
> message and then packs the end of the message with random text to bypass spam
> filters. On top of getting past the filters these messages also pollute the
> training data with random words not related to the message content. The common
> feature of these spam messages seems to be inline image attachments. If
> thunderbird could filter based on attachment content-type / file extension then
> we could easily set filter rules for these spam messages in addition to the
> other worthy uses for this capability.
> 
> This suggestion has been around for years now, what's the chance of giving it
> some attention?
> 
Just for the record, you can do this in Thunderbird 2 alpha 1:

http://www.neilturner.me.uk/2006/Aug/06/stopping_image_spam_in_th.html

Personally I have also added the clause "and not in personal address book" and have been happly running with it for months.

Comment 18

[Steve]

I have been a user of Eudora for over 10 years, and have a keen interest in it's further development and reincarnation as an open-source program. 

One feature which I had repeatedly requested from Qualcomm was the ability to apply actions to attached files based on a number of criteria. Essentially, my hope was to be able to apply actions (move, copy, delete, forward, rename, alert) based on any combination of criteria, including filename and/or extension, contents of headers including To address, From address, Subject, or message body contents.

A feature such as this could easily handle this feature request, and at the same time provide so much more power and flexibility. It could come with some default filters (potential virus alerts, for example) pre-installed and enabled by a simple checkbox. 

I'm not a programmer, but a business user and IT support person. Thanks for the opportunity to voice my support for this feature request, and hopefully expand it's scope.

Comment 19

[David :Bienvenu]

Sorry I lost track of this patch - I was just looking for it recently. 

I assume it doesn't work at all for imap messages, right?

Comment 20

[Dara Dorsaneo]

Could I jump in on this feature request and suggest that it also allows mime-type filtering. So you could detect embedded or attached video/pictures etc.

I opened bug 415891 to try to enable this but my submission was incorrect and it was resolved as invalid.

I think this request is closest to what I want to achieve and so would like it considered here.

Comment 22

[Wayne Mery (:wsmwk)]

David in comment #19
> Sorry I lost track of this patch - I was just looking for it recently. 
> 
> I assume it doesn't work at all for imap messages, right?

Patch author (Will) is gone.  Needs new owner.  (-> filters)

Comment 23

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Comment on attachment 160039 [details] [diff] [review]
Filter by attachment - updated to tbird 0.8

Patch has bitrotted:

$ patch -p1 --dry-run < ~/Desktop/tbird-0.8-filter-by-attachment-filename.patch 
patching file mailnews/base/search/public/nsIMsgSearchTerm.idl
Hunk #1 succeeded at 87 (offset 12 lines).
patching file mailnews/base/search/public/nsMsgBodyHandler.h
Hunk #1 FAILED at 6.
1 out of 1 hunk FAILED -- saving rejects to file mailnews/base/search/public/nsMsgBodyHandler.h.rej
patching file mailnews/base/search/public/nsMsgSearchCore.idl
Hunk #2 FAILED at 76.
Hunk #3 FAILED at 123.
Hunk #4 succeeded at 247 (offset 10 lines).
2 out of 4 hunks FAILED -- saving rejects to file mailnews/base/search/public/nsMsgSearchCore.idl.rej
patching file mailnews/base/search/resources/content/FilterEditor.js
Hunk #1 succeeded at 44 with fuzz 2 (offset 6 lines).
Hunk #2 FAILED at 439.
Hunk #3 FAILED at 562.
2 out of 3 hunks FAILED -- saving rejects to file mailnews/base/search/resources/content/FilterEditor.js.rej
patching file mailnews/base/search/resources/content/searchTermOverlay.js
Hunk #1 FAILED at 22.
Hunk #2 succeeded at 39 (offset 1 line).
Hunk #3 succeeded at 141 (offset -15 lines).
Hunk #4 FAILED at 212.
Hunk #5 FAILED at 225.
Hunk #6 succeeded at 306 with fuzz 2 (offset 32 lines).
Hunk #7 succeeded at 524 with fuzz 1 (offset 48 lines).
3 out of 7 hunks FAILED -- saving rejects to file mailnews/base/search/resources/content/searchTermOverlay.js.rej
can't find file to patch at input line 235
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -urp mozilla-0.8-orig-clean/mailnews/base/search/resources/locale/en-US/FilterEditor.dtd mozilla-0.8-willsmods/mailnews/base/search/resources/locale/en-US/FilterEditor.dtd
|--- mozilla-0.8-orig-clean/mailnews/base/search/resources/locale/en-US/FilterEditor.dtd        2004-06-20 00:41:38.000000000 +0800
|+++ mozilla-0.8-willsmods/mailnews/base/search/resources/locale/en-US/FilterEditor.dtd 2004-09-20 04:46:45.521264192 +0800
--------------------------
File to patch:

Comment 24

[Free]

This is an old request, and still very valid.

With more and more mails containing attachments, and often the main part of a mail is the attachment, it is quite painful to have to find a particular old email where a part of the name of that attachment is known.

Right now the only way I know of to do this, is to open the mail archive file in a text editor (which has a regex type search facility) and use that for searching, then scrolling up to the header information to find the sender/date, and locating the email that way.

I do imagine a lot of people need to both find emails that contain attachments whose names are partially known, or to apply filters on certain types of attachments.

It is hoped that implementing this would not be too difficult, and that one of the many much appreciated volunteers who make Thunderbird & Seamonkey such a delight to use, would find time to apply themselves to including this very useful feature.

Comment 25

[Kent James (:rkent)]

"It is hoped that implementing this would not be too difficult"

This is difficult because with the current filter design, the message body is not available to filters at the time they are evaluated, and to accurately determine attachment name, type, and even existence, you need to do MIME processing over the entire message body. There is however the body search, that DOES read the body though is a complex way. Perhaps that could be extended to also do MIME processing as parts of its actions.

"This is an old request, and still very valid"

I believe that those of us who have worked on the core filter technology in the past are aware of this issue, and would love to be able to do the internal rework needed to make this happen. But it is not trivial.

Comment 26

[Derek Shaw]

This has been requested again by 
https://bugzilla.mozilla.org/show_bug.cgi?id=647790

But it's a little more complicated (naturally) -- see comment 5 for that bug
https://bugzilla.mozilla.org/show_bug.cgi?id=647790#c5

In any event, filtering by attachments is way overdue, whatever form it takes.  It would be sad if the user chose to set up MSOutlook to filter for .docx attachemnts to autoreply that "docx attachments are not accepted here".  The ISP doesn't provide any filtering capabilities, and there is no budget for setting up and maintaining a separate mail server to do the job.

Comment 27

[kitchin]

Note filtering on "Attachment Status" is enabled *only if* you select the drop-down "Filter After Junk Classification." See my enhancement Bug 849232.

Given that, filtering by attachment name should be possible under the same circumstances?

Comment 28

[Kent James (:rkent)]

The problem is that attachment name (and for that matter attachment status) is only available after a message has been passed through a MIME parser. Currently, there is no mechanism (even in an extension using custom search terms such as FiltaQuilla) to do MIME processing of a message prior to a search, so information that relies on such a search cannot be used as a search term. This is a major architectural flaw that is not easily remedied.

The "Attachment Status" that is used as a search term is initially taken from a header that is unreliable, and therefore can change when the message is actually MIME parsed. That is undesirable behavior for a message filter, but is a compromise given the architectural limits.

Comment 29

[kitchin]

Kent, are you saying after-Junk filters use an unreliable Attachment Status? What about the paperclip icon in the message list? Is that reliable?

Comment 30

[Kent James (:rkent)]

(In reply to kitchin from comment #29)
> Kent, are you saying after-Junk filters use an unreliable Attachment Status?

Yes. In the initial parsing of an incoming message, "HasAttachment" is the same as "has content type multipart/mixed" generated by this code in nsParseMailbox.cpp:

1683           substring = PL_strcasestr(content_type->value, "multipart/mixed");
1684           if (substring)
1685           {
1686             uint32_t newFlags;
1687             m_newMsgHdr->OrFlags(nsMsgMessageFlags::Attachment, &newFlags);
1688           }

That is not a reliable indicator of has attachment, but it is the best that is available from just looking at the headers and not parsing the whole message.

> What about the paperclip icon in the message list? Is that reliable?

AFAICT this is only fixed when the message is viewed, from UI code such as this in nsMsgHdrViewOverlay.js:
 
607       // If we have an attachment, set the nsMsgMessageFlags.Attachment flag
608       // on the hdr to cause the "message with attachment" icon to show up
609       // in the thread pane.
...
619         this.mSaveHdr.markHasAttachments(true);

So I believe that this is only run when you actually click to view a message, but it is then saved and correct in the future (but I have not tested this). That also means that the paperclip icon is only valid after you have clicked on the message at least once.

The "After-junk" filter context was specifically added to allow filtering on junk processing terms like junkpercent and junkstatus. Although the bayes filter runs the message through the MIME parser, and therefore has knowledge of attachment status and attachment file names, there is no attempt to store that information for use in other parts of the program. So running the message through the bayes processor prior to filtering does not help to get accurate attachment information (though it would not be difficult to modify that code to store the attachment information in some way that was accessible by an after-junk search term or filter.)

This is all written from code examination and prior experience and not testing, so there is a small chance I have missed something here.

Comment 33

[Axel Grude [:realRaven]]

I just got this feature request for my Addon quickFilters:
https://www.mozdev.org/bugs/show_bug.cgi?id=26234

A question, are the performance implications for parsing the body (in order to extract attachment names) not the same as adding any filter that looks in the body itself? I would expect even just a single filter with a "Body" condition would require all incoming mail to be downloaded and parsed in full.

Comment 34

[Kent James (:rkent)]

(In reply to Axel Grude [:realRaven] from comment #33)
> I just got this feature request for my Addon quickFilters:
> https://www.mozdev.org/bugs/show_bug.cgi?id=26234
> 
> A question, are the performance implications for parsing the body (in order
> to extract attachment names) not the same as adding any filter that looks in
> the body itself? I would expect even just a single filter with a "Body"
> condition would require all incoming mail to be downloaded and parsed in
> full.

Funny, but as I was reviewing bugs and prior to seeing this question, I was thinking to myself what a great project it would be to change email processing so that MIME processing is done and stored on emails prior to hitting filters. Although there would be performance ramifications, generally I don't think that would be a major problem compared to the advantages of being able to filter on all characteristics, including particularly various attachment properties.

Comment 35

[Michael A. Koenecke]

It looks to me like the problem is that, some time ago, filtering was changed to only scan the text body of the message, contained within the defined boundaries (e.g., "b1_f03e825b3059fc0fe84a31b32977a18b" in a recent spam email I received). Understandable, because otherwise the scanner would have to run through potential megabytes of ASCII content. I wonder, though, if there might be a way to only scan, say, the first ten lines after the text boundary as well, which would allow one to evaluate the content of the attachment. One of the most annoying spam problems these days is innocuous-looking messages which contain viruses, either in .zip or other files. Services like SpamAssassin are not going to flag, say, messages allegedly from FedEx saying "We were unable to deliver your package. If one could define the body of the message, for scanning purposes, as "Boundary plus X number of characters," speed could be maintained but attachments could also be evaluated. Depends on the message format requirements, of course, but one would think it might be a possible approach.