Bug 279191 (mailphish)

Add Phishing Detection Support to Thunderbird

RESOLVED FIXED in Thunderbird1.1

Status

Thunderbird
Mail Window Front End
RESOLVED FIXED
13 years ago
6 years ago

People

(Reporter: Scott MacGregor, Assigned: Scott MacGregor)

Tracking

Trunk
Thunderbird1.1

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(6 attachments)

(Assignee)

Description

13 years ago
Get a phishing detector going for Thunderbird.

I'm sure it can be improved quite a bit but this starts to catch some of the
more obvious scams.
(Assignee)

Updated

13 years ago
Status: NEW → ASSIGNED
Target Milestone: --- → Thunderbird1.1
(Assignee)

Comment 1

13 years ago
Created attachment 171936 [details] [diff] [review]
first cut at a phishing detector

When the user clicks on a URL that we think is a phishing URL, he now gets
prompted before we open it. Handles two cases so far. Hopefully we can add more
as we figure out how:

1) The host name of the actual URL is an IP address
2) the link text is a URL whose host name does not match the host name of the
actual URL.

I added support for a silentMode so later on we can hopefully walk an existing
message DOM and call into this routine on each link element in the DOM. This
would allow us to insert an email scam warning bar in the message window down
the road.
(Assignee)

Comment 2

13 years ago
Created attachment 171937 [details]
screen shot of the 2 dialogs
(Assignee)

Comment 3

13 years ago
Comment on attachment 171936 [details] [diff] [review]
first cut at a phishing detector

This patch just gets the ball rolling and is better than nothing. Hopefully
we'll continue to improve it by adding more checks.

Read my comments up above for more details about the patch.
Attachment #171936 - Flags: superreview?(bienvenu)

Updated

13 years ago
Attachment #171936 - Flags: superreview?(bienvenu) → superreview+

Comment 4

13 years ago
Maybe add some graphic of warning nature, and/or a very short timer like one
when installing Fx extension (not too long/annoying)
Blocks: 254913
(In reply to comment #3)
> (From update of attachment 171936 [details] [diff] [review] [edit])
> This patch just gets the ball rolling and is better than nothing. Hopefully
> we'll continue to improve it by adding more checks.
> 
> Read my comments up above for more details about the patch.
> 

FYI - I'm working on porting this to seamonkey.

Comment 6

13 years ago
Maybe you can do some lookups with the URI's against the SURBL database
http://www.surbl.org/. 
If you consider using SURBL, then you might want to consider a local whitelist
of domains not to query for
Comment on attachment 171936 [details] [diff] [review]
first cut at a phishing detector

(hmm, phishingDetector.js does not mention an original author... shouldn't it?)

+  var hrefURL = Components.classes["@mozilla.org/network/standard-url;1"].
+		 createInstance(Components.interfaces.nsIURI); 
+  
+  hrefURL.spec = href;

is there a reason to avoid nsIIOService::newURI? there are other types of URIs
than standard-url...

Updated

13 years ago
Component: Message Compose Window → Mail Window Front End
OS: Windows XP → All
Hardware: PC → All
Version: unspecified → Trunk

Comment 8

13 years ago
I think that showing both URLs in the first dialog isn't a good idea. Users only
skim read the detailed text of warning boxes like this one and there's a risk
that they'll just see "www.mybank.com", think "that's what I was expecting", and
click "Yes". It also makes the message longer than it needs to be.

If we just said "Thunderbird thinks this link is suspicous. This link is to the
site www.phishersrus.com, not to the site shown in the e-mail." users are more
likely to have that jarring "phishersrus - who on earth are they?" moment, which
will hopefully make them pay enough attention to work out that they probably
want to click "No".

Also we might try to go for more user-oriented terms than "host" ("computer",
"site"?) and URL (link?). 

All that said this is a great idea.

Comment 9

13 years ago
Hrm... I think some careful testing of this stuff should be done against legit
emails. In particular, URLs containing an IP address are something that
spamassassin scores on, but I've actually had to lower the score for it (on the
mailserver I run at work) due to false positives. Some legitimate newsletters
and solicited marketing emails seem to use numeric IPs. Why they do that I have
no idea, but checking for that alone will have false positives, which in turn
will desensitise people to the warnings. I guess we can see how that goes...

Comment 10

13 years ago
(In reply to comment #1)
> When the user clicks on a URL that we think is a phishing URL, he now gets
> prompted before we open it. Handles two cases so far. Hopefully we can add more
> as we figure out how:
> 
> 1) The host name of the actual URL is an IP address
> 2) the link text is a URL whose host name does not match the host name of the
> actual URL.
> 

Phishers have recently taken to using spyware / malware to override DNS by
adding entries to users' HOSTS files (see
http://news.com.com/Phishers+learn+new+tricks/2100-7349_3-5544193.html), so
another useful check would be to see if the URL domain has an entry in the
user's HOSTS file.

For performance you could read and cache the HOSTS file at startup and
optionally refresh it periodically.

Comment 11

13 years ago
Thanks, Scott, for implementing this.

Once I implemented bug 116242 (if ever), that would be useful to determine, if
there's a URL in the link text, even if there's additional text before and after
it or if it's abbreviated.

> Phishers have recently taken to using spyware / malware to override DNS by
> adding entries to users' HOSTS files

If that happened, the war is lost.

I agree with comment 8.

Another counter-measure would be to simply completely remove suspicious links
(the <a>, not the text).
mscott:

You may want to check out:
http://reportphish.org/

data is available to researchers... there might be some information available
through them (now or in the future) that might help build good testcases.

Getting a good list of testcases could help take this tool to the next level.

Comment 13

13 years ago
(In reply to comment #1)
> When the user clicks on a URL that we think is a phishing URL, he now gets
> prompted before we open it. Handles two cases so far. Hopefully we can add more
> as we figure out how:
> 
> 1) The host name of the actual URL is an IP address

I also believe a local whitelist mechanism will be necessary. If the IP is for a
site that the user visits often (and they are always warned with a prompt),
mozilla will soon start receiving complaints (if there is no clear way to turn
off prompts for that site).

Comment 14

13 years ago
You may like to look at this:
http://www.eudora.com/email/features/scamwatch.html
.. to see how Eudora already handles this. The main addition to the Thunderbird
proposal is the use of a popup warning when the mouse is over a suspect link.
This way the user is warned before they even click the link that it may be bogus.
 I don't think the top-level domain check they use is good, especially for
non-US users (there are real sites like ebay.com.au, yahoo.co.uk etc.. )

 Something I would like to see is altering the colour of a suspect link to
something non-default (eg. bright red), so that they stand out, especially
compared to other non-suspect URLs in the same message.
 This is because many scam emails I see now have several legit links in them
(usually to the home page or privacy sections of a website), but the "click here
to login" one is bogus.
 If bogus links were a different colour, then dodgy links would become obvious
instantly (without even moving the mouse over them, let alone clicking on them),
and you might even get into the habit of instantly recognizing a phishing email
by seeing red links in it as soon as you opened the message
(html colouring of links aside of course).

Comment 15

13 years ago
What about working together with http://www.antiphishing.org/ 
Looks like they have a database of phishing mails
+  if (!aSilentMode) // allow the user to over ride the decision
+    isPhishingURL = confirmSuspiciousURL(phishingType, hrefURL, linkTextURL.value);

I believe that should be:
+  if (!aSilentMode && isPhishingURL) // allow the user to over ride the decision

Otherwise, linkTextURL.value isn't set, and you get a JS error:
JavaScript strict warning: chrome://messenger/content/phishingDetector.js, line
71: reference to undefined property linkTextURL.value

There's no need to call the confirmation function if we don't think it's a
phishing URL.

Comment 17

13 years ago
Scott, this is a great idea. My apologies for the unsolicited yap, but I really
want to make a couple of UI suggestions that build on comment 4, comment 8 and
comment 14.

When the user mouses over or tabs onto a could-be-phishing link, the link should
change from 

www.imaphisher.com

to

!!! www.imaphisher.com 

where !!! is a small (character height) warning sign. The icon should be
platform-specific. For Windows, see the warning sign under
http://msdn.microsoft.com/library/en-us/dnwue/html/ch09f.asp. Sounds like a job
for CSS. This is complementary to the warning bar you're thinking about for the
message window, because they would be seen at different times by the user.

When displaying the message box itself, use the same warning sign, not the
question mark sign that's used now. The question mark is misleading, because it
downplays the gravity of the message, and contrary to platform UI guidelines
(link above), which advise that the question mark is just bad in general and
shouldn't be used at all.

Finally, here's a suggestion for the text itself.

Thunderbird thinks this link is suspicious! [Note emphasis through exclamation
and blank line following this sentence.]

It may be trying to impersonate the web page you want to visit. [Note
"impersonate", an active verb.] Are you sure you want to go to
www.imaphisher.com? [Use only the hostname, not the full URL. Highlight the URL
using a warning color like bright yellow.]
Will this work in Mozilla Mail (Suite)?

Comment 19

13 years ago
It's a great idea. 

I think the following jargon will cause many users' eyes to glaze over:  URL,
host, numerical ip address.  

They won't understand and we'll lose their attention.  A quick rewrite in
end-user languange:

   Thunderbird thinks this link is suspcious.  The actual addresss,
   real.domain.tld, is different than the one displayed in the e-mail,
   fake.domain.tld.  Are you sure you want to visit this link?

   Thunderbird thinks this link is suspcious. The link's address,
   123.123.123.123, uses numbers.  Most legitimate links use
   names. Are you sure you want to visit this link?"


More users will understand it.  Sophisticated users will know we're talking
about URLs, hosts and numerical IPs.

(I'm aware that this sentence isn't precise: "The link's address,
 123.123.123.123, uses numbers", but unsophisticated users won't notice, 
and sophisticated users will understand anyway.)

Comment 20

13 years ago
(In reply to comment #2)
> Created an attachment (id=171937) [edit]
> screen shot of the 2 dialogs
> 

Just a small suggestion:
Wouldn't it be better to have an exclamation mark icon instead of a question
mark baloon icon on the dialog? Or wouldn't that be correct if you look at the
purpose of the dialog? (I don't know what the official dialog icon policy is)

Comment 21

13 years ago
(In reply to comment #15)
> What about working together with http://www.antiphishing.org/ 
> Looks like they have a database of phishing mails

Another site that provides this sort of feature is http://toolbar.netcraft.com/

They don't seem to have an open API, but I imagine the MoFo could work with them
to get access to this "black list" of phishing sites.

Comment 22

13 years ago
Ref:  Comment #10

If this means filtering against my HOSTS file, please don't!  I use it as a
local DNS table to speed access to obscure domains.  Any domain that I see in my
status bar with "Looking up" for more than a second or two gets added.  

Comment 23

13 years ago
By populating the hosts file you are perpetuating the problem.

The more you access the domain, the more it will be cached on your ISPs DNS
servers. [ Sorry for the spam. ]

Comment 24

13 years ago
By populating the hosts file you are perpetuating the problem.

The more you access the domain, the more it will be cached on your ISPs DNS
servers. [ Sorry for the spam. ]

Comment 25

13 years ago
I agree with #11.  Just mark the link dead and provide no ability to click
through at all.  If the user insists on doing it, make them work (perhaps a menu
option, but a cut&paste of the text would be fine too).

As soon as you get aggressive with link sanity, people will start sending out
more responsible links.  IMHO.  (I also agree that the message needs to be
pretty darn basic, and blue is the wrong colour.  Go for bright flashing red as
JPM says:  http://www.financialcryptography.com/mt/archives/000312.html in the
comment)

Excellent stuff, Scott!

Comment 26

13 years ago
i suggest adding a "Help..." or "more info..." button with links to pages
explaining the phenomenom of phishing in more detail to average users. keep in
mind that phisinhg mails often look very official, and users might be confused
that "Thunderbird thinks" they're suspicious. besides that, great work!
(Assignee)

Comment 27

13 years ago
Created attachment 172409 [details] [diff] [review]
support for obscured IP addresses

scammers frequently obscure the IP address by using a dword for the host name,
hex encoding the IP address or octal encoding the IP address. They even do a
mixture of  both (i.e. hex.oct.oct.hex)

This patch addresses that case.

Test cases for this patch:

“dword” – http://3515261219/
“octal” – http://0321.0206.0241.0043/
“hexadecimal” – http://0xD1.0x86.0xA1.0x23/ or http://0xD186A123/
"mixed format" - http://0321.0x86.161.0043

Comment 28

13 years ago
Ref: comment #23 (repeated in #24)

I am using the Hosts file the way it was initially intended.  In company
internal networks, it is common for some destinations have no domain names, only
IP addresses.  For referring to those destinations symbolically, users create
their own names in their local Hosts files.  

Putting bogus entries in Hosts files to block access to undesirable domains is a
consequence of how the files are used, not their real purpose.  

Thus, any fix that uses the local Hosts file as a blacklist would be contrary to
the purpose of that file and would have the effect of making the product
unusable in large organizational networks.  

By the way, my daily refresh of my Hosts file to get any changes to IP addresses
does send the domain name to or through my ISP's DNS.  

Comment 29

13 years ago
(Crossposted by accident on https://bugzilla.mozilla.org/show_bug.cgi?id=254913#c16)

Some IMAP Anchors seem to trigger the phishing warning when using an internal IP
address (192.168.0.1) as IMAP server address.

The anchor link was:
imap://valentijn@192.168.0.1:143/fetch%3EUID%3E.INBOX.Trash%3E5099#32816
And triggered the anti-phishing dialog reporting that the link was directed to
192.168.0.1 (our internal mail server).

I think this warning should not be triggered when the target address is the same
as the IMAP server.
(Assignee)

Comment 30

13 years ago
Created attachment 173686 [details] [diff] [review]
only check http and https urls

this additional change fixes comment #29 where non http urls are incorrectly
getting identified as phishing urls (i.e. imap urls....)

Comment 31

13 years ago
(In reply to comment #30)
> only check http and https urls

Would it be useful to include ftp in this check to prevent spoofing of download
links?

Comment 32

13 years ago
(In reply to comment #31)
> (In reply to comment #30)
> > only check http and https urls
> 
> Would it be useful to include ftp in this check to prevent spoofing of download
> links?
> 

Not only FTP, also IRC and others?
Why not check ALL url's except the internal ones?
(Assignee)

Comment 33

13 years ago
*** Bug 282362 has been marked as a duplicate of this bug. ***

Comment 34

13 years ago
Created attachment 174809 [details]
bugzilla email

I've just received a bugzilla email (attached), that's triggered the 'scam'
bar, and I can't see any reason why it has, although coincidentally its the
homograph bug...:

all link text matches link targets
no ip address links

Comment 35

13 years ago
(In reply to comment #34)
> Created an attachment (id=174809) [edit]
> bugzilla email
> 
> I've just received a bugzilla email (attached), that's triggered the 'scam'
> bar, and I can't see any reason why it has, although coincidentally its the
> homograph bug...:
> 
> all link text matches link targets
> no ip address links

I've just received another one. It looks like the presence of any number in the
domain triggers the warning, not just an IP address.
(Assignee)

Comment 36

13 years ago
Created attachment 174903 [details] [diff] [review]
avoid false positives for domain names that contain digits

Good catch dougweb. If our calculated IP address for the url is 0.0.0.0 then we
should treat it as an invalid IPv4 address and as such not an email scam.
Here's a potential issue:

What about the many users with access points, and other products that are
capable of sending email's with links to do things (like view logs, or turn on
and off).

Perhaps blackhole IP blocks should be whitelisted?

Off hand I can think of a few :
192.168.x.x
10.10.x.x
191.8.x.x

I belive there are some others.

Comment 38

13 years ago
Should IPs with the correctly linked DNS name also be blocked?
For example:
nslookup www.google.com
results in: 216.239.57.99, 216.239.39.99, 216.239.37.99

For example, if a link points to 216.239.37.99 but the linked text says it's
www.google.com, this link is certainly not a scam link.

Comment 39

13 years ago
Is there a repository of phishing scam email messages somewhere that are being
referenced when deciding which criteria will be used for this detector, or is it
just being assumed that email messages with IP addresses in them are phishing
scams and that phishing scams do not contain FTP links without any reference to
real world data?
(Assignee)

Comment 40

13 years ago
http://antiphishing.org/
(Assignee)

Comment 41

13 years ago
and I also use:

http://www.millersmiles.co.uk/

Comment 42

13 years ago
I had a look at those sites, but I didn't see any statistics, or any way of
retrieving any statistics very easily. Are the detection methods in Thunderbird
being developed based on an objective measurement of how phishing scams work, or
just "eyeballing" a few messages and guessing? I ask because the detection so
far has only insisted on pestering me about legitimate links and messages, which
I find annoying. Although I would love to remove this feature, I know that won't
happen, so I'm interested in making it as accurate as possible.

Comment 43

13 years ago
Why not check to see if the href, the status bar text, and the link text match
(if the link text purports to be a url? For instance, if we have an href of
http://1.2.3.4 and status bar text that reads http://secure.ebay.com/, then we
have a problem. However, if we have an href that reads http://badguysareus.com
and the status bar reads http://badguysareus.com, and the link text also reads
http://badguysareus.com, then we're fine.

I don't understand why Thunderbird wants to be so presumptious to assume that it
knows better than I what constitutes a scam. This sort of thing is *exactly*
what makes Microsoft products so annoying to use. They assume you're an idiot,
then they make design decisions based on that assumption, and they give you no
opportunity to override their decision. In effect, they're saying, "you're an
idiot and we know what's best for you ... and you will not be permitted to
decide otherwise."

It's fine if Thunderbird wants to make some attempt at warning clueless users
about phishing scams, but *please* let me turn it off at least.
(Assignee)

Comment 44

13 years ago
(In reply to comment #43)
> Why not check to see if the href, the status bar text, and the link text match
> (if the link text purports to be a url? For instance, if we have an href of
> http://1.2.3.4 and status bar text that reads http://secure.ebay.com/, then we
> have a problem.  

That's exactly what it currently does.

Of course you'll be able to turn it off. I just haven't added that yet. Maybe
you should be sticking to release builds if you aren't interested in waiting for
features to get fully implemented in nightly builds. 

Comment 45

13 years ago
(In reply to comment #44)
> Of course you'll be able to turn it off. I just haven't added that yet. Maybe
> you should be sticking to release builds if you aren't interested in waiting for
> features to get fully implemented in nightly builds. 

That's OK. I don't mind partially implemented features. I just became concerned
when I started seeing this. Knowing that it will be possible to disable it
alleviates my concerns. I was only concerned because in the past there have been
unilateral decisions made by the Mozilla team that were both unpopular and
unwanted. I was concerned that this would be the case here as well.

Comment 46

13 years ago
I've also noticed that any email with a <form> triggers the warning bar - that's
definately not right...

Comment 47

13 years ago
Scott, the phishing detector doesn't address email messages like the one Seth 
opened the other "detect phishing" bug for -- attachment 155939 [details] (bug 254913).  
In that case, the link is to a part of the email itself (not an anchor, but a 
MIME part).  Internally, it's a cid: link; when hovered, it shows as a mailbox: 
link.

I don't know if there's ever a legitimate reason to have a link to a part, 
particularly when that part is not shown as an attachment (multipart/related) -- 
but see bug 241572.

Technically, that message is not phishing so much as a trojan, but it's still a 
scam.  See also bug 276991.

Comment 48

13 years ago
Since when are programmers in charge of deciding whether there is a reason for
me to have email configured in any particular manner? If a message doesn't
violate a specification, leave it alone. I'm a user, not a moron. Make Tardbird
for the morons. Leave me something to use please.
(Assignee)

Comment 49

13 years ago
*** Bug 285665 has been marked as a duplicate of this bug. ***
Blocks: 286876
*** Bug 288817 has been marked as a duplicate of this bug. ***
Alias: mailphish
(Assignee)

Updated

12 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 12 years ago
Resolution: --- → FIXED

Comment 51

12 years ago
I vote for a whitelist as part of this feature

I like this feature, for people who don't know how to take care of themselves (a
lot of people, frighteningly enough).  However, I'm getting advertising email
that I *every time* have to click "Not a Scam" and then "Show Images" for.  Yes,
I like some advertising emails every now and then; they are fun.  If it works
this way then I think many users will get used with mindlessly clicking those
buttons.  Once that happens, what good are they?

For myself - I am forced to click *a lot*.  My vote is for a whitelist that
would be effective for all emails from that same domain/sender/something. 
Perhaps a question when you click "Not a Scam" if you want to apply that as a
preference from now on, for that domain/sender/something.  For the corporate
users perhaps a special preferences file somewhere that they could keep
up-to-date.  Great idea though!

Comment 52

12 years ago
Never mind Comment 51 - if I add the company in my address book, everything
works fine.
*** Bug 254913 has been marked as a duplicate of this bug. ***
No longer blocks: 254913
You need to log in before you can comment on or make changes to this bug.