328749 - Phishing Detector APIs

Status: RESOLVED FIXED

(Thunderbird :: Mail Window Front End, defect)

Description

[Scott MacGregor]

We should develop a set of phishing APIs to allow web service people to develop phishing extensions for Thunderbird that are web based instead of relying on the static checks the builtin detector uses. 

I don't envision us shipping by default with one of these built in services due to privacy concerns, but it would be nice to make it easy to support them. 

Even better if we could find some web service partners to help us shape these APIs!

Comment 1

[Håkan Waara]

Not sure if this is the general bug for the proposed phishing API service, but I think it would be an interesting idea to also let Camino use these APIs (as proposed in bug 329292 comment 18)!

CC pink, sfraser too.

Comment 2

[Scott MacGregor]

See:

http://wiki.mozilla.org/Thunderbird2:Phishing

Comment 3

[Daniel Veditz [:dveditz]]

*** Bug 349456 has been marked as a duplicate of this bug. ***

Comment 4

[Magnus Melin [:mkmelin]]

*** Bug 355629 has been marked as a duplicate of this bug. ***

Comment 5

[Scott MacGregor]

Attachment: work in progress

Work in progress. We don't have an official provider for the black and white list tables so I've left the actual provider preferences that initialialize and make the tables work out of this patch for now.

Design comments to follow.

Comment 6

[Scott MacGregor]

Thunderbird already has a set of simple static tests it can run on a message to determine if it's a scam. These test include:

1) IP addresses as URLs (http://101.233.5.2)
2) obscured IP addresses (hex encoded, etc.)
3) host name mismatches between the url and the associated link node text (<a href="http://myevilsite.com">http://www.mozilla.org</a>
4) HTML email containing form elements

Messages identified as spam get a big yellow status bar across the top identifying it as a scam and clicking on the actual suspicious url triggers a confirmation alert before the link will load.
 
We want to enhance what we currently have with locally downloaded black and white URL lists. We should leverage the code added to mozilla\toolkit\components\url-classifier for actually managing these tables for Firefox 2. It should be easy for extensions to plug and play additional providers for the tables. 

We are not going to support Firefox's feature for sending urls to a server for live phishing validation.

When a message is displayed, we'll analyze the url for any link nodes and any action urls associated with form elements. Each url gets looked up in the black and white list tables. If the url is in a black list, the message gets marked as a scam. If the url is in the white list we do nothing. If the url is in neither list, then we fall back to our existing static tests. If those tests conclude it's a scam, it gets marked as a scam. By leveraging the white list table, we can hopefully cut down on some of the false positives the static tests generate.

Comment 7

[Scott MacGregor]

Attachment: screen shot of the new phishing options UI

Screen shot with new strings for the options UI. I'll post a patch with the strings shortly so we can get them checked in before the upcoming string freeze.

Comment 8

[Scott MacGregor]

Attachment: add the new strings

Add the strings for this feature before the l10n freeze. These strings aren't used in the UI yet.

Comment 9

[Scott MacGregor]

Attachment: screen shot showing support for reporting phishing urls

I've added a context menu to the message pane that allows users to help improve our phishing detector by reporting phishing urls. Selecting this menu item runs the reportPhishingUrl associated with the current phishing provider in the default browser. The url that was selected is appended to the report phishing url in the form of &url=http://myevilsite.com. The url is encoded before it is appended.

This feature should be a great way for end users to help participate in making phishing support even better.

Comment 10

[Scott MacGregor]

Attachment: updated patch includes support for reporting phishing urls

I think this patch is ready for review now. See comment 6 for an overview.

Some notes:

1) I decided to call the new directory in mail\components "phishing" instead of "safebrowsing" even though the shared build flag with firefox is safebrowsing.

2) The new code in components\phishing is similar to the code in browser\components\safebrowsing. I've removed the code that supports remote look ups when porting it to Thunderbird.

3) The most interesting change in the patch is to: phishingDetector.js
* url look ups are now always async. When the look up completes, we'll call back into the message header code and set the phishing bar if approriate. We always check to make sure the current displayed message is the message that was loaded when the look up was initiated.
* We first perform our standard static tests on the url, regardless of the outcome, we still look up the url in our local table. That way, if a url fails the static tests, but is in the white list for the local table, we won't mark it as a scam.
*If the message contains forms, instead of flagging it as a phish, we now examine each action url in the form and look up the url, judging it on its own merits.

4) Depending on the performance we get from the local lists, it might be interesting to explore relaxing our static tests which tend to be very aggressive. Or maybe we have prefs to individually turn each static test off.

5) For now, we don't have a phishing provider we can use, so I've left the prefs for the provider urls empty. We can check those in later, when we've got a provider we can hook up.

6) New Options UI for e-mail scam settings (see the screen shot)

7) We now have a context menu item that appears for any non mailto link for reporting e-mail scams. It appends the url to:
http://{moz:locale}.phish-report.mozilla.com/?hl={moz:locale}

8) We need to build the url-classifier component in toolkit for Thunderbird. This is done by defining MOZ_SAFE_BROWSING in configure.in

9) packages-static includes the new files

Comment 11

[David :Bienvenu]

Comment on attachment 247114 [details] [diff] [review]
updated patch includes support for reporting phishing urls

very cool!

this copyright doesn't seem right:

+# The Initial Developer of the Original Code is
+# Netscape Communications Corporation.
+# Portions created by the Initial Developer are Copyright (C) 1998
+# the Initial Developer. All Rights Reserved.

does it make sense to use the browser pref here?

+const kPhishWardenEnabledPref = "browser.safebrowsing.enabled";

do we care about ipv6 addresses like we care about ipv4 addresses?

Comment 12

[Scott MacGregor]

(In reply to comment #11)
> (From update of attachment 247114 [details] [diff] [review] [edit])

> does it make sense to use the browser pref here?
> 
> +const kPhishWardenEnabledPref = "browser.safebrowsing.enabled";

I did this intentionally. Not sure if it is the right move though. My thought was if external service providers want to hook themselves up to Firefox as an extension, they wouldn't have to do anything for that same extension to work for Thunderbird too. So I left all of the browser.safebrowsing prefs with the same name as in Firefox. 
> 
> do we care about ipv6 addresses like we care about ipv4 addresses?

I just filed Bug 362434 to keep track of that. That would be nice to have.

Comment 13

[Scott MacGregor]

*** Bug 351828 has been marked as a duplicate of this bug. ***

Comment 14

[Thomas Bertels]

I suppose that it comes from the port, but are Firefox-oriented links intended to stay ?

Comment 15

[Phil Ringnalda (:philor)]

Bug 387196 has bitrotted this, so we now get console-spew of "Error: this.listManager_.safeExists is not a function".

Comment 16

[Scott MacGregor]

Thanks for pointing that out Phil. I'm re-syncing our phishing detector code to match the changes made for Firefox right now.

Comment 17

[Scott MacGregor]

Attachment: [checked in]possible fix

this ports the recent firefox changes to thunderbird.

Comment 18

[Scott MacGregor]

Comment on attachment 276292 [details] [diff] [review]
[checked in]possible fix

Lemme know if you still see this error tomorrow Phil.

Comment 19

[Phil Ringnalda (:philor)]

Took me slightly more than tomorrow, but yes, a nice clean error console now. Thanks!

Comment 20

[Scott MacGregor]

Attachment: [checked in]additional fix

My previous patch wasn't quite right, I just checked in a fix.

Comment 21

[Magnus Melin [:mkmelin]]

I think reasonable to mark this fixed, and take additional work (whatever that may be, i don't see much of a plan for those here) to another bug.