402435 - phishing protection seems to be fundamentally broken

Status: VERIFIED FIXED

(Toolkit :: Safe Browsing, defect)

Description

[Bill Gianopoulos [:WG9s]]

Refiling since my previous bug 401642 has been hijacked.

The URL:

http://rngrmyk.com/images/yahoo_login/YAHOOPHO.HTM

Appears to be in the google phishing site database as it is correclty blocked by Firefox 2.0.0.9.

It is not blocked by current trunk.

Comment 1

[Bill Gianopoulos [:WG9s]]

The alpha8 milestone build also correctly identifies this as a phishing site.

Comment 2

[Bill Gianopoulos [:WG9s]]

I have tried this both with the use downloaded list and ask Google every time options.  Makes no difference.  Firefox 2.0.0.9 and GranParadiso alpha 9 still work, current trunk does not.

Comment 3

[Bill Gianopoulos [:WG9s]]

I also pulled http://sb.google.com/safebrowsing/update?version=goog-black-url:1:1 and verified that http://rngrmyk.com/images/yahoo_login/YAHOOPHO.HTM is present in the blacklist.

Comment 4

[Bill Gianopoulos [:WG9s]]

(In reply to comment #2)
> I have tried this both with the use downloaded list and ask Google every time
> options.  Makes no difference.  Firefox 2.0.0.9 and GranParadiso alpha 9 still
                                                                   ^^^^^^^
                                                                   alpha 8
> work, current trunk does not.
> 

Comment 5

[Bill Gianopoulos [:WG9s]]

I verified bug 399233 as causing this regression via backout.

Comment 6

[Dave Camp (:dcamp)]

Do you have "check every time" enabled?  399233 would have broken that, and it could hide missing db info.

It looks like the database google is feeding us with the updated protocol is different from what they're serving in goog-black-url.  Specifically, it appears that all the zapak.t35.com sites have been collapsed into a single domain blacklist entry.  (because they only send hashes, we can't just compare the lists, so that was the result of guessing and checking).

I've found a couple items from the list that DO seem to work:

* zapak.t35.com is a whole-domain blacklist entry now
* http://classifiedbillboardadvertising.com/Yahoo.htm is blocked, suggesting that not-whole-domain blacklist entries (at least partially) work too.

There IS an entry for a URL at rngrmyk.com.  The new protocol only sends hashes, so I don't know what URL they've blacklisted in that domain.  I've sent them email asking, I'll know more once I get a response.

Comment 7

[Dave Camp (:dcamp)]

Attachment: fix whole-domain matches

However, whole-domain matches do appear to be busted.

Comment 8

[Tony Chang (Google)]

Comment on attachment 287350 [details] [diff] [review]
fix whole-domain matches

Please add a comment explaining this and some unit tests.

Comment 9

[Mike Beltzner [:beltzner, not reading bugmail]]

Blocking and accepting M9 target, but do we think this really blocks the beta? Are we issuing the beta release with a guarantee of Safe Browsing data?

Comment 10

[Frédéric Buclin]

(In reply to comment #9)
> Blocking and accepting M9 target, but do we think this really blocks the beta?
> Are we issuing the beta release with a guarantee of Safe Browsing data?

More users are going to use Fx3b1 as their daily browser (because beta sounds more stable than alpha) and so it would make sense to protect them as much as possible, especially when you know the patch is ready yet.

Comment 11

[Dave Camp (:dcamp)]

Attachment: now with unit tests and a comment.

Comment 12

[Dave Camp (:dcamp)]

I think we should probably put this patch in, full-domain blacklist entries are pretty common.

I'd like to hear back from google about which URL is being blocked in rngrmyk.com.  I'd say we should block beta if we're failing to catch properly blacklisted URLs, but not block if the data we're being fed is just incomplete.

Comment 13

[Dave Camp (:dcamp)]

It was pointed out that as an M9 blocker I didn't need approval.

Leaving this bug open until we figure out what's up with the rngrmyk.com entry.

Checking in src/nsUrlClassifierDBService.cpp;
/cvsroot/mozilla/toolkit/components/url-classifier/src/nsUrlClassifierDBService.cpp,v  <--  nsUrlClassifierDBService.cpp
new revision: 1.35; previous revision: 1.34
done
Checking in tests/unit/test_dbservice.js;
/cvsroot/mozilla/toolkit/components/url-classifier/tests/unit/test_dbservice.js,v  <--  test_dbservice.js
new revision: 1.6; previous revision: 1.5
done

Comment 14

[Dave Camp (:dcamp)]

Garrett at Google tells me that the URL on the list is rngrmyk.com/images/ViewItem.html, which is blocked correctly on trunk.

They're looking in to why the YAHOOPHO.HTM url isn't appearing on the new list, but that's a separate (and not our) bug.

Comment 15

[Tony Chung [:tchung]]

Verified fix on Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:1.9a9pre) Gecko/2007110520 Minefield/3.0a9pre.

Navigating to http://classifiedbillboardadvertising.com/Yahoo.htm takes me to a phishing attack page with the big red yield sign.

Comment 16

[Bill Gianopoulos [:WG9s]]

REOPENING

Since the original issue as I described it in the initial comment still exists, I do not understand the justification for marking this bug fixed.

Comment 17

[Reed Loden [:reed]]

Bill, are you sure that site is even in the blacklist? The blacklist URL changed, so the URL you used in comment #4 isn't the right blacklist database anymore. If you can figure out the new blacklist URL and check if that site is on it, that would be helpful. If it's not on the list, it isn't a Firefox bug, but a Google bug.

Comment 18

[Dave Camp (:dcamp)]

The new blacklist is not publically available, but I verified that the URL in the initial comment is not on the new list.  It's a google bug.

Comment 19

[Bill Gianopoulos [:WG9s]]

(In reply to comment #18)
> The new blacklist is not publically available, but I verified that the URL in
> the initial comment is not on the new list.  It's a google bug.
> 

IF i configure the browser to ask Google, rather than using the database, if I use the current trunk it does not say it is a phishing site, yet if I use a build from before this checkin (like alpha8 for example), it says it is.  Is the query every time code using a different Google list that it used to?

Comment 20

[Bill Gianopoulos [:WG9s]]

OK.  Let's try it this way.  can anyone please give me one URL (other than the hardcoded one) that triggers the anti-phishing alert in a current trunk build?

Comment 21

[Dave Camp (:dcamp)]

The query-every-time code checks the old list, and was (purposely) disabled in the bug you cited.

(In reply to comment #20)
> OK.  Let's try it this way.  can anyone please give me one URL (other than the
> hardcoded one) that triggers the anti-phishing alert in a current trunk build?

As mentioned in comments #6, #13, and #14,  anything in zapak.t35.com, http://classifiedbillboardadvertising.com/Yahoo.htm, and http://rngrmyk.com/images/ViewItem.html should be blocked.

Obviously the list might change in the future so these probably aren't good long-term testing URLs, but that is a problem for 401642.

Comment 22

[Bill Gianopoulos [:WG9s]]

Sorry for the bugspam.  I am really not trying to be a pin in the ass here, but int the past there was a n anti-phishing bakeoff between Firefox 2 and IE 7. and IE 7 came of looking rather pathetic.

What I am afraid of happening is that when the BETA comes out.  Someone is going to run a test like that again, and I suspect Firefox 3 is going to look bad even versus IE7 but especially versus Firefox 2.

This is not going to look good, and it probably won't matter to anyone whose fault it is.

I have tried various known phishing sites form various list sources and so far the current trunk has failed to identify any of them as phishing sites.  Firefox 2 got most of them.

Comment 23

[Bill Gianopoulos [:WG9s]]

(In reply to comment #21)
> As mentioned in comments #6, #13, and #14,  anything in zapak.t35.com,
> http://classifiedbillboardadvertising.com/Yahoo.htm, and
> http://rngrmyk.com/images/ViewItem.html should be blocked.

None of the abouve come up as blocked for me either under Windows or Linux using the BETA 1 builds form overnight.

Comment 24

[Bill Gianopoulos [:WG9s]]

(In reply to comment #23)
> (In reply to comment #21)
> > As mentioned in comments #6, #13, and #14,  anything in zapak.t35.com,
> > http://classifiedbillboardadvertising.com/Yahoo.htm, and
> > http://rngrmyk.com/images/ViewItem.html should be blocked.
> 
> None of the abouve come up as blocked for me either under Windows or Linux
> using the BETA 1 builds form overnight.
> 
Oh and i did this with a fresh profile so i would not be using a urlclassifer database created by a previous version

Comment 25

[Bill Gianopoulos [:WG9s]]

(In reply to comment #21)
> The query-every-time code checks the old list, and was (purposely) disabled in
> the bug you cited.
Oh wait if the query every time code is disabled then this IS completely broken now because, as I cited in bug 402469, if you start with a fresh profile, the urlclassifer3.sqlite file never appears to get completely loaded.  So, if the query every time option is bypassed and the only option is to use the database that isn't there ...	

Comment 26

[Dave Camp (:dcamp)]

Yeah, as I just mentioned in 402469, it will take awhile for the complete database to be pulled down :/

Comment 27

[Bill Gianopoulos [:WG9s]]

(In reply to comment #26)
> Yeah, as I just mentioned in 402469, it will take awhile for the complete
> database to be pulled down :/
> 

Well I have been waiting about a week now.  Seems to load maybe 50KB each time i use the browser.

Comment 28

[Bill Gianopoulos [:WG9s]]

OK re-resolving as it appears my real issue is my other bug 402469.

Comment 29

[Reed Loden [:reed]]

(In reply to comment #18)
> The new blacklist is not publically available, but I verified that the URL in
> the initial comment is not on the new list.  It's a google bug.

When will this be fixed? http://rrnryspace.com/index.cfm-fuseaction314Dlogin.process8526MyTokens79843964886883084155.htm shows up as a phishing site on branch but not on trunk. That's just sad that we have a far less useful anti-phishing feature on trunk.

Comment 30

[AndrewM]

(In reply to comment #21)
> As mentioned in comments #6, #13, and #14,  anything in zapak.t35.com,
> http://classifiedbillboardadvertising.com/Yahoo.htm, and
> http://rngrmyk.com/images/ViewItem.html should be blocked.

I have just been catching up with my bugmail, and rather alarmingly, when I just tried http://classifiedbillboardadvertising.com/Yahoo.htm, it did not show up as a phishing site. And the site itself is still up, so anyone who is using a nightly (or perhaps Beta 1? I haven't installed it) with phishing protection enabled could potentially still be exposed to this fraudulent Yahoo! Mail login page. Can anyone else confirm if it is blocked?

The good news is that http://zapak.t35.com/, http://rngrmyk.com/images/ViewItem.html and http://rrnryspace.com/index.cfm-fuseaction314Dlogin.process8526MyTokens79843964886883084155.htm all show up as phishing sites for me.

I should emphasize that this is with my months-old profile, so the urlclassifier3.sqlite has had ample time to download. (Although having said that, I did notice something strange which might or might not be a bug that I mentioned in bug 402469 comment 13.)

Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b2pre) Gecko/2007112305 Minefield/3.0b2pre ID:2007112305