1240027 - No phishing/malware warnings displayed on the test page after receiving the first database update

Status: RESOLVED INVALID

(Toolkit :: Safe Browsing, defect)

Description

[Vasilica Tamas, QA [:vtamas]]

Reproducible on: Firefox 46.0a1, Firefox 45.0a2 and Firefox 44 beta 9 across all platforms

STR
1.Launch Firefox with a clean profile.
2.Wait 5 minutes for the lists to finish downloading.
3.Access the following test page: https://testsafebrowsing.appspot.com/
4.Verify that the first 3 links under "WebPage Warnings" trigger the appropriate warnings.

ER
The first 3 links successfully trigger the appropriate warnings.


AR
[1] No warnings displayed for phishing/malware/unwanted software.
- See screenshots: http://i.imgur.com/Wvef7Ol.png 
                   http://i.imgur.com/mycaa6Z.png

[2] The 3rd link (http://testsafebrowsing.appspot.com/s/unwanted.html) triggers the adequate warning only on:
- Firefox 44 beta 9 under Windows XP 64-bit and Windows 8.1 32-bit.
- Firefox 46.0a1 under Ubuntu 12.04 64-bit. 
- See screenshot: http://i.imgur.com/sB05mTn.png


Additional notes:
- This issue is reproducible on Firefox 46.0a1(2016-01-14), Firefox 45.0a2 (2016-01-15) and Firefox 44 beta 9 (20160114165817) under Windows XP 64-bit, Windows 8.1 32-bit, Mac OS X 10.10.4 and Ubuntu 12.04 64-bit.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Confirmed on 43.0.4 also. Google API issue or something?

Comment 2

[Matt Wobensmith [:mwobensmith][:matt:]]

Confirmed on Mac, Fx44.0b9.

None of the SafeBrowsing tests under "Download Warnings" (1,2,3) are working either.

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking since this is an important service. Matt, gcp, francois, can you take a look?

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

The first three web page warnings are working correctly for me with 44.0b9 on OS X.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

The "confirmations" in this bug don't really help because you're not specifying whether the feature appears broken for you and in what circumstance or whether you followed the exact same STR.

I'll explain what I mean or what the importance is. From the original description:
>2.Wait 5 minutes for the lists to finish downloading.

This is false. As in, it takes more than 5 minutes for the full lists to finish downloading. We update every 30-45 minutes and we don't know in which chunk the Google test pages are (we know that our own testpages *are* in the first update we force, but those don't test the interaction with Google's server).

I tested and with my Nightly and default profile the test page works.
I made fresh build with logging on an empty profile, and allow the first update through (which appears to work fine): the test page fails.

Based on this, I think the problem is that the data for Google's test page is currently not in the first update they send out. (I think it's actually hard for them to guarantee that)

So, what I would really like to know is: is there anyone with a "live, real" profile (that should have up to date DBs) that is failing on the test pages?

Comment 6

[Gian-Carlo Pascutto [:gcp]]

I took a fresh profile, forced a full database update by fiddling with the prefs and in the end, I got warnings for the first 4 downloads and the unwanted software, but not phishing or malware.

It appears those entries are missing from Google's currently live set.

Comment 7

[François Marier [:francois]]

After taking a fresh profile and waiting 5 minutes, I got only #2 and #3 of the "Download Warnings" section to work. This is expected since we do a remote lookup to the Google server and block the downloads as a result of that.

After forcing a few list updates [1], I got #1 of the "Downloads Warnings" to work. This one is blocked because the hostname appears on a local list so it won't work until the local list contains that entry, like gcp explained in comment 5.

After a lot more forced list updates, I got #2 and #3 of the "WebPage Warnings" to work. Again, this depends on specific entries being present on the local lists, so it's disappointing but not unexpected. I gave up before getting #1 of "WebPage Warnings" to work.

So, as an immediate action, I think we should:

- change the smoketest instructions to only test #2 and #3 of the "Download Warnings"

The other tests might be too flaky to run as a smoketest since we don't have any guarantees as to how long it will take for the entries to be downloaded in the malware/phishing/unwanted lists.

Given that this is a pretty important feature and we otherwise don't have full functional tests for it, I'm going to ask Google how they test this in Chrome and file a follow-up bug to find a way to get this tested before each release.


[1] To force list updates, set browser.safebrowsing.provider.google.nextupdatetime to 1 and then restart Firefox.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

Can we close this as RESOLVED INVALID? Or should we keep it open till there is a reliable user-visible test?

Comment 9

[Tony Mechelynck [:tonymec]]

In this SeaMonkey build which had been running for approx. 4 hours and is supposed to include the same anti-phishing logic as Fx46 I get no recognizable malware warning when opening in a new tab each of the links in the 1st page mentioned in comment #0.

Note that most of these links are for Dos/Windows executables and what I get is the customary "(*) execute via [Wine (default)|▼] or ( ) download to disk" dialog which I get (with a filetype-dependent default app) for any MIME type unknown to L64 SeaMonkey. I hit [Cancel] on these.

UA:"Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0 SeaMonkey/2.43a1"
ID:20160109003001 en-US
c-c:4f4f6a3674c9e16efacf62fd8963c8be5a31b07c
m-c:0f363ae95dc90d593394ef464aa500804c824962

The next Fx46.0a1 nightly is supposed to published soon, I'll install it when I see it then wait, again not just 5 minutes but hours, in order for the caches to stabilize, and see afterwards what happens. I'll post the results here but maybe tomorrow rather than today.

P.S. I'm on openSUSE Leap 42.1 but I get my Mozilla apps from Mozilla, not from my distro.

Comment 10

[Gian-Carlo Pascutto [:gcp]]

(In reply to Tony Mechelynck [:tonymec] from comment #9)
> In this SeaMonkey build which had been running for approx. 4 hours and is
> supposed to include the same anti-phishing logic as Fx46 

The executable warnings are via Application Reputation/Download Protection. I have no idea if SeaMonkey includes this or has it enabled?

Comment 11

[François Marier [:francois]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #8)
> Can we close this as RESOLVED INVALID? Or should we keep it open till there
> is a reliable user-visible test?

I'd say we disable the smoketests like I described in comment 9 and then mark this bug as fixed (and follow-up on the dependent bug).

Comment 12

[Tony Mechelynck [:tonymec]]

(In reply to Tony Mechelynck [:tonymec] from comment #9)
> In this SeaMonkey build which had been running for approx. 4 hours and is
> supposed to include the same anti-phishing logic as Fx46 I get no
> recognizable malware warning when opening in a new tab each of the links in
> the 1st page mentioned in comment #0.
> 
> Note that most of these links are for Dos/Windows executables and what I get
> is the customary "(*) execute via [Wine (default)|▼] or ( ) download to
> disk" dialog which I get (with a filetype-dependent default app) for any
> MIME type unknown to L64 SeaMonkey. I hit [Cancel] on these.
> 
> UA:"Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0
> SeaMonkey/2.43a1"
> ID:20160109003001 en-US
> c-c:4f4f6a3674c9e16efacf62fd8963c8be5a31b07c
> m-c:0f363ae95dc90d593394ef464aa500804c824962
> 
> The next Fx46.0a1 nightly is supposed to published soon, I'll install it
> when I see it then wait, again not just 5 minutes but hours, in order for
> the caches to stabilize, and see afterwards what happens. I'll post the
> results here but maybe tomorrow rather than today.
> 
> P.S. I'm on openSUSE Leap 42.1 but I get my Mozilla apps from Mozilla, not
> from my distro.

In this build of Firefox, after leaving it running but idle for about 14 hours:
UA:"Mozilla/5.0 (X11; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" (en-US) ID:20160116030240 CSet:9879757aa0d4f88df8c79cde4a777ac7eff0152f

I get the following results on "Open link in new tab":
1st Section, links 1 and 2: one-line white on black saying "Requesting <url>" as in SeaMonkey
2nd Section, links 1 to 3: obvious warning pages (dark grey with a dark red block in the middle and the familiar "Mozilla security cop" icon in red; white text; two buttons "Get me out of here!" and "Why was this page blocked?"; at far down right in very small type, "Ignore this warning").
2nd Section, link 4: something which offers itself as an Amazon login page. I didn't try to log in.
3rd Section, link 1: I get an alert: "This is a BIN file, 12 bytes. Would you like to download? [Cancel] [Save File]". I clicked [Cancel]. (this file has extension .com, usually meaning an MSDOS executable fully linked with absolute 16-bit offset-only addressing; on Linux64 where I am such files belong to "another universe").
3rd section, links 2 to 6: I get an alert saying (I quote from memory): "Firefox doesn't know how to open this file" then two radio buttons: (*) Open with [Wine (default)|▼] and ( ) Save to disk; one checkbox: [ ] Do this automatically for files like this from now on; and finally two buttons [Cancel] [OK]. I clicked [Cancel]. (These are .exe files, usually meaning an MSDOS or Windows relocatable executable; I am on Linux where such files can indeed only be run by a Windows emulator package such as Wine.)
4th section, only link: I get a black-on-white page whose first line says "Should send badip pings", followed by an input box and several links; I didn't try.

With the exception of the obvious warnings for Section 2 links 1-3, these results in Firefox after ~14 hours idling are the same as those I got by the same procedures in SeaMonkey after ~4 hours working (mostly, but not exclusively, reading bugmail and the corresponding BMO bug reports) in comment #9.

Reminder: I am on Linux, where Windows executables normally don't apply.

Comment 13

[Tony Mechelynck [:tonymec]]

oops:
1st Section, links 1 and 2: 1-line black on white (etc.)

Comment 14

[Gian-Carlo Pascutto [:gcp]]

(In reply to Tony Mechelynck [:tonymec] from comment #12)
> 3rd Section, link 1: I get an alert: "This is a BIN file, 12 bytes. Would
> you like to download? [Cancel] [Save File]". I clicked [Cancel]. (this file
> has extension .com, usually meaning an MSDOS executable fully linked with
> absolute 16-bit offset-only addressing; on Linux64 where I am such files
> belong to "another universe").
> 3rd section, links 2 to 6: I get an alert saying (I quote from memory):
> "Firefox doesn't know how to open this file" then two radio buttons: (*)
> Open with [Wine (default)|▼] and ( ) Save to disk; one checkbox: [ ] Do this
> automatically for files like this from now on; and finally two buttons
> [Cancel] [OK]. I clicked [Cancel]. (These are .exe files, usually meaning an
> MSDOS or Windows relocatable executable; I am on Linux where such files can
> indeed only be run by a Windows emulator package such as Wine.)

Files are only scanned after they're fully downloaded, if you cancel the download there is no warning, so this is the expected result.

Comment 15

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Not sure what kind of a fix is needed here but it's too late for Fx44.

Comment 16

[François Marier [:francois]]

(In reply to Ritu Kothari (:ritu) from comment #15)
> Not sure what kind of a fix is needed here but it's too late for Fx44.

The fix would be to amend the smoketests like I suggested in comment 7:

- change the smoketest instructions to only test #2 and #3 of the "Download Warnings"

Once that's done, you can mark this issue as fixed I think.

We'll investigate how to do this better in bug 1240195.

Comment 17

[Gian-Carlo Pascutto [:gcp]]

Given that this is a test problem, not an actual browser problem, I've reset the flags.

Comment 18

[Tony Mechelynck [:tonymec]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #17)
> Given that this is a test problem, not an actual browser problem, I've reset
> the flags.

The reason test URLs were given in comment #0, rather than the URLs of actual malware pages, is precisely in order to be able to test the functionality without incurring damage if our malware detection goes awry.

If it takes the browser hours to be aware that a given "test" URL, known to be in the database, is listed as "bad", how can we be sure that _real_ malware will be intercepted? IMHO failing to intercept one of these test pages ought to be regarded as exactly as severe a problem as failing to intercept a real menace of equivalent gravity according to database rankings.

I suppose that we should cache the malware database locally somewhere (maybe in a child of the parent directory of the profiles) from one session to the next, and refresh it at startup (asynchronously if it is still "reasonably fresh", even synchronously before displaying the first tab if it is nonexistent or obsolete) and then asynchronously at some "reasonable" frequency.

If this blocking update is regarded as too severe a performance deficit, then every browser build should come with the current version of the malware database preinstalled, but it should still refresh it periodically while running.

All this might be a possible purpose of bug 1240195 though.

Comment 19

[Gian-Carlo Pascutto [:gcp]]

(In reply to Tony Mechelynck [:tonymec] from comment #18)
> If it takes the browser hours to be aware that a given "test" URL, known to
> be in the database, is listed as "bad", how can we be sure that _real_
> malware will be intercepted?

It won't if it's a fresh installation and the database is still being downloaded. 

> I suppose that we should cache the malware database locally somewhere (maybe
> in a child of the parent directory of the profiles) from one session to the
> next, and refresh it at startup (asynchronously if it is still "reasonably
> fresh", even synchronously before displaying the first tab if it is
> nonexistent or obsolete) and then asynchronously at some "reasonable"
> frequency

We already do this, you're just describing how the system works, minus your suggestion to block startup for a download from a third party server to finish first. 

But a new installation has an empty database, and it takes a while to incrementally download it, due to restrictions we have on hammering the (third party) servers. We also can't predict how fast the users internet is anyway.

> If this blocking update is regarded as too severe a performance deficit,
> then every browser build should come with the current version of the malware
> database preinstalled

This works the wrong way around. The most recently updated part of the database is the most important one, because phishing and malware sites tend to mostly disappear quickly, That's also why it's downloaded first. Bundling a DB would add significant complexity, require repackaging very often, and it would still get you the part of the DB you want *least*.

In reality, Firefox gets the most recent/most important part of the database within 5 minutes after starting up the first time. But this does *not* necessarily include the test URLs.

Test URLs are a bit special because they are static. I suspect (for reasons that have to do with how the update protocol works) that Google rotates them through the database and when this test ended up failing, they were sitting near the very end.

Comment 20

[François Marier [:francois]]

I believe the smoketests are fixed so there's nothing else to do here.

Also Google tells me that in version 4 of the protocol, they will prioritize the test URLs so that they get sent to clients in the first update.