1240195 - Find a way to test that malware/phishing/unwanted sites are correctly blocked by Safe Browsing

Status: NEW

(Toolkit :: Safe Browsing, defect, P3)

Description

[François Marier [:francois]]

As mentioned in bug 1240027, we don't have any guarantees as to how long it's going to take for the local lists to contain the entries required to make https://testsafebrowsing.appspot.com work correctly.

We need to find a way to test this reliably before each release.

Comment 1

[Tony Mechelynck [:tonymec]]

(In reply to François Marier [:francois] from comment #0)
> As mentioned in bug 1240027, we don't have any guarantees as to how long
> it's going to take for the local lists to contain the entries required to
> make https://testsafebrowsing.appspot.com work correctly.
> 
> We need to find a way to test this reliably before each release.

Do you have an approximate upper bound, even if it's hours? See bug 1240027 comment #9.

Comment 2

[Gian-Carlo Pascutto [:gcp]]

(In reply to Tony Mechelynck [:tonymec] from comment #1)
> Do you have an approximate upper bound, even if it's hours? See bug 1240027
> comment #9.

We don't, that's the point of this bug I think.

Comment 3

[Tony Mechelynck [:tonymec]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #2)
> (In reply to Tony Mechelynck [:tonymec] from comment #1)
> > Do you have an approximate upper bound, even if it's hours? See bug 1240027
> > comment #9.
> 
> We don't, that's the point of this bug I think.

Well, bug 1240027 comment #12 and 13 show that at least three of the tests ran after 14 hours idling. Of course this is probably a grossly exaggerated upper bound. For the other tests I detailed what I did in more detail in bug 1240027 comment #12 than in bug 1240027 comment #9 to let you evaluate if maybe I didn't get far enough. OTOH, on Linux where I am, MSDOS/Windows executables can be regarded as irrelevant. Maybe additional tests are needed for (Linux-type) ELF binaries and for (Mac-type) executables or .dmg Unified Builds.

Comment 4

[François Marier [:francois]]

(In reply to Tony Mechelynck [:tonymec] from comment #3)
> (In reply to Gian-Carlo Pascutto [:gcp] from comment #2)
> > (In reply to Tony Mechelynck [:tonymec] from comment #1)
> > > Do you have an approximate upper bound, even if it's hours? See bug 1240027
> > > comment #9.
> > 
> > We don't, that's the point of this bug I think.
> 
> Well, bug 1240027 comment #12 and 13 show that at least three of the tests
> ran after 14 hours idling. Of course this is probably a grossly exaggerated
> upper bound.

I did the same and left a new Firefox profile running for 24 hours. After that long, all of the Safe Browsing test URLs work.

So while it's probably a reasonably consistent upper bound, it's not very useful for the purpose of automated tests or manual smoketests.

Comment 5

[Tony Mechelynck [:tonymec]]

> So while it's probably a reasonably consistent upper bound, it's not very
> useful for the purpose of automated tests or manual smoketests.

For automated tests, indeed not.
For manual smoketests, maybe a cautionary warning can be put wherever it is detailed which tests to do and how, telling the smoketesters that these safe-browsing tests should be run in a Firefox instance which has _already_ been running for "some high enough" number of hours before the test.

Comment 6

[François Marier [:francois]]

I asked Google about this and they said that this will be much easier when we use version 4 of the protocol because the test entries are always sent first.

For v2.2, they don't have a good answer.