Write a Marionette test that downloads malware and verifies that it's blocked.

NEW
Unassigned

Status

Testing
Firefox UI Tests
2 years ago
11 months ago

People

(Reporter: francois, Unassigned)

Tracking

(Blocks: 1 bug)

43 Branch
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
We should download the three files under "Download Warnings":

  https://testsafebrowsing.appspot.com/

and check that Firefox blocks them.
(Reporter)

Updated

2 years ago
Component: Safe Browsing → Firefox UI Tests
Product: Toolkit → Testing
QA Contact: hskupin
(Reporter)

Comment 1

2 years ago
We probably want to fix bug 1237396 first since download protection will fail if the Safe Browsing lists are not available.
Depends on: 1237396
How reliable is this external website? For remote content we try to limit our access to internal servers. Means we have http://www.mozqa.com for this purpose. This is just to reduce test failures due to network latency.
(Reporter)

Comment 3

2 years ago
(In reply to Henrik Skupin (:whimboo) from comment #2)
> How reliable is this external website? For remote content we try to limit
> our access to internal servers. Means we have http://www.mozqa.com for this
> purpose. This is just to reduce test failures due to network latency.

It's run by the Google Safe Browsing team and I haven't seen it down yet. So probably reliable enough for us to use in our tests.

If it doesn't work, then we'll have to host our own fake "malware" file and then ask Google if they could flag it as bad on their end.
Assignee: nobody → mwobensmith
(In reply to François Marier [:francois] from comment #0)
> We should download the three files under "Download Warnings":
> 
>   https://testsafebrowsing.appspot.com/
> 
> and check that Firefox blocks them.

So even with the new test to check the initial safe browsing files have been downloaded, I wonder how we should organize the tests. I don't think it makes sense to have different test files, which would all operate on a new profile and have to download the files again. What I would suggest is the following:

* One single test file for safebrowsing
* A setup class method which is run once and actually checks for the files being present
* A setup test method to ensure we prepare the tests
* A new test for this particular issue (we can add more tests later)
* A teardown test method to clean up test artifacts
* A teardown class method for general clean-up and to restart with a fresh profile

Francois, does it sound fine with you? Matt, are you still interested to create this test?
Flags: needinfo?(mwobensmith)
Flags: needinfo?(francois)
Henrik, should the test from bug 1237396 be included in this somewhere? I think that any test of safe browsing would begin after verifying that the tables have been downloaded first.

I am committed to writing Marionette test(s) for both safe browsing (this) and tracking protection. Since they both rely on downloading these tables, they are related. We may or may not want to put them together.
Flags: needinfo?(mwobensmith) → needinfo?(hskupin)
(Reporter)

Comment 6

2 years ago
(In reply to Henrik Skupin (:whimboo) from comment #4)
> * One single test file for safebrowsing
> * A setup class method which is run once and actually checks for the files
> being present
> * A setup test method to ensure we prepare the tests
> * A new test for this particular issue (we can add more tests later)
> * A teardown test method to clean up test artifacts
> * A teardown class method for general clean-up and to restart with a fresh
> profile
> 
> Francois, does it sound fine with you? Matt, are you still interested to
> create this test?

That sounds good to me.
Flags: needinfo?(francois)
(In reply to Matt Wobensmith [:mwobensmith][:matt:] from comment #5)
> Henrik, should the test from bug 1237396 be included in this somewhere? I
> think that any test of safe browsing would begin after verifying that the
> tables have been downloaded first.

As mentioned in my comment 4 and what Francois agreed on now is that we should get rid of the specific test from bug 1237396 and put the code in the setupClass method. It's the primary requirement for all safebrowsing tests and by doing that we can ensure that we have to wait only once. In regards of that I checked a couple of instances of that test being run on Taskcluster. The timing varies a bit but in general it looks like that the test is taking ~20s to run. So I really would like to see that we have to wait only once for any other safebrowsing test which depends on those downloaded files. 

> I am committed to writing Marionette test(s) for both safe browsing (this)
> and tracking protection. Since they both rely on downloading these tables,
> they are related. We may or may not want to put them together.

That's great! Let me know when you have questions and other blocking items. I'm happy to assist you through the creation of those tests.
Flags: needinfo?(hskupin)
(Reporter)

Comment 8

a year ago
I assume that Matt doesn't mind if someone else takes care of this.
Assignee: mwobensmith → nobody
(Reporter)

Updated

11 months ago
Blocks: 1376036
(Reporter)

Comment 9

11 months ago
We can use the official test file (https://testsafebrowsing.appspot.com/s/content.exe) as long as it's not hosted on a mozilla.org or mozilla.net domain.
You need to log in before you can comment on or make changes to this bug.