1258194 - Add Marionette tests for download history across restarts

Status: NEW

(Toolkit :: Downloads API, task)

Description

[:Paolo Amadini]

Testing download history end-to-end across restarts is quite complex because it not only requires a testing framework that supports restarts, but also requires simulating download sources with special characteristics such as failing application reputation checks.

The simplest test would start a download, wait for it to be completed, restart the browser and check that it is present in the download history.

More complex tests would involve, for example, starting a download that failed an application reputation check, restarting the browser, and checking that the download can be unblocked and is then accessible on disk.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hi Paolo, given that you CC'ed me I assume the above should be written as a firefox ui test. I will move the bug into the right component. 

Should we use this bug as tracking bug for each of the test cases you would like to have? In such a case we might be able to find contributors who want to work on each of those. IMHO it's too complex to cover everything in a single bug. 

Regarding download sources, we usually use mozqa.com for our test data. If there is something we can setup please let me know. Otherwise are there other sites already available which we could utilize?

Comment 2

[:Paolo Amadini]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Hi Paolo, given that you CC'ed me I assume the above should be written as a
> firefox ui test. I will move the bug into the right component. 

Thanks!

> Should we use this bug as tracking bug for each of the test cases you would
> like to have? In such a case we might be able to find contributors who want
> to work on each of those. IMHO it's too complex to cover everything in a
> single bug. 

Yes, that's fine.

> Regarding download sources, we usually use mozqa.com for our test data. If
> there is something we can setup please let me know. Otherwise are there
> other sites already available which we could utilize?

I know most of our test suites cannot hit the network, if Firefox UI Tests can reach or have a local copy of mozqa.com then you could certainly set up a few typical files for normal download tests.

Some tests, however, will necessarily require some simulated component. I'm thinking of Application Reputation checking that is a remote service provided by Google and uses a binary format for its requests and responses. If we could call custom testing code in Firefox during UI Tests, we can have it set up a fake mechanism to provide the reputation verdicts.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to :Paolo Amadini from comment #2)
> > Regarding download sources, we usually use mozqa.com for our test data. If
> > there is something we can setup please let me know. Otherwise are there
> > other sites already available which we could utilize?
> 
> I know most of our test suites cannot hit the network, if Firefox UI Tests
> can reach or have a local copy of mozqa.com then you could certainly set up
> a few typical files for normal download tests.

Yes, we can reach network resources whereby we are trying to limit that. Actually we have three levels:

* Local test data served via local web server (wptserve)
* Remote test data served by mozqa.com
* Remote test data by other websites (very fragile and if possible we try to use the first two ones by creating a test case for this situation)

> Some tests, however, will necessarily require some simulated component. I'm
> thinking of Application Reputation checking that is a remote service
> provided by Google and uses a binary format for its requests and responses.
> If we could call custom testing code in Firefox during UI Tests, we can have
> it set up a fake mechanism to provide the reputation verdicts.

I think we should not do that but instead use the real service. We had a similar case with a regression in safe browsing which hasn't been seen because we weren't testing the Google backend. CC'ing Francois here.

I wonder how much we overlap with the test creation bugs already filed as listed on bug 1250329. Maybe we can combine some steps so we only have to run those tests once.

Comment 4

[François Marier [:francois]]

(In reply to Henrik Skupin (:whimboo) from comment #3)
> (In reply to :Paolo Amadini from comment #2)
> > Some tests, however, will necessarily require some simulated component. I'm
> > thinking of Application Reputation checking that is a remote service
> > provided by Google and uses a binary format for its requests and responses.
> > If we could call custom testing code in Firefox during UI Tests, we can have
> > it set up a fake mechanism to provide the reputation verdicts.
> 
> I think we should not do that but instead use the real service. We had a
> similar case with a regression in safe browsing which hasn't been seen
> because we weren't testing the Google backend. CC'ing Francois here.
> 
> I wonder how much we overlap with the test creation bugs already filed as
> listed on bug 1250329. Maybe we can combine some steps so we only have to
> run those tests once.

It seems to me like there are two different things to test: the code that handles and blocks downloads as appropriate, and the code that talks to the Google server.

I think there's value in having a fake mechanism to exercise all parts of the download manager that handle the different verdicts. That could be a very reliable (and simple) way to catch any regressions early in the process.

Then bug 1237766 can be limited to testing one malware download to ensure that we can reach Google's servers and effectively block one kind of malware.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

That sounds great to me. Given that we would need a mock here, it may be better to start with bug 1237766 first.

Comment 6

[:Paolo Amadini]

(In reply to Henrik Skupin (:whimboo) from comment #5)
> That sounds great to me. Given that we would need a mock here, it may be
> better to start with bug 1237766 first.

It's a good start. It does not talk about persistence of the ability to unblock or permanently delete those downloads across restarts, which is one more step.

It's also orthogonal to testing the persistence of succeeded downloads across restarts.

As you mentioned, this may be treated as a meta-bug, but I'm not sure if bug 1237766 qualifies as a dependency since it's not about restarts.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to :Paolo Amadini from comment #6)
> As you mentioned, this may be treated as a meta-bug, but I'm not sure if bug
> 1237766 qualifies as a dependency since it's not about restarts.

We will have to extend our page object model (aka Firefox Puppeteer) to support all the new ui elements we have to work with. It would also be necessary for the other bug - maybe not all ui elements as we would need here).

So best would be if you could file new bugs for each of the wanted tests here, so I can see what we need and could file a puppeteer related bug.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Paolo, would this still be something good to have?

Comment 9

[:Paolo Amadini]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #8)

Paolo, would this still be something good to have?

Yes, because automated test coverage across restarts is still missing in this area, as far as I know. Regressions in the least common cases are the most likely to be unnoticed, for example the persistence of the state of blocked downloads.

As noted in comment 0, though, test cases may still be difficult to implement.