967456 - Move all remaining non-local tests from the functional to the remote testrun

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

There are still some tests in the functional testrun which require a remote connection. Those are e.g. the geolocation test and at least some search tests. Before we make them local I would suggest that we move them all to the remote testrun.

Andreea, would you mind mentoring this bug?

Comment 1

[Daniel Gherasim]

I'll take & fix this bug as we need cleaner functional tests now.

Comment 2

[Daniel Gherasim]

Tests I believe should be moved from functional to remote are : 
functional/testGeoLocation/testShareLocation.js - (geolocation service)

functional/testSearch/testAddMozSearchProvider.js - (mozqa.com website)
functional/testSearch/testOpenSearchAutodiscovery.js - (mozqa.com website)
functional/testSearch/testSearchSuggestions.js - (accessing different search providers)
functional/testSearch/testSearchSelection.js - (mozqa.com website)

Anyone who has something to add or comment feel free to do it.

Comment 3

[Daniel Gherasim]

Attachment: bug_967456_fix.patch

Patch for what I have suggested is up.

Comment 4

[Andrei Eftimie]

Comment on attachment 8392825 [details] [diff] [review]
bug_967456_fix.patch

Review of attachment 8392825 [details] [diff] [review]:
-----------------------------------------------------------------

This is a good start.

I've run a functional testrun with my network interface down, and besides the tests you already moved these also failed:
> firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension
> firefox/tests/functional/restartTests/testSoftwareUpdateAutoProxy

We should be able to have a green functional testrun without any network connectivity.

::: firefox/tests/remote/testSearch/manifest.ini
@@ +1,2 @@
> +[testAddMozSearchProvider.js]
> +[testOpenSearchAutodiscovery.js]

This test passes without requiring a network connection.
If this is valid we might leave it in the functional testrun.

I do see a 404 page (we are not loading the Search results page)
If the test is faulty and should actually fail here, please file a bug to fix it.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #4)
> I've run a functional testrun with my network interface down, and besides
> the tests you already moved these also failed:
> > firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> > firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension

That's interesting. Why do they fail? Those two tests shouldn't need network at all. I wonder if there is still a pref set which causes it. We should do the investigation in a follow-up bug.

> > firefox/tests/functional/restartTests/testSoftwareUpdateAutoProxy

We would have to fake AUS to make it local, so yet its a remote test.

Comment 6

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #5)
> (In reply to Andrei Eftimie from comment #4)
> > I've run a functional testrun with my network interface down, and besides
> > the tests you already moved these also failed:
> > > firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> > > firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension
> 
> That's interesting. Why do they fail? Those two tests shouldn't need network
> at all. I wonder if there is still a pref set which causes it. We should do
> the investigation in a follow-up bug.

We ping AMO for the blocklist.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Andrei Eftimie from comment #6)
> We ping AMO for the blocklist.

Why do you think that? What we do is to change the pref and redirect the request to a local page. Please explain your findings in more detail.

Comment 8

[Andrei Eftimie]

(In reply to Henrik Skupin (:whimboo) from comment #7)
> (In reply to Andrei Eftimie from comment #6)
> > We ping AMO for the blocklist.
> 
> Why do you think that? What we do is to change the pref and redirect the
> request to a local page. Please explain your findings in more detail.

That's interesting.
What I saw was a notification panel telling me (roughly) that Firefox has blocked the install of the addon.

I'll rerun the tests to check this in more detail.

Comment 9

[Daniel Gherasim]

Attachment: bug_967453_fix2.patch

An update here, 

> functional/testSearch/testAddMozSearchProvider.js - (mozqa.com website)
> functional/testSearch/testOpenSearchAutodiscovery.js - (mozqa.com website)
> functional/testSearch/testSearchSelection.js - (mozqa.com website)

This will become completely functional once bug 987594 get's fixed.

> firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension

Filed bug 987612 for this to also become only functional.


So the only tests to be moved now will be:
> functional/restartTests/testSoftwareUpdateAutoProxy
> functional/testGeoLocation/
> functional/testSearch/testSearchSuggestions.js

Uploaded a patch with this.

Comment 10

[Andrei Eftimie]

Comment on attachment 8396327 [details] [diff] [review]
bug_967453_fix2.patch

Review of attachment 8396327 [details] [diff] [review]:
-----------------------------------------------------------------

This doesn't apply cleanly anymore since we unskipped the GeoLocation test.

(In reply to daniel.gherasim from comment #9)
> Created attachment 8396327 [details] [diff] [review]
> bug_967453_fix2.patch
> 
> So the only tests to be moved now will be:
> > functional/restartTests/testSoftwareUpdateAutoProxy
> > functional/testGeoLocation/
> > functional/testSearch/testSearchSuggestions.js

I think we should also be able to make testSearchSuggestion a local test.
We're testing Firefox's ability to properly change Search Engines. Using 2 local search engines and switching between them (make them provide static responses) would also make this test run without requiring any network connectivity.
This should also be part of bug 987594

Comment 11

[Andrei Eftimie]

The blocklist tests have been fixed in bug 987612. A few others have also been fixed.

Still affected tests that should be either fixed to work without an internet connection or moved to remote:

> restartTests/testSoftwareUpdateAutoProxy
We'll have to investigate if this can be made a local test, or move it in the remote folder.

> testSearch/testAddMozSearchProvider.js
> testSearch/testSearchSelection.js
> testSearch/testSearchSuggestions.js
These 3 will get fixed by bug 987594.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We don't want to fix this bug anymore.