Closed Bug 967456 Opened 8 years ago Closed 7 years ago

Move all remaining non-local tests from the functional to the remote testrun

Categories

(Mozilla QA Graveyard :: Mozmill Tests, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: whimboo, Unassigned)

References

()

Details

(Whiteboard: [lang=js][good first bug])

Attachments

(1 file, 1 obsolete file)

There are still some tests in the functional testrun which require a remote connection. Those are e.g. the geolocation test and at least some search tests. Before we make them local I would suggest that we move them all to the remote testrun.

Andreea, would you mind mentoring this bug?
I'll take & fix this bug as we need cleaner functional tests now.
Assignee: nobody → daniel.gherasim
Tests I believe should be moved from functional to remote are : 
functional/testGeoLocation/testShareLocation.js - (geolocation service)

functional/testSearch/testAddMozSearchProvider.js - (mozqa.com website)
functional/testSearch/testOpenSearchAutodiscovery.js - (mozqa.com website)
functional/testSearch/testSearchSuggestions.js - (accessing different search providers)
functional/testSearch/testSearchSelection.js - (mozqa.com website)

Anyone who has something to add or comment feel free to do it.
Attached patch bug_967456_fix.patch (obsolete) — Splinter Review
Patch for what I have suggested is up.
Attachment #8392825 - Flags: review?(andrei.eftimie)
Attachment #8392825 - Flags: review?(andreea.matei)
Comment on attachment 8392825 [details] [diff] [review]
bug_967456_fix.patch

Review of attachment 8392825 [details] [diff] [review]:
-----------------------------------------------------------------

This is a good start.

I've run a functional testrun with my network interface down, and besides the tests you already moved these also failed:
> firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension
> firefox/tests/functional/restartTests/testSoftwareUpdateAutoProxy

We should be able to have a green functional testrun without any network connectivity.

::: firefox/tests/remote/testSearch/manifest.ini
@@ +1,2 @@
> +[testAddMozSearchProvider.js]
> +[testOpenSearchAutodiscovery.js]

This test passes without requiring a network connection.
If this is valid we might leave it in the functional testrun.

I do see a 404 page (we are not loading the Search results page)
If the test is faulty and should actually fail here, please file a bug to fix it.
Attachment #8392825 - Flags: review?(andrei.eftimie)
Attachment #8392825 - Flags: review?(andreea.matei)
Attachment #8392825 - Flags: review-
(In reply to Andrei Eftimie from comment #4)
> I've run a functional testrun with my network interface down, and besides
> the tests you already moved these also failed:
> > firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> > firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension

That's interesting. Why do they fail? Those two tests shouldn't need network at all. I wonder if there is still a pref set which causes it. We should do the investigation in a follow-up bug.

> > firefox/tests/functional/restartTests/testSoftwareUpdateAutoProxy

We would have to fake AUS to make it local, so yet its a remote test.
(In reply to Henrik Skupin (:whimboo) from comment #5)
> (In reply to Andrei Eftimie from comment #4)
> > I've run a functional testrun with my network interface down, and besides
> > the tests you already moved these also failed:
> > > firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> > > firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension
> 
> That's interesting. Why do they fail? Those two tests shouldn't need network
> at all. I wonder if there is still a pref set which causes it. We should do
> the investigation in a follow-up bug.

We ping AMO for the blocklist.
(In reply to Andrei Eftimie from comment #6)
> We ping AMO for the blocklist.

Why do you think that? What we do is to change the pref and redirect the request to a local page. Please explain your findings in more detail.
(In reply to Henrik Skupin (:whimboo) from comment #7)
> (In reply to Andrei Eftimie from comment #6)
> > We ping AMO for the blocklist.
> 
> Why do you think that? What we do is to change the pref and redirect the
> request to a local page. Please explain your findings in more detail.

That's interesting.
What I saw was a notification panel telling me (roughly) that Firefox has blocked the install of the addon.

I'll rerun the tests to check this in more detail.
Status: NEW → ASSIGNED
An update here, 

> functional/testSearch/testAddMozSearchProvider.js - (mozqa.com website)
> functional/testSearch/testOpenSearchAutodiscovery.js - (mozqa.com website)
> functional/testSearch/testSearchSelection.js - (mozqa.com website)

This will become completely functional once bug 987594 get's fixed.

> firefox/tests/functional/restartTests/testAddons_installUninstallHardBlocklistedExtension
> firefox/tests/functional/restartTests/testAddons_installUninstallSoftBlocklistedExtension

Filed bug 987612 for this to also become only functional.


So the only tests to be moved now will be:
> functional/restartTests/testSoftwareUpdateAutoProxy
> functional/testGeoLocation/
> functional/testSearch/testSearchSuggestions.js

Uploaded a patch with this.
Attachment #8392825 - Attachment is obsolete: true
Attachment #8396327 - Flags: review?(andrei.eftimie)
Attachment #8396327 - Flags: review?(andreea.matei)
Comment on attachment 8396327 [details] [diff] [review]
bug_967453_fix2.patch

Review of attachment 8396327 [details] [diff] [review]:
-----------------------------------------------------------------

This doesn't apply cleanly anymore since we unskipped the GeoLocation test.

(In reply to daniel.gherasim from comment #9)
> Created attachment 8396327 [details] [diff] [review]
> bug_967453_fix2.patch
> 
> So the only tests to be moved now will be:
> > functional/restartTests/testSoftwareUpdateAutoProxy
> > functional/testGeoLocation/
> > functional/testSearch/testSearchSuggestions.js

I think we should also be able to make testSearchSuggestion a local test.
We're testing Firefox's ability to properly change Search Engines. Using 2 local search engines and switching between them (make them provide static responses) would also make this test run without requiring any network connectivity.
This should also be part of bug 987594
Attachment #8396327 - Flags: review?(andrei.eftimie)
Attachment #8396327 - Flags: review?(andreea.matei)
Attachment #8396327 - Flags: review-
Assignee: daniel.gherasim → nobody
Mentor: andreea.matei
Whiteboard: [mentor=andreeamatei][lang=js][good first bug] → [lang=js][good first bug]
Status: ASSIGNED → NEW
The blocklist tests have been fixed in bug 987612. A few others have also been fixed.

Still affected tests that should be either fixed to work without an internet connection or moved to remote:

> restartTests/testSoftwareUpdateAutoProxy
We'll have to investigate if this can be made a local test, or move it in the remote folder.

> testSearch/testAddMozSearchProvider.js
> testSearch/testSearchSelection.js
> testSearch/testSearchSuggestions.js
These 3 will get fixed by bug 987594.
Depends on: 987612, 987594
We don't want to fix this bug anymore.
Mentor: matei.andreea89
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
Product: Mozilla QA → Mozilla QA Graveyard
You need to log in before you can comment on or make changes to this bug.