849962 - Marker bug to move functional tests with remote testcases to remote testrun

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Tests, defect)

Description

[Jonathan French (:jfrench)]

This is a marker bug on Henrik's behalf, to move all mozmill test files related to remote(external url) content to the remote testrun, so we can do per testrun patches. I'll let Henrik provide more information as needed.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I think all has been said here. Those functional tests which need remote testcase data should be moved into the remote testrun.

Comment 2

[Jonathan French (:jfrench)]

Henrik also said it nicely elsewhere in a separate thread, so paraphrasing his quote here.

The functional testrun including tests which reference external urls, should be not depend on network access, and should be able to do everything via localhost. Or via the upcoming changes, we can flip between localhost, or other network servers.

Comment 3

[Rahul Gandhi]

Sir
I want to handle this bug,it will be great opportunity for me to work on this bug.i am well equipped with the knowledge of c/c++, java, javascript, html/css.

Comment 4

[Andreea Matei [:AndreeaMatei]]

Hi Rahul, thanks for your interest!

Let me know if you have some more questions in getting started. 
Here's how you can install Mozmill:
https://developer.mozilla.org/en-US/docs/Mozmill

Please work with the latest nightly downloaded from here [1] and always start with a clean repository (we change it mostly every day with new pushes).

[1] ftp://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central/

Helpful style guide:
https://developer.mozilla.org/en-US/docs/Mozmill_Tests/Mozmill_Style_Guide

I'm also available on IRC if you need help.

Comment 5

[Samvedana (:Samvedana)]

Hi Andreea,

Can you please provide more detail to start working on this bug?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Samvedana, if you want to get started on this bug you might want to find each test under tests/functional which does not make use of 'collector.addHttpResource()' but references an external URL.

Comment 7

[Balazs Juhasz (:balazs)]

Attachment: move functional tests using external URL to remote test group (default) rev1.0

After analysis, I found that the following tests use currently external URL:

functional/testDownloading/testDownloadStates.js
functional/testLayout/testNavigateFTP.js
functional/testPreferences/testRemoveAllCookies.js
functional/testPrivateBrowsing/testAboutCache.js
functional/testPrivateBrowsing/testFlashCookie.js
functional/testSecurity/testDVCertificate.js
functional/testSecurity/testGreenLarry.js
functional/testSecurity/testMixedContentPage.js
functional/testSecurity/testSafeBrowsingNotificationBar.js
functional/testSecurity/testSafeBrowsingWarningPages.js
functional/testSecurity/testSecurityInfoViaMoreInformation.js
functional/testSecurity/testSecurityNotification.js
functional/testSecurity/testSSLDisabledErrorPage.js
functional/testSecurity/testSubmitUnencryptedInfoWarning.js
functional/testSecurity/testUnknownIssuer.js
functional/testSecurity/testUntrustedConnectionErrorPage.js
functional/testToolbar/testStopReloadButtons.js

functional/restartTests/testAddons_installFromFTP/test1.js
functional/restartTests/testAddons_RestartlessExtensionWorksAfterRestart/test2.js

These tests are moved to remote. In case of the 2 restart test, the whole directories moved to remote.

Functional testrun:
http://mozmill-crowd.blargon7.com/#/functional/report/26d5854914a68aa13b333f2815b1ce0c
http://mozmill-crowd.blargon7.com/#/functional/report/26d5854914a68aa13b333f2815b23556

Remote testrun:
http://mozmill-crowd.blargon7.com/#/remote/report/26d5854914a68aa13b333f2815b27cf5
http://mozmill-crowd.blargon7.com/#/remote/report/26d5854914a68aa13b333f2815b29668

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 803282 [details] [diff] [review]
move functional tests using external URL to remote test group (default) rev1.0

Review of attachment 803282 [details] [diff] [review]:
-----------------------------------------------------------------

Balazs, great investigation. I did a quick check and noticed that you missed to actually move the tests. Did you miss a last qref?

Comment 9

[Balazs Juhasz (:balazs)]

Comment on attachment 803282 [details] [diff] [review]
move functional tests using external URL to remote test group (default) rev1.0

The moving of files is only visible inside the patch file. In Diff view it is not visible.

Comment 10

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 803282 [details] [diff] [review]
move functional tests using external URL to remote test group (default) rev1.0

Review of attachment 803282 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for your work Balazs, pretty awesome to finally see those tests moved!

We would need another test moved though, I see in functional/restartTests/testDefaultBookmark.js is using:
const TEST_DATA = "https://www.mozilla.org/" +
                  localization.normalizeLocale() +
                  "/firefox/central/"
so it should end up in the remote side.

Other than that, everything seems right! :)

::: tests/remote/restartTests/manifest.ini
@@ -1,2 @@
> -[include:testDummy/manifest.ini]
> -# Dummy test necessary since all remote tests are disabled

Not sure why this has been removed.

Comment 11

[Balazs Juhasz (:balazs)]

> We would need another test moved though, I see in
> functional/restartTests/testDefaultBookmark.js is using:
> const TEST_DATA = "https://www.mozilla.org/" +
>                   localization.normalizeLocale() +
>                   "/firefox/central/"
> so it should end up in the remote side.

I saw this reference also, but as I understand the test it only checks if the bookmar is exist and it's link is pointing to this TEST_DATA.It not opens the TEST_DATA location, so no remote data is needed. I decided to leave it in functional tests, but if it is not correct, I will move it to remote.

> ::: tests/remote/restartTests/manifest.ini
> @@ -1,2 @@
> > -[include:testDummy/manifest.ini]
> > -# Dummy test necessary since all remote tests are disabled
> 
> Not sure why this has been removed.

I removed this test from manifest file based on the comment. In this patch some tests moved from functional/restartTests to remote/restartTests. testAddons_installFromFTP test is enabled so not every tests are disabled anymore.

Comment 12

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 803282 [details] [diff] [review]
move functional tests using external URL to remote test group (default) rev1.0

Hm, indeed, you're correct. Good catch! I'll land this today.

Comment 13

[Andreea Matei [:AndreeaMatei]]

Landed:
http://hg.mozilla.org/qa/mozmill-tests/rev/5bfec73621db (default)

We will want this backported to other branches and an update will be needed cause I tried to apply it on aurora and it didn't work.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

23 does no longer exists but ensure to land it on both beta and release for 24.

Comment 15

[Balazs Juhasz (:balazs)]

Attachment: backported patch (mozilla-aurora) rev1.0

Backported patch for mozilla-aurora branch.

Functional testruns:
http://mozmill-crowd.blargon7.com/#/functional/report/dc3d4d2f66af31b9e75e291fa0370c2d
http://mozmill-crowd.blargon7.com/#/functional/report/dc3d4d2f66af31b9e75e291fa0370d81

Remote testruns:
http://mozmill-crowd.blargon7.com/#/remote/report/dc3d4d2f66af31b9e75e291fa0371c63
http://mozmill-crowd.blargon7.com/#/remote/report/dc3d4d2f66af31b9e75e291fa037281e

The patch file also works for the following branches:
- mozilla-beta
- mozilla-release
- mozilla-esr24

For mozilla-esr17 I will create a separate backport.

Comment 16

[Balazs Juhasz (:balazs)]

Attachment: backported patch (mozilla-esr17) rev1.0

Functional testruns:
http://mozmill-crowd.blargon7.com/#/functional/report/dc3d4d2f66af31b9e75e291fa0372fbe
http://mozmill-crowd.blargon7.com/#/functional/report/dc3d4d2f66af31b9e75e291fa0373b99

Remote testruns:
http://mozmill-crowd.blargon7.com/#/remote/report/dc3d4d2f66af31b9e75e291fa0374002
http://mozmill-crowd.blargon7.com/#/remote/report/dc3d4d2f66af31b9e75e291fa037454c

Comment 17

[Andreea Matei [:AndreeaMatei]]

Comment on attachment 804095 [details] [diff] [review]
backported patch (mozilla-esr17) rev1.0

Review of attachment 804095 [details] [diff] [review]:
-----------------------------------------------------------------

Backported:
http://hg.mozilla.org/qa/mozmill-tests/rev/63a310d9c807 (aurora)
http://hg.mozilla.org/qa/mozmill-tests/rev/e1238762e47d (beta)
http://hg.mozilla.org/qa/mozmill-tests/rev/fee46eaf2041 (release)
http://hg.mozilla.org/qa/mozmill-tests/rev/027faaef9a4b (esr24)
http://hg.mozilla.org/qa/mozmill-tests/rev/5ed3836767c1 (esr17)

Yey, we have more remote tests! Thanks Balasz, awesome job. Feel free to contact me if you want to work on other bugs as well.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's not only that this is fixed, but also that we do no longer have remote (external) test data in that testrun anymore! That's a great step forward, which we were waiting for really long time. Thanks a lot Balazs!