1197162 - Move remote tests into the functional folder and use tags for identification

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Right now we have stored the tests which access remote data in a separate test folder beside the functional tests. Given that in general those type of tests are the same it doesn't make sense to have them located at different locations.

The one thing which is important is that there has to be an option to only let the local tests run. This would be necessary once our tests are in mozilla-central and buildbot/taskcluster is running our tests. Given their requirements no externally hosted data should be accessed.

I briefly talked with Chris and we think that the --tags option should be the right thing to use here. I will have to explore how this will look like.

With that change we would only have to add a single script to mozharness instead of two, and also on Treeherder the results would be cut by 50% which is immense.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

What we can do here and which seems totally sufficient is to add indeed a tag for tests which test data is locally hosted. It will look like that in the manifest file:

> [test_no_certificate.py]
> tags = local
> [test_safe_browsing_notification.py]

Given that we do not care about remote tests only, I would only add tags for local. That way only the local tests can be executed via buildbot/taskcluster. It can be done the following way:

> firefox-ui-tests --binary=path --tag=local

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, I reverted my idea and also added a remote tag to those tests which are getting run remotely. Thing is that with default settings for manifest file we do not really have to add that many lines. The folders which contain more tests with local data, get `local` as default. For remote data the other way around. For individual tests in such a folder which differ I overwrote the tag for the test itself.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: github_pull_request.txt

With this patch we no longer have dedicated remote tests. All tests (beside the update tests) are in the same folder now.

Please keep in mind that the landing of the patch will break mozmill-ci until an appropriate patch for it has been landed too. Until that happened I can temporarily disable remote tests for the landed branches very easily.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We want to get this changed for every supported branch.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The following issue will keep track of the necessary changes in our CI:
https://github.com/mozilla/mozmill-ci/issues/623

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8651027 [details] [review]
github_pull_request.txt

I didn't run this locally, but 2 passes through the changes look good.  I don't see anything of great concern and really like the use of tags.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

https://github.com/mozilla/firefox-ui-tests/commit/480e89239f1a0725157c3fd7038dd645bd01e2ae (master)

When it works well over the weekend I'm going to backport this early next week.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I actually got the backport for aurora done last Friday but missed to add a comment. So here the full list of backports:

https://github.com/mozilla/firefox-ui-tests/commit/1827f1432435f764b4276acab88f8b4069f90627 (aurora)
https://github.com/mozilla/firefox-ui-tests/commit/78fe69596ab933834bd28b4e03001616e20896aa (beta)
https://github.com/mozilla/firefox-ui-tests/commit/34a22af7d8d7c6d89b205e7d4fffb423d8ee6e20 (release)
https://github.com/mozilla/firefox-ui-tests/commit/2357ddeb9c3e894521f62462e78e6b7937d5c28d (esr38)