Right now we have stored the tests which access remote data in a separate test folder beside the functional tests. Given that in general those type of tests are the same it doesn't make sense to have them located at different locations. The one thing which is important is that there has to be an option to only let the local tests run. This would be necessary once our tests are in mozilla-central and buildbot/taskcluster is running our tests. Given their requirements no externally hosted data should be accessed. I briefly talked with Chris and we think that the --tags option should be the right thing to use here. I will have to explore how this will look like. With that change we would only have to add a single script to mozharness instead of two, and also on Treeherder the results would be cut by 50% which is immense.
What we can do here and which seems totally sufficient is to add indeed a tag for tests which test data is locally hosted. It will look like that in the manifest file: > [test_no_certificate.py] > tags = local > [test_safe_browsing_notification.py] Given that we do not care about remote tests only, I would only add tags for local. That way only the local tests can be executed via buildbot/taskcluster. It can be done the following way: > firefox-ui-tests --binary=path --tag=local
Ok, I reverted my idea and also added a remote tag to those tests which are getting run remotely. Thing is that with default settings for manifest file we do not really have to add that many lines. The folders which contain more tests with local data, get `local` as default. For remote data the other way around. For individual tests in such a folder which differ I overwrote the tag for the test itself.
Created attachment 8651027 [details] [review] github_pull_request.txt With this patch we no longer have dedicated remote tests. All tests (beside the update tests) are in the same folder now. Please keep in mind that the landing of the patch will break mozmill-ci until an appropriate patch for it has been landed too. Until that happened I can temporarily disable remote tests for the landed branches very easily.
Attachment #8651027 - Flags: review?(jmaher)
We want to get this changed for every supported branch.
status-firefox40: --- → affected
status-firefox41: --- → affected
status-firefox42: --- → affected
status-firefox-esr38: --- → affected
Summary: Move remote tests into the functional folder and use tags to for identification → Move remote tests into the functional folder and use tags for identification
The following issue will keep track of the necessary changes in our CI: https://github.com/mozilla/mozmill-ci/issues/623
Comment on attachment 8651027 [details] [review] github_pull_request.txt I didn't run this locally, but 2 passes through the changes look good. I don't see anything of great concern and really like the use of tags.
Attachment #8651027 - Flags: review?(jmaher) → review+
https://github.com/mozilla/firefox-ui-tests/commit/480e89239f1a0725157c3fd7038dd645bd01e2ae (master) When it works well over the weekend I'm going to backport this early next week.
status-firefox43: affected → fixed
Target Milestone: --- → Firefox 43
I actually got the backport for aurora done last Friday but missed to add a comment. So here the full list of backports: https://github.com/mozilla/firefox-ui-tests/commit/1827f1432435f764b4276acab88f8b4069f90627 (aurora) https://github.com/mozilla/firefox-ui-tests/commit/78fe69596ab933834bd28b4e03001616e20896aa (beta) https://github.com/mozilla/firefox-ui-tests/commit/34a22af7d8d7c6d89b205e7d4fffb423d8ee6e20 (release) https://github.com/mozilla/firefox-ui-tests/commit/2357ddeb9c3e894521f62462e78e6b7937d5c28d (esr38)
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
status-firefox40: affected → fixed
status-firefox41: affected → fixed
status-firefox42: affected → fixed
status-firefox-esr38: affected → fixed
You need to log in before you can comment on or make changes to this bug.