1214372 - [meta] [15Q4] Move firefox-ui-tests in-tree and get Mozmill CI updated to use common.tests.zip

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Our firefox-ui-tests are currently located at https://github.com/mozilla/firefox-ui-tests. We want to move them into the tree to get them executed on a checkin-basis via taskcluster (linux only for now) or Jenkins.

See also bug 1197234 for the firefox-media-tests. Both can happen independently from each other.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

A good read for setting up builds on task cluster may be Ehsan's blog post:
http://ehsanakhgari.org/blog/2015-09-29/my-experience-adding-new-build-type-taskcluster

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Just pasting a comment from bug 1212609 to not forget about version of packages:


> Personally I would like to be in sync with the version of Firefox. That means:
> 
> Firefox 44 (Nightly) -> 44.0.0
> Firefox 43 (Aurora) -> 43.0.0
> 
> When some UI updates happened in Firefox whereby only some get backported to Aurora we could have:
> 
> Firefox 44 (Nightly) -> 44.1.0
> Firefox 43 (Aurora) -> 43.1.0
> 
> For bug fixes we bump the patch level:
> 
> Firefox 44 (Nightly) -> 44.1.1
> Firefox 43 (Aurora) -> 43.1.1
> 
> Every external consumer of puppeteer can then specify a lower and upper bound for dependencies:
> 
> setup.py:
> 
> install_requires=['firefox_puppeteer>=43.0.0, <44.0.0']
> 
> Keep in mind that we could have conflicts for update tests when the update ui is getting changed drastically between versions. But that is not a primary concern and we might be able to workaround it.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

There were lots of other issues to fix this quarter. So beside getting the tests into tree, we have to update our Mozmill CI system to run all the tests via the tests zip file.

I also had several discussions about TC during last weeks Mozlando and it will be a larger task to get done. So this part will move to Q1.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

So I checked the individual test archives and our tests should be added to common.tests.zip given that this package contains Marionette framework and all the related tests.

/marionette/driver/
/marionette/marionette/
/marionette/tests/browser/
[..]
/marionette/tests/testing/
/marionette/transport/

So I would suggest the following as target for our harness, puppeteer, and tests:

/marionette/harnesses/firefox_ui/
/marionette/puppeteer/firefox/
/marionette/tests/firefox_ui/

For the location inside the tree:

/testing/marionette/harnesses/firefox_ui/
/testing/marionette/puppeteer/firefox/
/testing/marionette/tests/firefox_ui/

Whereby the tests will be moved successively from that location to appropriate locations under /browser or /toolkit.

David, would that be ok for now? Or don't you like the location for tests and we should directly fold them into the appropriate folders?

Comment 5

[David Burns :automatedtester]

That seems fine. If we need to move them at a later date we can.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Alright so lets publish packages to PyPI. As following some thoughts in how we execute the tests...

mozilla-central / (mozilla-aurora)
--------------------------------
All Python packages are located in-tree and will be packaged under marionette (as done via https://dxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#532) in common.tests.zip. That zip file will be used by the mozharness script as run by taskcluster or mozmill-ci. Both systems should as best use the packages as shipped with the tests archive for versioning reasons. 

mozilla-beta / mozilla-release / mozilla-esr38
----------------------------------------------
Our packages are initially not in the tree, but could be installed from within the firefox-ui-tests repository checkout. This also allows us to have specific versions for that branch and should be somewhat equal to the tests archive and are easily to swap.

external tests
--------------
Those could use the released versions of firefox_puppeteer, and firefox_ui_harness.

Problems
========
Given that our tests are exclusively in the tree we won't have an easy way anymore to fetch them via a github checkout. For the candidate builds we know the revision id via pulse, but not for releases (which are needed for update tests). As long as our tests are not run on RelEng systems, we might have to query hg.mo for a rev (tag) which represents the release version, pick its parent changeset, and finally get the test package. Due to that reason I would vote to keep the release update tests to run via mozmill. The above should not be a problem for aurora and nightly updates.

We still need firefox-ui-tests as package due to several exports (server root, manifests) which we have to kill soon after the landing in-tree.

We have versioning problems with mozharness. As of now we only use the mozilla-central version for every supported branch. We cannot flip that to branch specific versions because esr38 wont have it, and maybe even beta and release don't have all necessary features. That means we need different steps to be executed based on the branch or availability of the tests in the test archive. I hope that nothing in mozmill ci would have to be changed, beside supplying the test package name.

Tasks
=====
We should release our packages for every branch in case a UI change has happened in Firefox. So for mozilla-central I would use version 3.0.0 for both packages. Beta and release might get 2.0.0 due to UI changes. And for esr38 I will use 1.0.0.

This is kinda complicated. And I hope we don't break that much tests due to the changes.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

The final work on this bug has been done by last week so that every bit is now in mozmill-ci production. We are running via test packages not only for mozilla-central and mozilla-aurora but also for mozilla-beta and mozilla-release (as of 45.0). Also the patch on bug 1239808 got finally landed which will allow us to fetch the correct test packages for the platform to test on.