1232967 - Create Python packages for firefox_ui_harness, firefox_puppeteer, and firefox-ui-tests

Status: RESOLVED FIXED

(Testing :: Firefox UI Tests, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For the move of our firefox ui harness and puppeteer into the tree we need individual Python packages for both.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR

Lots of changes... Let me know what you think! Some of the files and package references we have to keep for now and will be removed once we copy that all into the tree.

Sadly Travis doesn't want to run it. So I have to check again tomorrow morning.

Comment 2

[Syd Polk :sydpolk]

As long as this is tested thoroughly, this is all structural. I see no red flags. I did leave one question about relative imports, but as long as it works, this LGTM.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8699224 [details] [review]
Github PR

Same here. Just a few small suggestions/questions.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I pushed the changes to a new branch on the official repository which will allow us to update the code in case changes are getting landed on mozilla-central.

https://github.com/mozilla/firefox-ui-tests/tree/python_packages

Personally I would like to get this code landed on mozilla-central and mozilla-aurora. If we do so we have to update the mozharness code to not run pip but setup_development.py. To ensure that it will also work for older branches which do not have those packages a modified version of setup_development.py should be landed.

Maja and Syd, would this cause any problem for you with the media tests? I know its not ideal right now but we would need that anyway once our tests are in tree and we have to run the firefox-ui-tests in Mozmill CI.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The other option would be that we release new packages for firefox_ui_harness and firefox_puppeteer to pypi. Then setup.py of firefox-ui-tests would still work. Given that we would need that anyway it sounds like the better approach.

Comment 6

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

media-tests only rely on the firefox_puppeteer package. Syd has the most up-to-date knowledge of how media-tests will run in the near future, so I think he's in a better position to assess the impact of this change.

I like the pypi option. If you publish to pypi, I think that might also help John Dorlus in bug 1226638.

Comment 7

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

That is, media-tests only rely on firefox_puppeteer now because that is where FirefoxTestCase is defined, but as I understand it FirefoxTestCase may eventually move into the harness package.

Comment 8

[Syd Polk :sydpolk]

We will have to change the requirements.txt file in firefox_media_tests to include the firefox_puppeteer package and remove the firefox-ui-tests submodule in the short-medium term, although it's not terribly urgent since the firefox-ui-tests github repo won't disappear anytime soon.

I don't see any of the mozharness scripts calling pip, so as long as that package is available, I think firefox-media-tests will be good to go. Don't know the implications of having media-tests in the tree are.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Alright so lets publish packages to PyPI. As following some thoughts in how we execute the tests...

mozilla-central / (mozilla-aurora)
--------------------------------
All Python packages are located in-tree and will be packaged under marionette (as done via https://dxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#532) in common.tests.zip. That zip file will be used by the mozharness script as run by taskcluster or mozmill-ci. Both systems should as best use the packages as shipped with the tests archive for versioning reasons. 

mozilla-beta / mozilla-release / mozilla-esr38
----------------------------------------------
Our packages are initially not in the tree, but could be installed from within the firefox-ui-tests repository checkout. This also allows us to have specific versions for that branch and should be somewhat equal to the tests archive and are easily to swap.

external tests
--------------
Those could use the released versions of firefox_puppeteer, and firefox_ui_harness.

Problems
========
Given that our tests are exclusively in the tree we won't have an easy way anymore to fetch them via a github checkout. For the candidate builds we know the revision id via pulse, but not for releases (which are needed for update tests). As long as our tests are not run on RelEng systems, we might have to query hg.mo for a rev (tag) which represents the release version, pick its parent changeset, and finally get the test package. Due to that reason I would vote to keep the release update tests to run via mozmill. The above should not be a problem for aurora and nightly updates.

We still need firefox-ui-tests as package due to several exports (server root, manifests) which we have to kill soon after the landing in-tree.

We have versioning problems with mozharness. As of now we only use the mozilla-central version for every supported branch. We cannot flip that to branch specific versions because esr38 wont have it, and maybe even beta and release don't have all necessary features. That means we need different steps to be executed based on the branch or availability of the tests in the test archive. I hope that nothing in mozmill ci would have to be changed, beside supplying the test package name.

Tasks
=====
We should release our packages for every branch in case a UI change has happened in Firefox. So for mozilla-central I would use version 3.0.0 for both packages. Beta and release might get 2.0.0 due to UI changes. And for esr38 I will use 1.0.0.

This is kinda complicated. And I hope we don't break that much tests due to the changes.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The last comment should actually have been submitted to the tracking bug.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (version bump for Nightly)

r+ from David via IRC.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8699634 [details] [review]
Github PR (version bump for Nightly)

https://github.com/mozilla/firefox-ui-tests/commit/a3832d51e830ce316c868adccb0eb7373c216865 (python-package branch)

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Packages for Nightly are up on PyPI:

https://pypi.python.org/packages/source/f/firefox_puppeteer/firefox_puppeteer-3.0.0.tar.gz
https://pypi.python.org/packages/source/f/firefox_ui_harness/firefox_ui_harness-1.0.0.tar.gz

I will continue to check for other puppeteer versions needed for older branches tomorrow.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (requirements.txt and travis updates)

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8700631 [details] [review]
Github PR (requirements.txt and travis updates)

Merged to mozilla-central:
https://github.com/mozilla/firefox-ui-tests/commit/9eaa396c45081d3cd3c60615170e0d89f32c2b19

I will backport it to all other branches tomorrow, given that this would stop us from having to backport the packaging PR down to esr38, which would be a hard work.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, looks like I missed one more thing... When I run mozharness with the fx ui packages I get a failure:

> 18:20:01     INFO -  /mozilla/code/gecko/testing/mozharness/build/venv/bin/python: can't open file '/mozilla/code/gecko/testing/mozharness/build/firefox_ui_tests/firefox_ui_harness/runtests.py': [Errno 2] No such file or directory

That reminds me that we do not use the CLI command to trigger the functional tests but the runtests.py script directly to not trigger the UAC prompt. The same applies to the update script.

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I filed bug 1234345 for the mozharness case to get a cli_functional.py script added. It should mostly fix the remaining issue.

Also I checked current results from a mozilla-central test for the requirements.txt and Travis changes. All looked fine. So I went ahead and backported the patch:

https://github.com/mozilla/firefox-ui-tests/commit/08ca5853f7c4e515b6abbf70b906c29cbbaf0871 (aurora)
https://github.com/mozilla/firefox-ui-tests/commit/a634ff461cd02fb795b560807c1a60d300f07266 (beta)
https://github.com/mozilla/firefox-ui-tests/commit/cb6e9ecd421c201e00cc9dc2bdafca4f222d4181 (release)

ESR38 has merge conflicts. So I have to get them solved first. I might have done this patch on a different bug. :/

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (correct packaging & backward compatibility)

Comment 19

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8700796 [details] [review]
Github PR (correct packaging & backward compatibility)

r+ lgtm, assuming travis build passes

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Landed PR as two distinct commits to have the version bump of firefox_ui_harness separate:

https://github.com/mozilla/firefox-ui-tests/commit/dfd2ed3f0c09a0bd3f88c72edb651d9e6d2c71d7
https://github.com/mozilla/firefox-ui-tests/commit/e0b40eca477b53de070418ca1cc834c937b0e779

I will finish the backport tomorrow. Also once the necessary mozharness changes went in, we can get the python_package branch merged to m-c.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Btw. firefox_ui_harness 1.1.0 has been released:

Submitting dist/firefox_ui_harness-1.1.0.tar.gz to https://pypi.python.org/pypi
Server response (200): OK

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (requirements.txt and travis updates - esr38 backport)

This is the backport for esr38. It applies cleanly and Travis doesn't have any problems in running the tests. There is a single test failure only which caused one of the builds to fail.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8701026 [details] [review]
Github PR (requirements.txt and travis updates - esr38 backport)

Merged PR to mozilla-esr38 as:

https://github.com/mozilla/firefox-ui-tests/commit/21732847913fd936ce5afe3af5d8ace007349dc4

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Remaining tasks to do here:

* Merge the python_packages branch to mozilla-central (github) once the mozharness patch on bug  	1212609 got landed
* Backport the packaging work to mozilla-aurora

Comment 25

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Given that I don't want to wait longer for getting all mozharness changes landed to support in-tree tests (which we plainly don't need for older branches) I just pushed an update oh mozharness to one of the latest mozilla-central revisions. It will include the required fix and also some other enhancements.

https://github.com/mozilla/mozmill-ci/issues/732

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I was waiting for some Aurora builds to appear for production and all looks fine with our tests:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-aurora&revision=20d95004d8d1&filter-searchStr=firefox%20ui&filter-tier=1&filter-tier=2&filter-tier=3

So I'm going to merge the python_packages branch into our mozilla-central branch on github now.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I squashed the python_packages branch and pushed it to mozilla-central:

https://github.com/mozilla/firefox-ui-tests/compare/76ffb49bee61...66d1de81d299

I will closely watch test results today for Nightly builds on production. In parallel I will start to create the packages for the mozilla-aurora branch.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The aurora-backport looks to be trivial. The cherry-pick applied both necessary changesets from this bug and bug 1237322 cleanly. The only thing we have to take care of is that the puppeteer version for mozilla-aurora cannot have bug 1237329 included. Which also means we need a separate release for puppeteer. Given that 3.1.0 has been released for Firefox 46.0, we can make use of 2.0.0 for Firefox 45.0.

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Github PR (backport of packages to mozilla-aurora)

Those backport changes are close to the code we have on mozilla-aurora. It only varies in branch names, versions of packages and mozilla-central only changes.

Comment 30

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8709059 [details] [review]
Github PR (backport of packages to mozilla-aurora)

Both Syd and Maja are not available today. So also adding David, who could review it later. David, if you think that Syd and Maja should have a look at the patch, please note it when doing the review.  Thanks.

Comment 31

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8709059 [details] [review]
Github PR (backport of packages to mozilla-aurora)

This looks not broken, but honestly it's hard to tell whether anything could have been missed.

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: interdiff (aurora patch to mozilla-central)

Maja asked for some clarity that nothing is missing.  So here the interdiff between the current packaging branch and mozilla-central.

Comment 33

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Before we can merge this patch to mozilla-aurora we have to get the puppeteer release uploaded to pypi and also mirrored to the internal server.

The release of firefox-puppeteer 2.0.0 happened here:

Submitting dist/firefox-puppeteer-2.0.0.tar.gz to https://pypi.python.org/pypi
Server response (200): OK

Comment 34

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I merged the python packages branch into mozilla-aurora now:
https://github.com/mozilla/firefox-ui-tests/commit/2acb1aa84758141fc438ad07c86999ab0ec0e2fe

So nothing else has to be done on the github repository. The remaining patch to add those packages to hg.mo I will attach to bug 1214372.