Create Python packages for firefox_ui_harness, firefox_puppeteer, and firefox-ui-tests

RESOLVED FIXED in Firefox 45


Firefox UI Tests
2 years ago
2 years ago


(Reporter: whimboo, Assigned: whimboo)


45 Branch
Dependency tree / graph

Firefox Tracking Flags

(firefox45 fixed, firefox46 fixed)




(7 attachments)

For the move of our firefox ui harness and puppeteer into the tree we need individual Python packages for both.
Created attachment 8699224 [details] [review]
Github PR

Lots of changes... Let me know what you think! Some of the files and package references we have to keep for now and will be removed once we copy that all into the tree.

Sadly Travis doesn't want to run it. So I have to check again tomorrow morning.
Attachment #8699224 - Flags: review?(spolk)
Attachment #8699224 - Flags: review?(mjzffr)

Comment 2

2 years ago
As long as this is tested thoroughly, this is all structural. I see no red flags. I did leave one question about relative imports, but as long as it works, this LGTM.


2 years ago
Attachment #8699224 - Flags: review?(spolk) → review+
Comment on attachment 8699224 [details] [review]
Github PR

Same here. Just a few small suggestions/questions.
Attachment #8699224 - Flags: review?(mjzffr) → review+
Summary: Create Python packages for firefox_ui_harness and firefox_puppeteer → Create Python packages for firefox_ui_harness, firefox_puppeteer, and firefox-ui-tests
I pushed the changes to a new branch on the official repository which will allow us to update the code in case changes are getting landed on mozilla-central.

Personally I would like to get this code landed on mozilla-central and mozilla-aurora. If we do so we have to update the mozharness code to not run pip but To ensure that it will also work for older branches which do not have those packages a modified version of should be landed.

Maja and Syd, would this cause any problem for you with the media tests? I know its not ideal right now but we would need that anyway once our tests are in tree and we have to run the firefox-ui-tests in Mozmill CI.
Flags: needinfo?(spolk)
Flags: needinfo?(mjzffr)
The other option would be that we release new packages for firefox_ui_harness and firefox_puppeteer to pypi. Then of firefox-ui-tests would still work. Given that we would need that anyway it sounds like the better approach.
media-tests only rely on the firefox_puppeteer package. Syd has the most up-to-date knowledge of how media-tests will run in the near future, so I think he's in a better position to assess the impact of this change.

I like the pypi option. If you publish to pypi, I think that might also help John Dorlus in bug 1226638.
Flags: needinfo?(mjzffr)
That is, media-tests only rely on firefox_puppeteer now because that is where FirefoxTestCase is defined, but as I understand it FirefoxTestCase may eventually move into the harness package.

Comment 8

2 years ago
We will have to change the requirements.txt file in firefox_media_tests to include the firefox_puppeteer package and remove the firefox-ui-tests submodule in the short-medium term, although it's not terribly urgent since the firefox-ui-tests github repo won't disappear anytime soon.

I don't see any of the mozharness scripts calling pip, so as long as that package is available, I think firefox-media-tests will be good to go. Don't know the implications of having media-tests in the tree are.
Flags: needinfo?(spolk)
Alright so lets publish packages to PyPI. As following some thoughts in how we execute the tests...

mozilla-central / (mozilla-aurora)
All Python packages are located in-tree and will be packaged under marionette (as done via in That zip file will be used by the mozharness script as run by taskcluster or mozmill-ci. Both systems should as best use the packages as shipped with the tests archive for versioning reasons. 

mozilla-beta / mozilla-release / mozilla-esr38
Our packages are initially not in the tree, but could be installed from within the firefox-ui-tests repository checkout. This also allows us to have specific versions for that branch and should be somewhat equal to the tests archive and are easily to swap.

external tests
Those could use the released versions of firefox_puppeteer, and firefox_ui_harness.

Given that our tests are exclusively in the tree we won't have an easy way anymore to fetch them via a github checkout. For the candidate builds we know the revision id via pulse, but not for releases (which are needed for update tests). As long as our tests are not run on RelEng systems, we might have to query for a rev (tag) which represents the release version, pick its parent changeset, and finally get the test package. Due to that reason I would vote to keep the release update tests to run via mozmill. The above should not be a problem for aurora and nightly updates.

We still need firefox-ui-tests as package due to several exports (server root, manifests) which we have to kill soon after the landing in-tree.

We have versioning problems with mozharness. As of now we only use the mozilla-central version for every supported branch. We cannot flip that to branch specific versions because esr38 wont have it, and maybe even beta and release don't have all necessary features. That means we need different steps to be executed based on the branch or availability of the tests in the test archive. I hope that nothing in mozmill ci would have to be changed, beside supplying the test package name.

We should release our packages for every branch in case a UI change has happened in Firefox. So for mozilla-central I would use version 3.0.0 for both packages. Beta and release might get 2.0.0 due to UI changes. And for esr38 I will use 1.0.0.

This is kinda complicated. And I hope we don't break that much tests due to the changes.
The last comment should actually have been submitted to the tracking bug.
Created attachment 8699634 [details] [review]
Github PR (version bump for Nightly)

r+ from David via IRC.
Attachment #8699634 - Flags: review+
Comment on attachment 8699634 [details] [review]
Github PR (version bump for Nightly) (python-package branch)
Attachment #8699634 - Flags: checkin+
Packages for Nightly are up on PyPI:

I will continue to check for other puppeteer versions needed for older branches tomorrow.
Blocks: 1233736
Created attachment 8700631 [details] [review]
Github PR (requirements.txt and travis updates)
Attachment #8700631 - Flags: review?(mjzffr)
Attachment #8700631 - Flags: review?(mjzffr) → review+
Comment on attachment 8700631 [details] [review]
Github PR (requirements.txt and travis updates)

Merged to mozilla-central:

I will backport it to all other branches tomorrow, given that this would stop us from having to backport the packaging PR down to esr38, which would be a hard work.
Hm, looks like I missed one more thing... When I run mozharness with the fx ui packages I get a failure:

> 18:20:01     INFO -  /mozilla/code/gecko/testing/mozharness/build/venv/bin/python: can't open file '/mozilla/code/gecko/testing/mozharness/build/firefox_ui_tests/firefox_ui_harness/': [Errno 2] No such file or directory

That reminds me that we do not use the CLI command to trigger the functional tests but the script directly to not trigger the UAC prompt. The same applies to the update script.
I filed bug 1234345 for the mozharness case to get a script added. It should mostly fix the remaining issue.

Also I checked current results from a mozilla-central test for the requirements.txt and Travis changes. All looked fine. So I went ahead and backported the patch: (aurora) (beta) (release)

ESR38 has merge conflicts. So I have to get them solved first. I might have done this patch on a different bug. :/
Created attachment 8700796 [details] [review]
Github PR (correct packaging & backward compatibility)
Attachment #8700796 - Flags: review?(mjzffr)
Comment on attachment 8700796 [details] [review]
Github PR (correct packaging & backward compatibility)

r+ lgtm, assuming travis build passes
Attachment #8700796 - Flags: review?(mjzffr) → review+
Landed PR as two distinct commits to have the version bump of firefox_ui_harness separate:

I will finish the backport tomorrow. Also once the necessary mozharness changes went in, we can get the python_package branch merged to m-c.
status-firefox45: --- → affected
status-firefox46: --- → fixed
Btw. firefox_ui_harness 1.1.0 has been released:

Submitting dist/firefox_ui_harness-1.1.0.tar.gz to
Server response (200): OK
Created attachment 8701026 [details] [review]
Github PR (requirements.txt and travis updates - esr38 backport)

This is the backport for esr38. It applies cleanly and Travis doesn't have any problems in running the tests. There is a single test failure only which caused one of the builds to fail.
Attachment #8701026 - Flags: review?(mjzffr)
Attachment #8701026 - Flags: review?(mjzffr) → review+
Comment on attachment 8701026 [details] [review]
Github PR (requirements.txt and travis updates - esr38 backport)

Merged PR to mozilla-esr38 as:
Attachment #8701026 - Flags: checkin+
Attachment #8700631 - Flags: checkin+
Attachment #8700796 - Flags: checkin+
Attachment #8699224 - Flags: checkin+
Remaining tasks to do here:

* Merge the python_packages branch to mozilla-central (github) once the mozharness patch on bug  	1212609 got landed
* Backport the packaging work to mozilla-aurora
Whiteboard: [blocked by mozharness patch on bug 1212609]
Blocks: 1236569
Blocks: 1228278
No longer blocks: 1228278
Product: Mozilla QA → Testing
Blocks: 1237322
No longer blocks: 1236569
Given that I don't want to wait longer for getting all mozharness changes landed to support in-tree tests (which we plainly don't need for older branches) I just pushed an update oh mozharness to one of the latest mozilla-central revisions. It will include the required fix and also some other enhancements.
Whiteboard: [blocked by mozharness patch on bug 1212609]
I was waiting for some Aurora builds to appear for production and all looks fine with our tests:

So I'm going to merge the python_packages branch into our mozilla-central branch on github now.
I squashed the python_packages branch and pushed it to mozilla-central:

I will closely watch test results today for Nightly builds on production. In parallel I will start to create the packages for the mozilla-aurora branch.
Target Milestone: --- → mozilla46
The aurora-backport looks to be trivial. The cherry-pick applied both necessary changesets from this bug and bug 1237322 cleanly. The only thing we have to take care of is that the puppeteer version for mozilla-aurora cannot have bug 1237329 included. Which also means we need a separate release for puppeteer. Given that 3.1.0 has been released for Firefox 46.0, we can make use of 2.0.0 for Firefox 45.0.
Created attachment 8709059 [details] [review]
Github PR (backport of packages to mozilla-aurora)

Those backport changes are close to the code we have on mozilla-aurora. It only varies in branch names, versions of packages and mozilla-central only changes.
Attachment #8709059 - Flags: review?(spolk)
Attachment #8709059 - Flags: review?(mjzffr)
Comment on attachment 8709059 [details] [review]
Github PR (backport of packages to mozilla-aurora)

Both Syd and Maja are not available today. So also adding David, who could review it later. David, if you think that Syd and Maja should have a look at the patch, please note it when doing the review.  Thanks.
Attachment #8709059 - Flags: review?(dburns)
Comment on attachment 8709059 [details] [review]
Github PR (backport of packages to mozilla-aurora)

This looks not broken, but honestly it's hard to tell whether anything could have been missed.
Attachment #8709059 - Flags: review?(mjzffr) → review+
Created attachment 8709435 [details] [diff] [review]
interdiff (aurora patch to mozilla-central)

Maja asked for some clarity that nothing is missing.  So here the interdiff between the current packaging branch and mozilla-central.
Attachment #8709435 - Flags: feedback?(mjzffr)
Attachment #8709435 - Flags: feedback?(mjzffr) → feedback+


2 years ago
Attachment #8709059 - Flags: review?(spolk) → review+
Attachment #8709059 - Flags: review?(dburns)
Before we can merge this patch to mozilla-aurora we have to get the puppeteer release uploaded to pypi and also mirrored to the internal server.

The release of firefox-puppeteer 2.0.0 happened here:

Submitting dist/firefox-puppeteer-2.0.0.tar.gz to
Server response (200): OK
Depends on: 1240810
I merged the python packages branch into mozilla-aurora now:

So nothing else has to be done on the github repository. The remaining patch to add those packages to I will attach to bug 1214372.
Last Resolved: 2 years ago
status-firefox45: affected → fixed
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.