1237600 - Installing firefox-ui-packages from firefox_ui_requirements.txt should not download packages from pypi

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Not sure why this happens but somehow the puppeteer package gets downloaded from pypi / internal mirror while it is present locally:

05:33:15     INFO -  Collecting firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3))
05:33:16     INFO -    Could not find a version that satisfies the requirement firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3)) (from versions: 3.0.0)
05:33:17     INFO -  No matching distribution found for firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3))

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For now I filed bug 1237599 to get the new packages mirrored to the internal pypi mirror. That should hopefully stop the bustage.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: build.log

Here an example build.log from Jenkins.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so even when I run `pip install -r mozbase_requirements.txt` from within testing/config/ I can see that some mozbase packages are getting downloaded from pypi and NOT installed from in-tree. 

Downloading/unpacking mozfile>=1.0 (from mozcrash==0.16->-r mozbase_requirements.txt (line 2))
  Getting page https://pypi.python.org/simple/mozfile/
  URLs to search for versions for mozfile>=1.0 (from mozcrash==0.16->-r mozbase_requirements.txt (line 2)):
  * https://pypi.python.org/simple/mozfile/
  Analyzing links from page https://pypi.python.org/simple/mozfile/
[..]
  Using version 1.2 (newest of versions: 1.2, 1.1, 1.0)
  Downloading mozfile-1.2.tar.gz
  Downloading from URL https://pypi.python.org/packages/source/m/mozfile/mozfile-1.2.tar.gz#md5=414e32aa6266fd4c7344fbcea1e6bccc (from https://pypi.python.org/simple/mozfile/)

So I don't think that this a problem for our firefox_ui_requirements.txt but a more general issue.

Andrew or Jonathan, do you know what that happens? Do I miss an option for the pip command? Thing is that also mozharness fails in that step and tries to download a package from the internal mirror. And this is definitely not wanted here.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Just found bug 908356 which originally added this requirements file to the tree. Looks like I miss the two_pass when running pip.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, it's indeed the missing two_pass option in `register_virtualenv_module()` which is causing this behavior. I should have a patch shortly.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the fix in place I get the following:

23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/harness
23:42:50     INFO -    Running setup.py (path:/tmp/pip-T61RY2-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/harness
23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/tests
23:42:50     INFO -    Running setup.py (path:/tmp/pip-5Fjeab-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/tests
23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/puppeteer/firefox
23:42:50     INFO -    Running setup.py (path:/tmp/pip-hkGshS-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/puppeteer/firefox

And that looks great!

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So this is a mozharness issue -> moving to the appropriate component.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1

Those changes fix our problem with the unwanted download of packages from pypi or the internal mirror.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

FYI I removed the "# Optional packages to be installed, e.g. for Jenkins" section because its not in use anymore.

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8705379 [details] [diff] [review]
Patch v1

Review of attachment 8705379 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ddce10c63a22

Comment 12

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/ddce10c63a22

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Currently we run our tests in mozmill-ci via the mozilla-central version of mozharness. But this might change for the 45ESR branch. I don't want to keep backward-compat code around for more than a year. So lets get the mozharness changes uplifted for 45:

https://hg.mozilla.org/releases/mozilla-aurora/rev/468638b88f3c