Closed Bug 1237600 Opened 4 years ago Closed 4 years ago

Installing firefox-ui-packages from firefox_ui_requirements.txt should not download packages from pypi

Categories

(Release Engineering :: Applications: MozharnessCore, defect, major)

defect
Not set
major

Tracking

(firefox45 fixed, firefox46 fixed)

RESOLVED FIXED
Tracking Status
firefox45 --- fixed
firefox46 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

Attachments

(2 files)

Not sure why this happens but somehow the puppeteer package gets downloaded from pypi / internal mirror while it is present locally:

05:33:15     INFO -  Collecting firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3))
05:33:16     INFO -    Could not find a version that satisfies the requirement firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3)) (from versions: 3.0.0)
05:33:17     INFO -  No matching distribution found for firefox-puppeteer<4.0.0,>=3.1.0 (from firefox-ui-tests==0.3.1->-r /Users/mozauto/jenkins/workspace/mozilla-central_functional/build/tests/config/firefox_ui_requirements.txt (line 3))
For now I filed bug 1237599 to get the new packages mirrored to the internal pypi mirror. That should hopefully stop the bustage.
Attached file build.log
Here an example build.log from Jenkins.
Ok, so even when I run `pip install -r mozbase_requirements.txt` from within testing/config/ I can see that some mozbase packages are getting downloaded from pypi and NOT installed from in-tree. 

Downloading/unpacking mozfile>=1.0 (from mozcrash==0.16->-r mozbase_requirements.txt (line 2))
  Getting page https://pypi.python.org/simple/mozfile/
  URLs to search for versions for mozfile>=1.0 (from mozcrash==0.16->-r mozbase_requirements.txt (line 2)):
  * https://pypi.python.org/simple/mozfile/
  Analyzing links from page https://pypi.python.org/simple/mozfile/
[..]
  Using version 1.2 (newest of versions: 1.2, 1.1, 1.0)
  Downloading mozfile-1.2.tar.gz
  Downloading from URL https://pypi.python.org/packages/source/m/mozfile/mozfile-1.2.tar.gz#md5=414e32aa6266fd4c7344fbcea1e6bccc (from https://pypi.python.org/simple/mozfile/)

So I don't think that this a problem for our firefox_ui_requirements.txt but a more general issue.

Andrew or Jonathan, do you know what that happens? Do I miss an option for the pip command? Thing is that also mozharness fails in that step and tries to download a package from the internal mirror. And this is definitely not wanted here.
Flags: needinfo?(jgriffin)
Flags: needinfo?(ahalberstadt)
Just found bug 908356 which originally added this requirements file to the tree. Looks like I miss the two_pass when running pip.
Ok, it's indeed the missing two_pass option in `register_virtualenv_module()` which is causing this behavior. I should have a patch shortly.
Flags: needinfo?(jgriffin)
Flags: needinfo?(ahalberstadt)
With the fix in place I get the following:

23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/harness
23:42:50     INFO -    Running setup.py (path:/tmp/pip-T61RY2-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/harness
23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/tests
23:42:50     INFO -    Running setup.py (path:/tmp/pip-5Fjeab-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/firefox-ui/tests
23:42:50     INFO -  Unpacking /mozilla/code/gecko/testing/mozharness/build/tests/puppeteer/firefox
23:42:50     INFO -    Running setup.py (path:/tmp/pip-hkGshS-build/setup.py) egg_info for package from file:///mozilla/code/gecko/testing/mozharness/build/tests/puppeteer/firefox

And that looks great!
So this is a mozharness issue -> moving to the appropriate component.
Component: Firefox UI Tests → Mozharness
Product: Testing → Release Engineering
QA Contact: hskupin → jlund
Attached patch Patch v1Splinter Review
Those changes fix our problem with the unwanted download of packages from pypi or the internal mirror.
Attachment #8705379 - Flags: review?(jlund)
FYI I removed the "# Optional packages to be installed, e.g. for Jenkins" section because its not in use anymore.
Comment on attachment 8705379 [details] [diff] [review]
Patch v1

Review of attachment 8705379 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)
Attachment #8705379 - Flags: review?(jlund) → review+
https://hg.mozilla.org/mozilla-central/rev/ddce10c63a22
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Currently we run our tests in mozmill-ci via the mozilla-central version of mozharness. But this might change for the 45ESR branch. I don't want to keep backward-compat code around for more than a year. So lets get the mozharness changes uplifted for 45:

https://hg.mozilla.org/releases/mozilla-aurora/rev/468638b88f3c
You need to log in before you can comment on or make changes to this bug.