Use virtualenv/pip from source checkout, if available

RESOLVED FIXED

Status

Release Engineering
Applications: MozharnessCore
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: gps, Assigned: gps)

Tracking

(Blocks: 3 bugs)

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox52 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(6 attachments, 3 obsolete attachments)

(Assignee)

Description

2 years ago
I've got some scope bloat I wrote as part of bug 1286900 related to virtualenv/pip management foo. It makes virtualenv/pip foo faster and more robust.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Updated

2 years ago
Attachment #8793112 - Attachment is obsolete: true
Attachment #8793112 - Flags: review?(ted)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Updated

2 years ago
Attachment #8793427 - Attachment is obsolete: true
Attachment #8793427 - Flags: review?(ted)
(Assignee)

Updated

2 years ago
Attachment #8793438 - Attachment is obsolete: true
Attachment #8793438 - Flags: review?(ted)
Duplicate of this bug: 1297515
Comment on attachment 8793049 [details]
Bug 1304176 - Set BaseScript.topsrcdir if we have a source checkout;

https://reviewboard.mozilla.org/r/79858/#review79434

::: testing/mozharness/mozharness/base/script.py:1789
(Diff revision 3)
>          self.script_obj = self
>  
> +        # Indicate we're a source checkout if VCS directory is present at the
> +        # appropriate place. This assumes we're at
> +        # testing/mozharness/mozharness/base/script.py.
> +        here = os.path.normpath(os.path.dirname(__file__))

It'd be good to `assert(here.endswith('testing/mozharness/mozharness/base/script.py'))` to make sure this doesn't get broken in a file move.
Attachment #8793049 - Flags: review?(ted) → review+
Comment on attachment 8793050 [details]
Bug 1304176 - Use vendored tooltool.py if available;

https://reviewboard.mozilla.org/r/79860/#review79440

::: testing/mozharness/mozharness/mozilla/tooltool.py:49
(Diff revision 3)
>          if os.path.exists(fn):
>              return fn
>  
>      def tooltool_fetch(self, manifest,
>                         output_dir=None, privileged=False, cache=None):
>          """docstring for tooltool_fetch"""

LOL
Comment on attachment 8793050 [details]
Bug 1304176 - Use vendored tooltool.py if available;

https://reviewboard.mozilla.org/r/79860/#review79442
Attachment #8793050 - Flags: review?(ted) → review+
Comment on attachment 8793051 [details]
Bug 1304176 - Use vendored virtualenv if available;

https://reviewboard.mozilla.org/r/79862/#review79444

::: testing/mozharness/mozharness/base/python.py:376
(Diff revision 4)
> +            ]
> +            virtualenv_options = c.get('virtualenv_options', [])
> +            # Don't create symlinks. If we don't do this, permissions issues may
> +            # hinder virtualenv creation or operation. Ideally we should do this
> +            # below when using the system virtualenv. However, this is a newer
> +            # feature and isn't guaranteed to be supported.

Is this because we have bad permissions on some of our vendored packages or what?
Attachment #8793051 - Flags: review?(ted) → review+
Comment on attachment 8793109 [details]
Bug 1304176 - Remove PIP_TRUSTED_HOST and trust-host pip.conf option;

https://reviewboard.mozilla.org/r/79882/#review79446
Attachment #8793109 - Flags: review?(ted) → review+
Comment on attachment 8793110 [details]
Bug 1304176 - Remove --venv-path as an alias to --virtualenv-path;

https://reviewboard.mozilla.org/r/79884/#review79450
Attachment #8793110 - Flags: review?(ted) → review+
Comment on attachment 8793111 [details]
Bug 1304176 - Document and refactor query_virtualenv_path;

https://reviewboard.mozilla.org/r/79886/#review79452
Attachment #8793111 - Flags: review?(ted) → review+
(Assignee)

Comment 35

2 years ago
mozreview-review-reply
Comment on attachment 8793051 [details]
Bug 1304176 - Use vendored virtualenv if available;

https://reviewboard.mozilla.org/r/79862/#review79444

> Is this because we have bad permissions on some of our vendored packages or what?

I have a suspicion it is a bug in virtualenv. I think it is creating a symlink in the virtualenv to the system-managed site-packages directory where distutils is installed. It then attempts to overwrite that file, yielding a permissions failure.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 42

2 years ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2e07640f74cf
Set BaseScript.topsrcdir if we have a source checkout; r=ted
https://hg.mozilla.org/integration/autoland/rev/b64330d3db8d
Use vendored tooltool.py if available; r=ted
https://hg.mozilla.org/integration/autoland/rev/3fd83c9c0548
Use vendored virtualenv if available; r=ted
https://hg.mozilla.org/integration/autoland/rev/8dc87cef62fb
Remove PIP_TRUSTED_HOST and trust-host pip.conf option; r=ted
https://hg.mozilla.org/integration/autoland/rev/1c0787d40b97
Remove --venv-path as an alias to --virtualenv-path; r=ted
https://hg.mozilla.org/integration/autoland/rev/d5355738ce1e
Document and refactor query_virtualenv_path; r=ted
Depends on: 1305481
(Assignee)

Updated

2 years ago
Blocks: 1305804
Depends on: 1306421
The attempt to use --trusted-host breaks our in-progress work on TC Windows tests:

https://public-artifacts.taskcluster.net/Q36z7Ot9Sg-klAeWDrQr9g/0/public/logs/live_backing.log

17:49:02     INFO - Running command: ['build/venv/Scripts/easy_install.exe', '--find-links', 'http://pypi.pub.build.mozilla.org/pub', '--trusted-host', 'pypi.pub.build.mozilla.org', 'pywin32'] in Z:\task_1475256254\build
...
17:49:04     INFO -  usage: easy_install [options] requirement_or_url ...
17:49:04     INFO -     or: easy_install --help
17:49:04     INFO -  error: option --trusted-host not recognized
17:49:04  WARNING - Return code: 1
17:49:04     INFO - retry: Failed, sleeping 60 seconds before retrying


Do you have a suggestion for dealing with that?
Flags: needinfo?(gps)
(Assignee)

Comment 45

2 years ago
easy_install should not be used in 2016. I'll fix mozharness.
Flags: needinfo?(gps)
(Assignee)

Updated

2 years ago
Blocks: 1306691

Updated

2 years ago
See Also: → bug 1280474
(Assignee)

Updated

2 years ago
Blocks: 1307305
You need to log in before you can comment on or make changes to this bug.