Closed Bug 1361732 Opened 7 years ago Closed 7 years ago

Mozharness support for Python3 venv creation

Categories

(Release Engineering :: Applications: MozharnessCore, enhancement)

Product:
Release Engineering
Component:
Applications: MozharnessCore
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: gbrown)

References

Details

Attachments

(7 files, 1 obsolete file)

Bug 1361732 - Mozharness support for a second Python3 virtualenv
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1361732 - Mozharness' Talos class to use the Python3Virtualenv helper class.
59 bytes, text/x-review-board-request
mozilla
: review+
Details
WIP - Other pip installations methods
59 bytes, text/x-review-board-request
gbrown
: review+
Details
mitmproxy requirements
59 bytes, text/x-review-board-request
gbrown
: review+
Details
fix path to manifest
836 bytes, patch
jmaher
: review+
Details | Diff | Splinter Review
adjust PATH so python3 can find C runtime on Windows
3.20 KB, patch
jmaher
: review+
Details | Diff | Splinter Review
fix 'pip install'
2.41 KB, patch
jmaher
: review+
Details | Diff | Splinter Review 
We're going to be deploying Python 3.5 to Win7 and Win10 machines.

The recommended method for creating a virtualenv for Python3 is like this:
> python3 -v venv /path/to/venv

For the Windows machines (with the custom installation dir) will look like this:
* From cmd.exe
> C:\Python3.5\python.exe -m venv C:\Users\armenzg\venv 
* From start-shell.bat
> /c/Python3.5/python -m venv /c/Users/armenzg/venv

We will need Mozharness to support two different venvs and a different command for venv creation.
At this point, why not Python 3.6 -- it has a huge number of improvements over 3.5, and is out and stable.
In isolation, 3.6 makes sense.  In our infra, we have a 3.5 RPM and code that's expected to run in 3.5, and 3.6 introduces new syntax that will not run in 3.5.  So it depends on whether we're going for the latest and greatest, or more compatibility.
What platforms do you want to support, and do you need to spuport it on hardware or just in cloud instances?
The main focus at this point is for platforms that run Talos. Mainly Windows 10 and Windows 7.

I'm going to try using tooltool to use the Python version we want (bug 1361462).
My main concern on that bug is that using tooltool will not add too much overhead.

This bug is specifically for creating a second virtualenv and using a 3.5+ Python binary.
See Also: → 1361462
I'm very much in favor of doing this in-tree, and tooltool is a good way of doing so.
The WIP code pushed does the venv creation in-tree.
I will try to meet with aki today to see the right way of shaping the patch.
I'm not too excited on how unnatural is to add code to the Mixin ecosystem.

I will go back to tooltool in bug 1361462. If unzipping takes 2-3 mins like on my local VM, I won't be so sure if we want that extra overhead for every talos mitmproxy job.
Comment on attachment 8864520 [details]
Bug 1361732 - Mozharness support for a second Python3 virtualenv

https://reviewboard.mozilla.org/r/136202/#review139306

::: testing/mozharness/mozharness/base/python.py:803
(Diff revision 1)
>  
>  
> +# This needs to be inherited only if you have already inherited ScriptMixin
> +class Python3Virtualenv(object):
> +    ''' Support Python3.5+ virtualenv creation.'''
> +    initialized = False

This is going to be self.initialized for the entire script object.  The name is not very informative: initialized for what?  self.py3_initialized or something might be better.

::: testing/mozharness/mozharness/base/python.py:815
(Diff revision 1)
> +        '''
> +        self.initialized = True
> +        # XXX: We need to verify that the version is greater than 3.??
> +        self.python_path = os.path.abspath(python_path)
> +        self.venv_path = os.path.abspath(venv_path)
> +        self.pip_path = os.path.join(self.path_to_python3_executables(), 'pip')

These three are going to conflict with py2 variables.  If you're planning on supporting both py2 and py3, I recommend using '3' in the names.
Attachment #8864520 - Flags: review?(aki)
Summary: Support Python3 venv creation → Mozharness support for Python3 venv creation
I've separated the commit into two.

aki, I would like this review before I have complete support for requirements' files and other pip options.

Ideally, the next iteration should support this command and requirements.txt:
> pip install --no-index --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host pypi.pub.build.mozilla.org -r requirements.txt

Contents of requirements.txt (all requirements for mitmproxy):
argh==0.26.2
asn1crypto==0.22.0
blinker==1.4
brotlipy==0.6.0
certifi==2017.4.17
cffi==1.10.0
click==6.7
construct==2.8.12
cryptography==1.8.1
cssutils==1.0.2
EditorConfig==0.12.1
h2==2.6.2
hpack==3.0.0
html2text==2016.9.19
hyperframe==4.0.2
idna==2.5
jsbeautifier==1.6.12
kaitaistruct==0.6
mitmproxy==2.0.2
packaging==16.8
passlib==1.7.1
pathtools==0.1.2
pyasn1==0.2.3
pycparser==2.17
pyOpenSSL==16.2.0
pyparsing==2.2.0
pyperclip==1.5.27
PyYAML==3.12
requests==2.13.0
ruamel.yaml==0.13.14
six==1.10.0
sortedcontainers==1.5.7
tornado==4.4.3
urwid==1.3.1
watchdog==0.8.3
Comment on attachment 8864520 [details]
Bug 1361732 - Mozharness support for a second Python3 virtualenv

https://reviewboard.mozilla.org/r/136204/#review139756

Overall looks good... sorry I didn't catch these nits beforehand.
r=me with changes.

::: testing/mozharness/mozharness/base/python.py:801
(Diff revision 4)
>      def _tinderbox_print(self, message):
>          self.info('TinderboxPrint: %s' % message)
>  
>  
> +# This needs to be inherited only if you have already inherited ScriptMixin
> +class Python3Virtualenv(object):

We may end up needing to rename this to end in `Mixin`, since python treats mixin objects differently in terms of linting and testing... for example, there is no `self.info` defined in the object; this comes from the script object.  If the name doesn't cause any issues, I think we'll be ok.

::: testing/mozharness/mozharness/base/python.py:816
(Diff revision 4)
> +        self.py3_initialized_venv = True
> +        self.py3_python_path = os.path.abspath(python_path)
> +        version = self.get_output_from_command(
> +                    [self.py3_python_path, '--version']).split()[-1]
> +        # Using -m venv is only used on 3.5+ versions
> +        assert version > '3.5.0'

We may want to self.fatal() rather than bare `assert`, so we get some logging as to why we're dying.

::: testing/mozharness/mozharness/base/python.py:830
(Diff revision 4)
> +            return os.path.join(self.py3_venv_path, 'bin')
> +
> +    def py3_venv_initialized(func):
> +        def call(self, *args, **kwargs):
> +            if not self.py3_initialized_venv:
> +                raise Exception('You need to call py3_venv_configuration() '

This exception can be a `self.fatal` as well.

::: testing/mozharness/mozharness/base/python.py:851
(Diff revision 4)
> +                halt_on_failure=True)
> +
> +    @py3_venv_initialized
> +    def py3_install_modules(self, modules):
> +        if not os.path.exists(self.py3_venv_path):
> +            raise Exception('You need to call py3_create_venv() first.')

If we're catching the exception somewhere, this is good.  If the intention is to kill the script call, `self.fatal` is preferable, so it shows in logs.

::: testing/mozharness/mozharness/base/python.py:857
(Diff revision 4)
> +
> +        for m in modules:
> +            self.run_command('%s install %s' % (self.py3_pip_path, m))
> +
> +
>  # __main__ {{{1

We can probably get rid of this chunk :)
Attachment #8864520 - Flags: review?(aki) → review+
Comment on attachment 8864968 [details]
Bug 1361732 - Mozharness' Talos class to use the Python3Virtualenv helper class.

https://reviewboard.mozilla.org/r/136620/#review139760

Looks good, though if we rename the class we'll need to update this patch as well.
Attachment #8864968 - Flags: review?(aki) → review+
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #13)
> I've separated the commit into two.
> 
> aki, I would like this review before I have complete support for
> requirements' files and other pip options.
> 
> Ideally, the next iteration should support this command and requirements.txt:
> > pip install --no-index --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host pypi.pub.build.mozilla.org -r requirements.txt

I believe these pypi links point at our 2.x puppet python repo.  If that works for you, great.  We also have a python3.5 repo, with a number of non-py2 compatible modules.  If we need to make that available, we may need to adjust those CNAMEs and URLs.
Well! Early parental leave. I won't be able to finish this.

I've pushed a couple more of commits to mozreview so other people can decide if they want to use them or not.

One of them is making the pip support requirement files instead of just supporting a list of modules.
The 2nd one is putting all mitmproxy dependencies into a requirements.txt file.

Take care. See you hopefully on June!
Assignee: armenzg → nobody
Assignee: nobody → gbrown
It looks like Armen has done the hard work here. I'm trying to verify it on try and running into some hopefully minor issues.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1aa286264afe25d9f3281b2e98515bac9554fdb6

https://archive.mozilla.org/pub/firefox/try-builds/gbrown@mozilla.com-1aa286264afe25d9f3281b2e98515bac9554fdb6/try-win32/try_win7_ix_test-tp5o-bm109-tests1-windows-build233.txt.gz

15:26:12     INFO - found python3: C:\slave\test\build\python3.6\python.exe
15:26:12     INFO - Running command: ['C:\\slave\\test\\build\\python3.6\\python.exe', '--version']
15:26:12     INFO - Copy/paste: C:\slave\test\build\python3.6\python.exe --version
15:26:12    ERROR - Return code: -1073741515
15:26:12     INFO - Getting output from command: ['C:\\slave\\test\\build\\python3.6\\python.exe', '--version']
15:26:12     INFO - Copy/paste: C:\slave\test\build\python3.6\python.exe --version
15:26:12     INFO - Using _rmtree_windows ...
15:26:12     INFO - Using _rmtree_windows ...
15:26:12    ERROR - Return code: -1073741515
15:26:12     INFO - raw version: None

verifies that the file C:\slave\test\build\python3.6\python.exe exists, but running 'C:\slave\test\build\python3.6\python.exe --version' returns a troubling exit code and no output.

I don't have convenient access to a Windows environment; it would be helpful if someone could verify that 'python3.6\python.exe --version' works. It has to, right? What could be going wrong here?
Googling for `windows "error code -1073741515"` gives me:
http://stackoverflow.com/questions/4189109/error-code-1073741515-when-using-editbin

which seems to suggest a missing dll.  Any possiblity of bundling the dll and/or statically linking python?
Thanks Aki, that seems like the right error code and reason. 

The archive already includes vcruntime140.dll, python3.dll, and python36.dll in the same directory as python.exe.
I tried running python.exe from the python directory, but that didn't help.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9db1506dfafd6da74f3bd9f51df52697b2aa521d
I tried extracting the tooltool python3 zip to my own Windows 10 computer: 'python.exe --version' works just fine, of course.
I think you can get access to a windows test loaner, or try running it from any windows box or vm without Visual Studio installed.
jmaher tried running manually on a Windows 10 loaner, had no trouble running python3, and even showed that venv creation was successful.
Hm. It might be worth comparing the test run's env and a successful manual run env.
FYI: On the win7 loaner, I installed the tooltool package and when I run python version I get "the program can't start because api-ms-win-crt-runtime-I1-1-0.dll is missing from your computer"

Found that DLL in:

c:\slave\test\build\application\firefox

Added that to the Win7 machine path and tried again and Python --version worked, 'Python 3.6.1'.
same on win10, we are onto something here :)
This seems consistent with https://docs.python.org/3.6/using/windows.html#embedded-distribution: "The embedded distribution does not include the Microsoft C Runtime and it is the responsibility of the application installer to provide this. The runtime may have already been installed on a user’s system previously or automatically via Windows Update, and can be detected by finding ucrtbase.dll in the system directory." (Link to https://www.microsoft.com/en-us/download/details.aspx?id=48145 .)
I think the loaners we have been looking are for unit tests, or are otherwise different from the environment we are running in.

Consider https://archive.mozilla.org/pub/firefox/try-builds/gbrown@mozilla.com-9fea94f2fdd714680a1680e0b50b480cc8e35126/try-win32/try_win7_ix_test-tp5o-bm109-tests1-windows-build263.txt.gz:

15:16:51     INFO - oops, missing dir: C:\CoreUtils\bin
15:16:51     INFO - oops, missing dir: C:\slave\test\build\application\firefox
15:16:51     INFO - Running command: dir /s python.exe in C:\
15:16:51     INFO -   Volume in drive C is OSDisk
15:16:51     INFO -   Volume Serial Number is 6C28-11B8
15:16:51     INFO -   Directory of C:\mozilla-build\buildbotve\scripts
15:16:51     INFO -  03/14/2013  02:19 PM            26,624 python.exe
15:16:51     INFO -                 1 File(s)         26,624 bytes
15:16:56     INFO -   Directory of C:\mozilla-build\python27
15:16:56     INFO -  04/10/2012  11:31 PM            26,624 python.exe
15:16:56     INFO -                 1 File(s)         26,624 bytes
15:17:09     INFO -   Directory of C:\slave\test\build\python3.6
15:17:09     INFO -  05/11/2017  03:16 PM            97,944 python.exe
15:17:09     INFO -                 1 File(s)         97,944 bytes
15:17:10     INFO -   Directory of C:\slave\test\build\venv\Scripts
15:17:10     INFO -  05/11/2017  03:16 PM            26,624 python.exe
15:17:10     INFO -                 1 File(s)         26,624 bytes
15:17:35     INFO -   Directory of C:\slave\test-pgo\build\venv\Scripts
15:17:35     INFO -  04/13/2016  11:33 AM            26,624 python.exe
15:17:35     INFO -                 1 File(s)         26,624 bytes
15:17:37     INFO -   Directory of C:\talos-slave\test\build\venv\Scripts
15:17:37     INFO -  10/23/2013  06:44 AM            26,624 python.exe
15:17:37     INFO -                 1 File(s)         26,624 bytes
15:18:26     INFO -       Total Files Listed:
15:18:26     INFO -                 6 File(s)        231,064 bytes
15:18:26     INFO -                 0 Dir(s)  432,812,347,392 bytes free
15:18:26     INFO - Return code: 0
15:18:26     INFO - Running command: dir /s api-ms-win-crt-runtime-I1-1-0.dll in C:\
15:18:26     INFO -   Volume in drive C is OSDisk
15:18:26     INFO -   Volume Serial Number is 6C28-11B8
15:18:35     INFO -  File Not Found
15:18:35    ERROR - Return code: 1


C:\slave\test\build\application\firefox does not exist (yet??).

api-ms-win-crt-runtime-I1-1-0.dll does not exist anywhere on C:.
(In reply to Geoff Brown [:gbrown] from comment #33)
> C:\slave\test\build\application\firefox does not exist (yet??).

C:\slave\test\build\application\firefox is created and populated by a later mozharness step. We can probably "cheat" by postponing the python3 installation until after firefox is installed, and using the C runtime files included with Firefox to satisfy the python3 requirements.

        Attachment #8867268 -
        Flags: review?(jmaher)

    
    

    
  
Comment on attachment 8867266 [details] [diff] [review]
adjust PATH so python3 can find C runtime on Windows

Review of attachment 8867266 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/configs/talos/windows_config.py
@@ +51,5 @@
>          "win64": "python3_x64.manifest",
> +    },
> +    "env": {
> +        # python3 requires C runtime, found in firefox installation
> +        "PATH": "%(PATH)s;c:\\slave\\test\\build\\application\\firefox;"

this overall seems wrong since python3 cannot run without firefox being installed?  For now this is OK, maybe a follow up bug filed to track fixing this more generically?

::: testing/mozharness/mozharness/mozilla/testing/talos.py
@@ +510,4 @@
>              output_dir=output_dir,
>              cache=self.config['tooltool_cache']
>          )
> +        python3_path = os.path.join(output_dir, 'python3.6', 'python.exe')

I don't like hardcoding python.exe here, that is windows specific and this is a generic file, could we query something in windows_config.py or do conditional logic if windows binary += ".exe" ?

        Attachment #8867266 -
        Flags: review?(jmaher) → review-

    
    

    
  
Comment on attachment 8867268 [details] [diff] [review]
fix path to manifest

Review of attachment 8867268 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/mozharness/mozilla/testing/talos.py
@@ +499,4 @@
>  
>      def fetch_python3(self):
>          manifest_file = os.path.join(
> +            self.talos_path,

any concerns with this affecting osx or linux?

        Attachment #8867268 -
        Flags: review?(jmaher) → review+

    
    

    
  
(In reply to Joel Maher ( :jmaher) from comment #37)
> > +    "env": {
> > +        # python3 requires C runtime, found in firefox installation
> > +        "PATH": "%(PATH)s;c:\\slave\\test\\build\\application\\firefox;"
> 
> this overall seems wrong since python3 cannot run without firefox being
> installed?  For now this is OK, maybe a follow up bug filed to track fixing
> this more generically?

It's horrid, I know. We need the C runtime and this is the only place it is currently available. I think the better solution is to have it pre-installed on the test image, but that probably has legal/licensing complications. I will file a follow-up bug.

    
    

    
  
(In reply to Joel Maher ( :jmaher
> >      def fetch_python3(self):
> >          manifest_file = os.path.join(
> > +            self.talos_path,
> 
> any concerns with this affecting osx or linux?

I think it will be fine on all platforms.

    
    

    
  


  
The 'pip install' command line was getting confused between a single string vs an array of strings; this sorts that out. 

I've added --trusted-host support to allow for http; we don't usually use that, but it works fine for python3 and is easier than deploying a new config file.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=200b82da695ed9a4d71a43985d3763a04622a7f5

13:17:58     INFO - Copy/paste: C:\slave\test\py3venv\Scripts\pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub --trusted-host pypi.pub.build.mozilla.org --trusted-host pypi.pvt.build.mozilla.org -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt
...
13:17:58     INFO -  'XPCOM_DEBUG_BREAK': 'warn'}
13:17:59     INFO -  Collecting argh==0.26.2 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 1))
13:18:00     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/argh-0.26.2-py2.py3-none-any.whl
13:18:01     INFO -  Collecting asn1crypto==0.22.0 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 2))
13:18:02     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/asn1crypto-0.22.0.tar.gz (84kB)
13:18:02     INFO -  Collecting blinker==1.4 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 3))
13:18:04     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/blinker-1.4.tar.gz (111kB)
13:18:04     INFO -  Collecting brotlipy==0.6.0 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 4))
13:18:06     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/brotlipy-0.6.0.tar.gz (437kB)
13:18:06     INFO -      Complete output from command python setup.py egg_info:
13:18:06     INFO -      Download error on https://pypi.python.org/simple/cffi/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
13:18:06     INFO -      Couldn't find index page for 'cffi' (maybe misspelled?)
13:18:06     INFO -      Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
13:18:06     INFO -      No local packages or working download links found for cffi>=1.0.0
13:18:06     INFO -      Traceback (most recent call last):
13:18:06     INFO -        File "<string>", line 1, in <module>
13:18:06     INFO -        File "C:\Users\CLTBLD~1.001\AppData\Local\Temp\pip-build-kxx89t08\brotlipy\setup.py", line 84, in <module>
13:18:06     INFO -          "Programming Language :: Python :: 3.5",
13:18:06     INFO -        File "C:\slave\test\build\python3.6\lib\distutils\core.py", line 108, in setup
13:18:06     INFO -          _setup_distribution = dist = klass(attrs)
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\dist.py", line 315, in __init__
13:18:06     INFO -          self.fetch_build_eggs(attrs['setup_requires'])
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\dist.py", line 361, in fetch_build_eggs
13:18:06     INFO -          replace_conflicting=True,
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\pkg_resources\__init__.py", line 850, in resolve
13:18:06     INFO -          dist = best[req.key] = env.best_match(req, ws, installer)
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\pkg_resources\__init__.py", line 1122, in best_match
13:18:06     INFO -          return self.obtain(req, installer)
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\pkg_resources\__init__.py", line 1134, in obtain
13:18:06     INFO -          return installer(requirement)
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\dist.py", line 429, in fetch_build_egg
13:18:06     INFO -          return cmd.easy_install(req)
13:18:06     INFO -        File "c:\slave\test\py3venv\lib\site-packages\setuptools\command\easy_install.py", line 659, in easy_install
13:18:06     INFO -          raise DistutilsError(msg)
13:18:06     INFO -      distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=1.0.0')
13:18:06     INFO -      ----------------------------------------
13:18:08     INFO -  Command "python setup.py egg_info" failed with error code 1 in C:\Users\CLTBLD~1.001\AppData\Local\Temp\pip-build-kxx89t08\brotlipy\
13:18:08    ERROR - Return code: 1

We failed, but if I'm reading that correctly, only because some of the mitmproxy requirements are not on our pypi.

        Attachment #8867347 -
        Flags: review?(jmaher)

    
    

    
  
Comment on attachment 8867344 [details] [diff] [review]
adjust PATH so python3 can find C runtime on Windows

Review of attachment 8867344 [details] [diff] [review]:
-----------------------------------------------------------------

some small nits/questions?

::: testing/mozharness/configs/talos/windows_config.py
@@ +51,5 @@
>          "win64": "python3_x64.manifest",
> +    },
> +    "env": {
> +        # python3 requires C runtime, found in firefox installation
> +        "PATH": "%(PATH)s;c:\\slave\\test\\build\\application\\firefox;"

can you add a bug here so in the future when we see this there is context if we haven't edited it correctly?

::: testing/mozharness/mozharness/base/python.py
@@ +888,4 @@
>          for requirement_path in requirements:
>              cmd += ['-r %s' % requirement_path]
>  
> +        self.run_command(cmd, env=self.query_env())

do we need to ' '.join(cmd) here like before?

        Attachment #8867344 -
        Flags: review?(jmaher) → review+

    
    

    
  
(In reply to Joel Maher ( :jmaher) from comment #43)
> can you add a bug here so in the future when we see this there is context if
> we haven't edited it correctly?

Sure, will do.

> do we need to ' '.join(cmd) here like before?

The mozharness run_command is really nice in that it can take a string or an array description of the command line -- I'm just using the [] form here.

    
    

    
  
Comment on attachment 8867347 [details] [diff] [review]
fix 'pip install'

Review of attachment 8867347 [details] [diff] [review]:
-----------------------------------------------------------------

oh I like this!

        Attachment #8867347 -
        Flags: review?(jmaher) → review+

    
    

    
  
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f0f6bd0a4543
Mozharness support for a second virtualenv using python3; r=aki
https://hg.mozilla.org/integration/mozilla-inbound/rev/f37e89ab37a8
Mozharness' Talos class to use the Python3Virtualenv helper class. r=aki
https://hg.mozilla.org/integration/mozilla-inbound/rev/ead82c8f9c67
pip install support for python3 venv; r=gbrown
https://hg.mozilla.org/integration/mozilla-inbound/rev/4e1be5755d6a
Add package requirements for mitmproxy; r=gbrown
https://hg.mozilla.org/integration/mozilla-inbound/rev/41bf163ee237
Use custom PATH for python3; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/6afcef838b7e
Fix path to python3 manifest; r=jmaher
https://hg.mozilla.org/integration/mozilla-inbound/rev/9730f88bdbd3
Fix pip install for python3; r=jmaher

    
    

    
  
Comment on attachment 8865271 [details]
WIP - Other pip installations methods

https://reviewboard.mozilla.org/r/136908/#review142222

This had some minor issues, but I'm using it as-is and then fixing issues in subsequent patches.

Thanks Armen - really appreciate you putting this together.

        Attachment #8865271 -
        Flags: review+

    
    

    
  
Comment on attachment 8865272 [details]
mitmproxy requirements

https://reviewboard.mozilla.org/r/136910/#review142226

        Attachment #8865272 -
        Flags: review+

    
  
Keywords: leave-open

    
    

    
  
This may be all we need to land in this bug, but let's leave open until we are sure.

With these patches, you can use the new functionality like this: https://hg.mozilla.org/try/rev/7b77518d6107b0f609563ebdc47a8e231128ace0. If you do, I expect not all of the mitmproxy requirements will be installed, as noted in comment 42. I'll try to sort that out next week.

    
    

    
  
I noticed our mitmproxy_requirements.txt is not the same as that specified on https://pypi.python.org/pypi/mitmproxy; I don't understand why they are different. I tried an appropriate modification in https://treeherder.mozilla.org/#/jobs?repo=try&revision=7ca3f6b3e0bae28791f4ce10362dfe75c413b35a. Sadly, it still fails pretty much the same way:

13:29:25     INFO -  Collecting blinker>=1.4 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 1))
13:29:26     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/blinker-1.4.tar.gz (111kB)
13:29:27     INFO -  Collecting brotlipy>=0.5.1 (from -r C:\slave\test\build\tests\talos\mitmproxy_requirements.txt (line 2))
13:29:28     INFO -    Downloading http://pypi.pvt.build.mozilla.org/pub/brotlipy-0.6.0.tar.gz (437kB)
13:29:29     INFO -      Complete output from command python setup.py egg_info:
13:29:29     INFO -      Download error on https://pypi.python.org/simple/cffi/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
13:29:29     INFO -      Couldn't find index page for 'cffi' (maybe misspelled?)
13:29:29     INFO -      Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found!
13:29:29     INFO -      No local packages or working download links found for cffi>=1.0.0


:aki, in comment 16, you mentioned a python 3.5 repo. How can I use that?
Flags: needinfo?(aki)

    
    

    
  
as a note, it looks like I got this working despite the cffi error:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7fe50918bd6ff176eb1aa134ce3e1ccd5bfba3fa

this just get mitmdump running successfully, we fail to install the certs, but that is more of an issue of finding paths/code- getting closer.

It would be nice to use the 3.x pypi to ensure proper libraries.

    
    

    
  
We have http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages-3.5/ which is mirrored across the releng puppet servers.  This is a sibling to http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages/ .  This might only be useful to puppet, however, since I don't know of a similar cname/alias like http://pypi.pvt.build.mozilla.org/pub/ (which is py2).

We may be able to set up another alias in pypi.pvt.build.mozilla.org to point to the py3 repo.  I'm not sure where that server config is defined.
Flags: needinfo?(aki)

    
  
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED

    
    

    
  
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: