1240530 - No attribute PROTOCOL_SSLv3 error for artifact builds on Linux

Reporter

Description

•

6 years ago

Trying to compile artifact build on Linux yields an error:

Output:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

% python --version
Python 2.7.9

mozconfig:

  mk_add_options AUTOCLOBBER=1

  ac_add_options --with-ccache
  ac_add_options --enable-rust

  ac_add_options --enable-application=browser
  ac_add_options --enable-artifact-builds

  mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../build/artifact-desktop-release

Gregory Szorc [:gps]

Comment 1

•

6 years ago

This might be an upstream bug in requests/urllib3.

Another workaround is to install Python 2.7.9+ to get a Python distribution with proper security support. I recommend https://github.com/yyuu/pyenv for installing Python in userland.

Gregory Szorc [:gps]

Comment 2

•

6 years ago

Just noticed from my output of an artifact build that we're installing requests-2.4.3. This is an ancient version that has known security vulnerabilities. It is also almost certainly not compatible with Python < 2.7.9 and the source of this bug.

Component: mach → Build Config

Gregory Szorc [:gps]

Comment 3

•

6 years ago

We have requests 2.5.1 vendored in m-c under python/requests. It should be upgraded to latest and `mach artifact` should use it.

Nick Alexander :nalexander [he/him]

Updated

•

6 years ago

Blocks: artifact

Nick Alexander :nalexander [he/him]

Comment 4

•

6 years ago

We might just bump the taskcluster client version, which may have advanced.  I haven't checked in a long time.

Chris Manchester (limited bugmail, email directly)

Assignee

Comment 5

•

6 years ago

(In reply to Nick Alexander :nalexander from comment #4)
> We might just bump the taskcluster client version, which may have advanced. 
> I haven't checked in a long time.

This fixes the issue for me locally (as does |./mach python ./mach build|) by picking up the in-tree requests, which is apparently recent enough to avoid this issue.

Chris Manchester (limited bugmail, email directly)

Assignee

Comment 6

•

6 years ago

Attached file MozReview Request: Bug 1240530 - Bump the taskcluster version installed by |./mach artifact| to avoid installing an out-dated requests. r=nalexander — Details

Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.

Review commit: https://reviewboard.mozilla.org/r/31525/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/31525/

Attachment #8709722 - Flags: review?(nalexander)

Chris Manchester (limited bugmail, email directly)

Assignee

Comment 7

•

6 years ago

ato, can you confirm the fix in comment 6 is effective?

Flags: needinfo?(ato)

Nick Alexander :nalexander [he/him]

Comment 8

•

6 years ago

Comment on attachment 8709722 [details]
MozReview Request: Bug 1240530 - Bump the taskcluster version installed by |./mach artifact| to avoid installing an out-dated requests. r=nalexander

https://reviewboard.mozilla.org/r/31525/#review28217

Code is fine.  Ship it if you're confident with your testing.

Attachment #8709722 - Flags: review?(nalexander) → review+

Andreas Tolfsen ❲:ato❳

Reporter

Comment 9

•

6 years ago

With the change applied, it hangs with the following stacktrace:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
New python executable in /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7
Not overwriting existing python script /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python (you must use /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7)
Installing setuptools, pip...done.
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil

Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

And when I interrupt it with SIGINT, I get the following:

^?Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ato/1/gecko/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 86, in _collect
    while not pipe.poll(sleep_interval):
KeyboardInterrupt
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Assignee: nobody → cmanchester

Status: NEW → ASSIGNED

Flags: needinfo?(ato)

Andreas Tolfsen ❲:ato❳

Reporter

Comment 10

•

6 years ago

Of course I should have done a clobber first.  After the clobber, I get a different error that I’m not sure is related to this bug?

https://gist.github.com/andreastt/742335c6b460b8211dde

Flags: needinfo?(cmanchester)

Nick Alexander :nalexander [he/him]

Comment 11

•

6 years ago

(In reply to Andreas Tolfsen (:ato) from comment #10)
> Of course I should have done a clobber first.  After the clobber, I get a
> different error that I’m not sure is related to this bug?
> 
> https://gist.github.com/andreastt/742335c6b460b8211dde

https://gist.github.com/andreastt/742335c6b460b8211dde#file-gistfile1-txt-L271 is pretty clear:

 0:14.32 Try running |hg pushlogsync|; see https://developer.mozilla.org/en-US/docs/Artifact_builds

It looks like something has gone wrong with your configuration, can you follow these instructions and make sure you're locally configured correctly?

Chris Manchester (limited bugmail, email directly)

Assignee

Comment 12

•

6 years ago

Definitely try `hg pushlogsync` before proceeding (I recall needing to do this). Note also this isn't going to work as expected unless you're developing against fx-team (bug 1240667 tracks removing that requirement).

Flags: needinfo?(cmanchester)

Pulsebot

Comment 13

•

6 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/a1c1d5cf87ce

Andreas Tolfsen ❲:ato❳

Reporter

Comment 14

•

6 years ago

nalexander, chmanchester: I should have read the earlier error message more carefully.  Running `hg pushlogsync` as suggested first resolves that issue.

With this change I’m able to do artifact-based builds on Linux, which is great news!  Thanks!

Carsten Book [:Tomcat]

Comment 15

•

6 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/a1c1d5cf87ce

Status: ASSIGNED → RESOLVED

Closed: 6 years ago

status-firefox46: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla46

BMO Automation

Updated

•

4 years ago

Product: Core → Firefox Build System

Bugzilla

No attribute PROTOCOL_SSLv3 error for artifact builds on Linux

Categories

(Firefox Build System :: General, defect)

Tracking

(firefox46 fixed)

People

(Reporter: ato, Assigned: chmanchester)

References

(Blocks 1 open bug)

Details

Crash Data

Security

(public)

User Story

Attachments

(1 file)

Description

Comment 1

Comment 2

Comment 3

Updated

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Comment 14

Comment 15

Updated