1240530 - No attribute PROTOCOL_SSLv3 error for artifact builds on Linux

Reporter

Description

•

4 years ago

Trying to compile artifact build on Linux yields an error:

Output:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

% python --version
Python 2.7.9

mozconfig:

  mk_add_options AUTOCLOBBER=1

  ac_add_options --with-ccache
  ac_add_options --enable-rust

  ac_add_options --enable-application=browser
  ac_add_options --enable-artifact-builds

  mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../build/artifact-desktop-release

Gregory Szorc [:gps]

Comment 1

•

4 years ago

This might be an upstream bug in requests/urllib3.

Another workaround is to install Python 2.7.9+ to get a Python distribution with proper security support. I recommend https://github.com/yyuu/pyenv for installing Python in userland.

Gregory Szorc [:gps]

Comment 2

•

4 years ago

Just noticed from my output of an artifact build that we're installing requests-2.4.3. This is an ancient version that has known security vulnerabilities. It is also almost certainly not compatible with Python < 2.7.9 and the source of this bug.

Component: mach → Build Config

Gregory Szorc [:gps]

Comment 3

•

4 years ago

We have requests 2.5.1 vendored in m-c under python/requests. It should be upgraded to latest and `mach artifact` should use it.

Nick Alexander :nalexander [he/him]

Updated

•

4 years ago

Blocks: artifact

Nick Alexander :nalexander [he/him]

Comment 4

•

4 years ago

We might just bump the taskcluster client version, which may have advanced.  I haven't checked in a long time.

Chris Manchester (:chmanchester)

Assignee

Comment 5

•

4 years ago

(In reply to Nick Alexander :nalexander from comment #4)
> We might just bump the taskcluster client version, which may have advanced. 
> I haven't checked in a long time.

This fixes the issue for me locally (as does |./mach python ./mach build|) by picking up the in-tree requests, which is apparently recent enough to avoid this issue.

Chris Manchester (:chmanchester)

Assignee

Comment 6

•

4 years ago

Attached file MozReview Request: Bug 1240530 - Bump the taskcluster version installed by |./mach artifact| to avoid installing an out-dated requests. r=nalexander — Details

Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.

Review commit: https://reviewboard.mozilla.org/r/31525/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/31525/

Attachment #8709722 - Flags: review?(nalexander)

Chris Manchester (:chmanchester)

Assignee

Comment 7

•

4 years ago

ato, can you confirm the fix in comment 6 is effective?

Flags: needinfo?(ato)

Nick Alexander :nalexander [he/him]

Comment 8

•

4 years ago

Comment on attachment 8709722 [details]
MozReview Request: Bug 1240530 - Bump the taskcluster version installed by |./mach artifact| to avoid installing an out-dated requests. r=nalexander

https://reviewboard.mozilla.org/r/31525/#review28217

Code is fine.  Ship it if you're confident with your testing.

Attachment #8709722 - Flags: review?(nalexander) → review+

Andreas Tolfsen 「:ato」

Reporter

Comment 9

•

4 years ago

With the change applied, it hangs with the following stacktrace:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
New python executable in /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7
Not overwriting existing python script /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python (you must use /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7)
Installing setuptools, pip...done.
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil

Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

And when I interrupt it with SIGINT, I get the following:

^?Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ato/1/gecko/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 86, in _collect
    while not pipe.poll(sleep_interval):
KeyboardInterrupt
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Assignee: nobody → cmanchester

Status: NEW → ASSIGNED

Flags: needinfo?(ato)

Andreas Tolfsen 「:ato」

Reporter

Comment 10

•

4 years ago

Of course I should have done a clobber first.  After the clobber, I get a different error that I’m not sure is related to this bug?

https://gist.github.com/andreastt/742335c6b460b8211dde

Flags: needinfo?(cmanchester)

Nick Alexander :nalexander [he/him]

Comment 11

•

4 years ago

(In reply to Andreas Tolfsen (:ato) from comment #10)
> Of course I should have done a clobber first.  After the clobber, I get a
> different error that I’m not sure is related to this bug?
> 
> https://gist.github.com/andreastt/742335c6b460b8211dde

https://gist.github.com/andreastt/742335c6b460b8211dde#file-gistfile1-txt-L271 is pretty clear:

 0:14.32 Try running |hg pushlogsync|; see https://developer.mozilla.org/en-US/docs/Artifact_builds

It looks like something has gone wrong with your configuration, can you follow these instructions and make sure you're locally configured correctly?

Chris Manchester (:chmanchester)

Assignee

Comment 12

•

4 years ago

Definitely try `hg pushlogsync` before proceeding (I recall needing to do this). Note also this isn't going to work as expected unless you're developing against fx-team (bug 1240667 tracks removing that requirement).

Flags: needinfo?(cmanchester)

Pulsebot

Comment 13

•

4 years ago

https://hg.mozilla.org/integration/mozilla-inbound/rev/a1c1d5cf87ce

Andreas Tolfsen 「:ato」

Reporter

Comment 14

•

4 years ago

nalexander, chmanchester: I should have read the earlier error message more carefully.  Running `hg pushlogsync` as suggested first resolves that issue.

With this change I’m able to do artifact-based builds on Linux, which is great news!  Thanks!

Carsten Book [:Tomcat]

Comment 15

•

4 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/a1c1d5cf87ce

Status: ASSIGNED → RESOLVED

Closed: 4 years ago

status-firefox46: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla46

BMO Automation

Updated

•

2 years ago

Product: Core → Firefox Build System

Bugzilla

No attribute PROTOCOL_SSLv3 error for artifact builds on Linux

Categories

(Firefox Build System :: General, defect)

Tracking

(firefox46 fixed)

People

(Reporter: ato, Assigned: chmanchester)

References

(Blocks 1 open bug)

Details

Crash Data

Security

(public)

User Story

Attachments

(1 file)

Description

Comment 1

Comment 2

Comment 3

Updated

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Comment 9

Comment 10

Comment 11

Comment 12

Comment 13

Comment 14

Comment 15

Updated