Closed Bug 1240530 Opened 4 years ago Closed 4 years ago

No attribute PROTOCOL_SSLv3 error for artifact builds on Linux

Categories

(Firefox Build System :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(firefox46 fixed)

RESOLVED FIXED
mozilla46
Tracking Status
firefox46 --- fixed

People

(Reporter: ato, Assigned: chmanchester)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

Trying to compile artifact build on Linux yields an error:

Output:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

% python --version
Python 2.7.9

mozconfig:

  mk_add_options AUTOCLOBBER=1

  ac_add_options --with-ccache
  ac_add_options --enable-rust

  ac_add_options --enable-application=browser
  ac_add_options --enable-artifact-builds

  mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../build/artifact-desktop-release
This might be an upstream bug in requests/urllib3.

Another workaround is to install Python 2.7.9+ to get a Python distribution with proper security support. I recommend https://github.com/yyuu/pyenv for installing Python in userland.
Just noticed from my output of an artifact build that we're installing requests-2.4.3. This is an ancient version that has known security vulnerabilities. It is also almost certainly not compatible with Python < 2.7.9 and the source of this bug.
Component: mach → Build Config
We have requests 2.5.1 vendored in m-c under python/requests. It should be upgraded to latest and `mach artifact` should use it.
We might just bump the taskcluster client version, which may have advanced.  I haven't checked in a long time.
(In reply to Nick Alexander :nalexander from comment #4)
> We might just bump the taskcluster client version, which may have advanced. 
> I haven't checked in a long time.

This fixes the issue for me locally (as does |./mach python ./mach build|) by picking up the in-tree requests, which is apparently recent enough to avoid this issue.
Currently |./mach artifact| installs an old version of the taskcluster client,
which installs an old version of requests that's incompatible with commonly
installed python versions. This bumps to a version of taskcluster client that
accepts and requests version < 3, so we pick up the in-tree version instead of
installing 2.4.3.

Review commit: https://reviewboard.mozilla.org/r/31525/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/31525/
Attachment #8709722 - Flags: review?(nalexander)
ato, can you confirm the fix in comment 6 is effective?
Flags: needinfo?(ato)
Comment on attachment 8709722 [details]
MozReview Request: Bug 1240530 - Bump the taskcluster version installed by |./mach artifact| to avoid installing an out-dated requests. r=nalexander

https://reviewboard.mozilla.org/r/31525/#review28217

Code is fine.  Ship it if you're confident with your testing.
Attachment #8709722 - Flags: review?(nalexander) → review+
With the change applied, it hangs with the following stacktrace:

% ./mach build
 0:00.15 Running |mach artifact install|.
 0:00.15 /home/ato/1/gecko/src/mach artifact install
New python executable in /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7
Not overwriting existing python script /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python (you must use /home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/bin/python2.7)
Installing setuptools, pip...done.
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil

Error running mach:

    ['artifact', 'install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'PROTOCOL_SSLv3'

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1498, in artifact_install
    artifacts = self._make_artifacts(tree=tree, job=job)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 1465, in _make_artifacts
    from mozbuild.artifacts import Artifacts
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/artifacts.py", line 59, in <module>
    import taskcluster
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/__init__.py", line 5, in <module>
    import client
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/taskcluster/client.py", line 15, in <module>
    import requests
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/home/ato/1/gecko/src/build/mach_bootstrap.py", line 340, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/ato/1/gecko/build/artifact-desktop-release/_virtualenv/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 73, in <module>
    ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: [u'/home/ato/1/gecko/src/mach', u'artifact', u'install']

  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 411, in build
    self._run_mach_artifact_install()
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/mach_commands.py", line 638, in _run_mach_artifact_install
    pass_thru=True, ensure_exit_code=True)
  File "/home/ato/1/gecko/src/python/mozbuild/mozbuild/base.py", line 642, in _run_command_in_srcdir
    return self.run_process(cwd=self.topsrcdir, **args)
  File "/home/ato/1/gecko/src/python/mach/mach/mixin/process.py", line 147, in run_process
    raise Exception('Process executed with non-0 exit code %d: %s' % (status, args))

And when I interrupt it with SIGINT, I get the following:

^?Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ato/1/gecko/src/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 86, in _collect
    while not pipe.poll(sleep_interval):
KeyboardInterrupt
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/python2.7/multiprocessing/util.py", line 325, in _exit_function
    p.join()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
    return self.poll(0)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Assignee: nobody → cmanchester
Status: NEW → ASSIGNED
Flags: needinfo?(ato)
Of course I should have done a clobber first.  After the clobber, I get a different error that I’m not sure is related to this bug?

https://gist.github.com/andreastt/742335c6b460b8211dde
Flags: needinfo?(cmanchester)
(In reply to Andreas Tolfsen (:ato) from comment #10)
> Of course I should have done a clobber first.  After the clobber, I get a
> different error that I’m not sure is related to this bug?
> 
> https://gist.github.com/andreastt/742335c6b460b8211dde

https://gist.github.com/andreastt/742335c6b460b8211dde#file-gistfile1-txt-L271 is pretty clear:

 0:14.32 Try running |hg pushlogsync|; see https://developer.mozilla.org/en-US/docs/Artifact_builds

It looks like something has gone wrong with your configuration, can you follow these instructions and make sure you're locally configured correctly?
Definitely try `hg pushlogsync` before proceeding (I recall needing to do this). Note also this isn't going to work as expected unless you're developing against fx-team (bug 1240667 tracks removing that requirement).
Flags: needinfo?(cmanchester)
nalexander, chmanchester: I should have read the earlier error message more carefully.  Running `hg pushlogsync` as suggested first resolves that issue.

With this change I’m able to do artifact-based builds on Linux, which is great news!  Thanks!
https://hg.mozilla.org/mozilla-central/rev/a1c1d5cf87ce
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.