1396356 - Both buildbotcustom and buildbot-configs tests are failing: AttributeError: 'module' object has no attribute 'SSL_ST_INIT'

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Discovered while landing bug 1384482. Even after I backed my patches out, tests are still failing with this library error.

BB config:
> Traceback (most recent call last):
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 113, in dump_master
>     c = loadMaster(path)
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
>     execfile(path, g, g)
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/test-output/bm106-tests1-macosx/master.cfg", line 7, in <module>
>     import buildbotcustom.misc
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/buildbotcustom/misc.py", line 17, in <module>
>     from buildbot.scheduler import Nightly, Scheduler, Triggerable
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/scheduler.py", line 2, in <module>
>     from buildbot.schedulers.basic import Scheduler, AnyBranchScheduler, Dependent
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/schedulers/basic.py", line 43, in <module>
>     from buildbot.status.builder import SUCCESS, WARNINGS
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/d-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 148, in main
>     dump = dump_master(args.masters[0])
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 113, in dump_master
>     c = loadMaster(path)
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/braindump/buildbot-related/dump_master_json.py", line 26, in loadMaster
>     execfile(path, g, g)
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/test-output/bm106-tests1-macosx/master.cfg", line 7, in <module>
>     import buildbotcustom.misc
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/buildbotcustom/misc.py", line 17, in <module>
>     from buildbot.scheduler import Nightly, Scheduler, Triggerable
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/scheduler.py", line 2, in <module>
>     from buildbot.schedulers.basic import Scheduler, AnyBranchScheduler, Dependent
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/schedulers/basic.py", line 43, in <module>
>     from buildbot.status.builder import SUCCESS, WARNINGS
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/buildbot-0.8.2_hg_7f288e5082f5_production_0.8-py2.7.egg/buildbot/status/builder.py", line 6, in <module>
>     from twisted.internet import reactor, defer, threads
>   Fileleng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 22, in <module>
>     from twisted.internet import tcp, udp
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/twisted/internet/tcp.py", line 24, in <module>
>     from OpenSSL import SSL
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
>     from OpenSSL import rand, crypto, SSL
>   File "/home/travis/build/mozilla-releng/build-buildbot-configs/.tox/py27/lib/python2.7/site-packages/OpenSSL/SSL.py", line 105, in <module>
>     SSL_ST_INIT = _lib.SSL_ST_INIT
> AttributeError: 'module' object has no attribute 'SSL_ST_INIT'

BB custom:
> Traceback (most recent call last):
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/bin/trial", line 22, in <module>
>     run()
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/scripts/trial.py", line 346, in run
>     test_result = trialRunner.run(suite)
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/trial/runner.py", line 796, in run
>     return self._runWithoutDecoration(test)
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/trial/runner.py", line 825, in _runWithoutDecoration
>     run()
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/trial/runner.py", line 820, in <lambda>
>     run = lambda: suite.run(result)
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/trial/runner.py", line 291, in run
>     self._bail()
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/trial/runner.py", line 276, in _bail
>     from twisted.internet import reactor
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/internet/reactor.py", line 37, in <module>
>     from twisted.internet import selectreactor
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 21, in <module>
>     from twisted.internet import posixbase
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 22, in <module>
>     from twisted.internet import tcp, udp
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/twisted/internet/tcp.py", line 24, in <module>
>     from OpenSSL import SSL
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
>     from OpenSSL import rand, crypto, SSL
>   File "/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/lib/python2.7/site-packages/OpenSSL/SSL.py", line 105, in <module>
>     SSL_ST_INIT = _lib.SSL_ST_INIT
> AttributeError: 'module' object has no attribute 'SSL_ST_INIT'
> ERROR: InvocationError: '/home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/bin/coverage run --omit=.tox/* --branch /home/travis/build/mozilla-releng/build-buildbotcustom/.tox/py27/bin/trial --reporter=verbose buildbotcustom.test'


I don't know what changed between now and 2 days ago, when bug 1330412 landed on BB config (and passed).

Nick, do you if this error happens commonly?

Comment 1

[Johan Lorenzo [:jlorenzo]]

Aki pointed out deps have changed: I found the following changes:

> cryptography==1.8.1 -> 2.0.3 
> idna==2.5 -> 2.6
> kombu==4.0.2 -> 4.1.0
> py==1.4.33 -> 1.4.34
> pycparser==2.17 -> 2.18

Interestingly, pyOpenSSL remained at 0.14.

After looking at pyOpenSssl, I noticed they just bind cryptography[1]. Looks like we should just revert cryptography back to 1.8.1 (or upgrade it to 1.8.2/1.9[2])

[1] https://github.com/pyca/pyopenssl/blob/0.14/OpenSSL/_util.py#L6
[2] https://github.com/pyca/cryptography/releases

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1396356 - Pin dependecies installed by Travis a=bustage

Review commit: https://reviewboard.mozilla.org/r/175744/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/175744/

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1396356 - Pin dependecies installed by Travis a=bustage

Review commit: https://reviewboard.mozilla.org/r/175748/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/175748/

Comment 4

[Johan Lorenzo [:jlorenzo]]

Landed: 
* https://hg.mozilla.org/build/buildbot-configs/rev/77293433edd702d367323c5077c4930a2868688e
* without buildbot pinned https://hg.mozilla.org/build/buildbot-configs/rev/c3a6d603ffa7e79cf9974c614a1846f000ee02a0
* https://hg.mozilla.org/build/buildbotcustom/rev/9a34d8f195eed01ac4b06f4b171195564d7a9001

Builds are back to green:
* https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/271334579
* https://travis-ci.org/mozilla-releng/build-buildbotcustom/builds/271335112

Thanks Aki for pointing the problem out!