1373338 - Fix gevent deps for osx signingserver

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[:dhouse]

Gevent compiling fails to find libevent headers on new osx signing servers in mdc1. libevent is in the puppet manifests; is there a problem with the ordering or with the libevent install?

Comment 1

[:dhouse]

mac-depsigning2.srv.releng.mdc1.mozilla.com:/Users/cltsign/.pip/pip.log
[...]
In file included from gevent/core.c:225:

gevent/libevent.h:9:10: fatal error: 'event.h' file not found

[...]
    status = self.run(options, args)
  File "/builds/signing/dep-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/commands/install.py", line 283, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/builds/signing/dep-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/req.py", line 1435, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/builds/signing/dep-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/req.py", line 706, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "/builds/signing/dep-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/util.py", line 697, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command /builds/signing/dep-key-signing-server/bin/python2.7 -c "import setuptools, tokenize;__file__='/builds/signing/dep-key-signing-server/build/gevent/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-PdFpSv-record/install-record.txt --single-version-externally-managed --compile --install-headers /builds/signing/dep-key-signing-server/include/site/python2.7 failed with error code 1 in /builds/signing/dep-key-signing-server/build/gevent

Comment 2

[:dhouse]

libevent is installed. maybe pip doesn't have the right include path:
[[e-mail] ~]$ ls -la /usr/local/include/*even*
-rw-rw-r--  1 root  wheel  2760 Jul 31  2014 /usr/local/include/event.h
[...]
/** @file event.h

  A library for writing event-driven network servers.

  The <event.h> header is deprecated in Libevent 2.0 and later; please
  use <event2/event.h> instead.  Depending on what functionality you
  need, you may also want to include more of the other event2/
  headers.
[...]

Comment 3

[:dhouse]

Manual install works when explicitly setting the path for headers and libs:

(dep-key-signing-server)[[e-mail] dep-key-signing-server]$ CFLAGS="-I /usr/local/include -L /usr/local/lib" pip install --no-deps --no-index  --find-links=http://releng-puppet2.srv.releng.mdc1.mozilla.com/python/packages  --find-links=http://releng-puppet1.srv.releng.mdc1.mozilla.com/python/packages  --find-links=http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages  --find-links=http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages  --find-links=http://releng-puppet1.srv.releng.use1.mozilla.com/python/packages  --find-links=http://releng-puppet1.srv.releng.usw2.mozilla.com/python/packages  gevent==0.13.6
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking gevent==0.13.6
  http://pypi.pvt.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pvt.build.mozilla.org has it available
[...]
    gcc -bundle -bundle_loader /tools/python27/bin/python2.7 -I /usr/local/include -L /usr/local/lib build/temp.macosx-10.10-x86_64-2.7/gevent/core.o -levent -o build/lib.macosx-10.10-x86_64-2.7/gevent/core.so
    Linking /builds/signing/dep-key-signing-server/build/gevent/build/lib.macosx-10.10-x86_64-2.7/gevent/core.so to /builds/signing/dep-key-signing-server/build/gevent/gevent/core.so
    
Successfully installed gevent
Cleaning up...

Comment 4

[:dhouse]

Comparing with mac-v2-signing10, depsigning2 had the same xcode and clang versions and the same install root for libevent (/usr/local/)

With re-running puppet and rebooting, three or more times, the gevent install passed on mac-depsigning1. (hammer, hammer) I tried the same on #3, but ran into failures finding/checking the cert so I manually installed gevent there like on #2.

Comment 5

[:dhouse]

Attachment: bug1373338_add-libevent-after-xcode.patch

Adding libevent in after the xcode include in the signingserver/instance.pp appears to be the fix. So I think that the paths may get changed with the xcode install or there is some other order of events issue for depsigning (CentOS also has the libevent include duplicated within the matching block. So we may not need it at the outer level, but I'm leaving it there for now): 

[[e-mail] ~]$ source /builds/signing/dep-signing-server/bin/activate
(dep-signing-server)[[e-mail] ~]$ pip freeze
[...]
gevent==0.13.6
[...]

Comment 6

[:dhouse]

remote:   https://hg.mozilla.org/build/puppet/rev/05cb20df45f97dac4643be8ab631da09b398ee1c
Travis passed
pushed to production:
remote:   https://hg.mozilla.org/build/puppet/rev/360314976ae60b25a91b65ab35a3c38395c114c4

Comment 7

[Amy Rich [:arr] [:arich]]

It looks like mac-depsigning3.srv.releng.mdc1.mozilla.com never puppetized correctly. I just tried to puppetize it and this error still comes up.

Comment 8

[:dhouse]

A different error than before is appearing now:

HTTP error 403 while getting http://pypi.pvt.build.mozilla.org/pub/gevent-0.13.6.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/)

Comment 9

[Aki Sasaki (not active)]

We have it in our python2 repo: http://releng-puppet2.srv.releng.scl3.mozilla.com/python/packages/gevent-0.13.6.tar.gz
I don't know how that translates to pypi.pvt.build.mozilla.org.

Comment 10

[Amy Rich [:arr] [:arich]]

If a curl is failing, I wonder if we're hitting bug 1387039. If you can't curl the package, please NI jbarnell for a flow fix.

Comment 11

[:dhouse]

Using https://pypi.pvt.b.m.o gets past the 403.

Comment 12

[:dhouse]

jbarnell, I'm seeing traffic on port 80 blocked from release engineering signing servers in mdc1. Could you add/adjust the flows/filtering to allow port 80 traffic from the releng vlans? (to only mozilla.org would be fine)

Comment 13

[:dhouse]

I think it is the pypi server refusing the request where it was not before. In same cases, the package was being pulled from the puppet masters instead of the pypi mirror, but when requested from pypi it consistently gets a 403.

Comment 14

[:dhouse]

This problem appeared because new python virtualenv's were set up for dep, nightly, and rel. Puppet is applying the wrong module to these mac-depsigning servers (signing instead of depsigning) and so as of Aug. 4th all three have the nightly and rel environments created. They did not fail on the gevent installs on #1 and #2. #3 appears to still have libevent incorrectly set up as I had to manually pip install with CFLAGS for the correct paths where the libevent headers and libs are located:

```
(nightly-key-signing-server)[[e-mail] tmp]$ /builds/signing/nightly-key-signing-server/bin/pip install --no-deps --no-index  gevent==0.13.6
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking gevent==0.13.6
  http://pypi.pvt.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pvt.build.mozilla.org has it available
  http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
  HTTP error 403 while getting http://pypi.pvt.build.mozilla.org/pub/gevent-0.13.6.tar.gz (from http://pypi.pvt.build.mozilla.org/pub/)


# switched to https in pip.conf
(nightly-key-signing-server)[[e-mail] tmp]$ pip install gevent
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking gevent
  http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
  Downloading gevent-0.13.8.tar.gz (300kB): 300kB downloaded
  Running setup.py (path:/builds/signing/nightly-key-signing-server/build/gevent/setup.py) egg_info for package gevent
[...]
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/tools/python27/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.10-x86_64-2.7/gevent/core.o

In file included from gevent/core.c:225:

gevent/libevent.h:9:10: fatal error: 'event.h' file not found


# adding correct paths
(nightly-key-signing-server)[[e-mail] tmp]$ CFLAGS="-I /usr/local/include -L /usr/local/lib" /builds/signing/nightly-key-signing-server/bin/pip install --no-deps --no-index  gevent==0.13.6
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking gevent==0.13.6
  http://pypi.pub.build.mozilla.org/pub uses an insecure transport scheme (http). Consider using https if pypi.pub.build.mozilla.org has it available
  Downloading gevent-0.13.6.tar.gz (289kB): 289kB downloaded
  Running setup.py (path:/builds/signing/nightly-key-signing-server/build/gevent/setup.py) egg_info for package gevent
[...]
    gcc -bundle -bundle_loader /tools/python27/bin/python2.7 -I /usr/local/include -L /usr/local/lib build/temp.macosx-10.10-x86_64-2.7/gevent/core.o -levent -o build/lib.macosx-10.10-x86_64-2.7/gevent/core.so
    Linking /builds/signing/nightly-key-signing-server/build/gevent/build/lib.macosx-10.10-x86_64-2.7/gevent/core.so to /builds/signing/nightly-key-signing-server/build/gevent/gevent/core.so
    
Successfully installed gevent
```

Comment 15

[:dhouse]

I tested on mac-depsigning1 and also get the 403 against http://pypi.pvt.build.mozilla.org. It looks like #1 and #2 used the gevent package from pip cache.

Comment 16

[:dhouse]

Confirmed 403 with a pip search on #1:
```
(nightly-key-signing-server)[[e-mail] ~]$ pip search --index=http://pypi.pvt.build.mozilla.org pip
Exception:
Traceback (most recent call last):
  File "/builds/signing/nightly-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/builds/signing/nightly-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/commands/search.py", line 40, in run
    pypi_hits = self.search(query, index_url)
  File "/builds/signing/nightly-key-signing-server/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/commands/search.py", line 54, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/tools/python27/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/tools/python27/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/tools/python27/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/tools/python27/lib/python2.7/xmlrpclib.py", line 1312, in single_request
    response.msg,
ProtocolError: <ProtocolError for pypi.pvt.build.mozilla.org/RPC2: 403 Forbidden>

Storing debug log for failure in /Users/cltsign/.pip/pip.log
```

Comment 17

[:dhouse]

Looking in the logs for the failures on the pypi webserver side:
Aug 13 01:31:01 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "GET /pub/ HTTP/1.1" 403 206 "-" "-"
Aug 13 05:31:01 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "GET /pub/ HTTP/1.1" 403 206 "-" "-"
Aug 13 15:31:01 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "GET /pub/ HTTP/1.1" 403 206 "-" "-"
Aug 13 17:44:26 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "GET / HTTP/1.1" 403 220 "-" "-"
Aug 14 14:21:48 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "POST /RPC2 HTTP/1.1" 403 206 "-" "-"
Aug 14 14:36:29 web2.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "POST /RPC2 HTTP/1.1" 403 206 "-" "-"
Aug 14 14:39:36 web1.releng.webapp.scl3.mozilla.com httpd: 10.22.81.211 pypi.pvt.build.mozilla.org "POST /RPC2 HTTP/1.1" 403 206 "-" "-" 

But then there is a strange error on the web server:
[[e-mail]3 ~]$ sudo tail -f /var/log/httpd/pypi.pvt.build.mozilla.org/error_log
[Mon Aug 14 14:14:09 2017] [error] [client 10.22.81.211] Symbolic link not allowed or link target not accessible: /mnt/netapp/relengweb/pypi/pub/talos-0.0.tar.gz
[Mon Aug 14 14:44:06 2017] [error] [client 10.22.81.211] Symbolic link not allowed or link target not accessible: /mnt/netapp/relengweb/pypi/pub/talos-0.0.tar.gz

So it may not be blocking the host as I was guessing (80 only allowed within releng network

Comment 18

[Aki Sasaki (not active)]

15:47 <dividehex> I've also noticed that the /signing\d+\.srv\.releng\.(mdc1|scl3)\.mozilla\.com/ node def is getting picked up by the mac-depsigning servers in mdc1
15:48 <aki> boo
15:48 <aki> maybe a /^signing ... ?
15:48 <dividehex> me might need to add a ^ to the front
15:48 <aki> gmta
15:48 <aki> that's probably why mac-depsigning is throwing errors :)
15:48 <dividehex> yep
15:49 <dividehex> I'll fix that

Comment 19

[:dhouse]

Thank you Aki! That explains the wrong modules being applied to depsigning.

Comment 20

[:dhouse]

I've logged into the mac-depsigningN servers in mdc1 and removed the unwanted /build/signing/*-key-signing-server directories. Each now has only the expected /build/signing/dep-signing-server.