Closed
Bug 1114965
Opened 11 years ago
Closed 11 years ago
Trees Closed due to self-serve down
Categories
(Infrastructure & Operations Graveyard :: CIDuty, task)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: cbook, Unassigned)
Details
Self serve down and so no builds are starting.
05:09 < nagios-releng> Tue 05:09:42 PST [4660] buildapi.pvt.build.mozilla.org:http - /buildapi/self-serve/jobs is CRITICAL: CRITICAL - Socket timeout after 10 seconds (http://m.mozilla.org/http+-+/buildapi/self-serve/jobs)
affects all trees so all trees are closed
|
||
Comment 1•11 years ago
|
||
Looking into this now...
|
|
||
Comment 2•11 years ago
|
||
The url that is not available is https://secure.pub.build.mozilla.org/buildapi/self-serve/jobs - investigating where this app is hosted...
|
|
||
Comment 3•11 years ago
|
||
This seems to be hosted on
web1.releng.webapp.scl3.mozilla.com
web2.releng.webapp.scl3.mozilla.com
Looking on these hosts now.
|
|
||
Comment 4•11 years ago
|
||
logs appear to be in:
/var/log/httpd/buildapi.pvt.build.mozilla.org/
Looking at them now
|
|
||
Comment 5•11 years ago
|
||
web1 doesn't seem to be getting any traffic:
[pmoore@web1.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$ tail -10 access_log
10.22.81.211 - - [23/Dec/2014:05:08:03 -0800] "GET /buildapi/recent/bld-linux64-spot-034?format=json&numbuilds=1 HTTP/1.1" 500 541 "-" "Python-urllib/2.6"
10.22.81.211 - - [23/Dec/2014:05:08:31 -0800] "GET /buildapi/recent/bld-linux64-spot-136?format=json&numbuilds=1 HTTP/1.1" 500 541 "-" "Python-urllib/2.6"
10.22.81.211 - - [23/Dec/2014:05:09:01 -0800] "GET /buildapi/running?format=json HTTP/1.1" 500 541 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
10.22.81.211 - - [23/Dec/2014:05:09:09 -0800] "GET /buildapi/running?format=json HTTP/1.1" 500 541 "-" "python-requests/2.3.0 CPython/2.6.6 Linux/2.6.32-220.13.1.el6.x86_64"
10.22.81.211 - - [23/Dec/2014:05:09:10 -0800] "GET /buildapi/running?format=json HTTP/1.1" 500 541 "-" "python-requests/2.3.0 CPython/2.6.6 Linux/2.6.32-220.13.1.el6.x86_64"
10.22.81.211 - - [23/Dec/2014:05:09:25 -0800] "GET /buildapi/recent/b-linux64-hp-0029?format=json&numbuilds=1 HTTP/1.1" 500 541 "-" "Python-urllib/2.6"
10.22.81.211 - - [23/Dec/2014:05:09:28 -0800] "GET /buildapi/recent/bld-linux64-spot-038?format=json&numbuilds=1 HTTP/1.1" 500 541 "-" "Python-urllib/2.6"
10.22.81.211 - - [23/Dec/2014:05:09:31 -0800] "GET /buildapi/pending?format=json HTTP/1.1" 500 541 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"
10.22.81.211 - - [23/Dec/2014:04:48:04 -0800] "GET /buildapi/recent/bld-linux64-spot-136?format=json&numbuilds=1 HTTP/1.1" 200 254 "-" "Python-urllib/2.6"
10.22.81.211 - - [23/Dec/2014:05:09:47 -0800] "GET /buildapi/recent/bld-linux64-spot-028?format=json&numbuilds=1 HTTP/1.1" 500 541 "-" "Python-urllib/2.6"
[pmoore@web1.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$ tail -10 error_log
[Tue Dec 23 05:09:01 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:09:03 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:09:31 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:01 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:10 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:10 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:25 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:29 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:10:31 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[Tue Dec 23 05:18:26 2014] [error] [client 10.22.81.211] Script timed out before returning headers: buildapi.wsgi
[pmoore@web1.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$ date
Tue Dec 23 05:33:03 PST 2014
[pmoore@web1.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$
|
|
||
Comment 6•11 years ago
|
||
The last entry for /self-serve/jobs was 37 mins ago:
10.22.81.211 - - [23/Dec/2014:04:59:27 -0800] "GET /buildapi/self-serve/jobs HTTP/1.1" 500 541 "-" "check_http/v1.4.16 (nagios-plugins 1.4.16)"
|
|
||
Comment 7•11 years ago
|
||
So it looks like requests are not hitting apache, so it is not a buildapi performance problem, rather that requests are not reaching it.
I will double check web2.releng.webapp.scl3.mozilla.com to see if we have the same issue there.
|
|
||
Comment 8•11 years ago
|
||
So, I tried curling locally to check all is ok, but was not able to:
[pmoore@web1.releng.webapp.scl3 mozilla]$ cat /etc/httpd/mozilla/domains/buildapi.pvt.build.mozilla.org.conf
# *:80 is cleartext allowed from releng network only
<VirtualHost *:80>
ServerName buildapi.pvt.build.mozilla.org
DocumentRoot /mnt/netapp/relengweb/home
# identify when we're getting SSL handled by the ZLB (which we always should be)
SetEnvIf SSLSessionID .+ HTTPS=on
RewriteEngine On
Include generated/allow-build-net-zlb.conf
# redirect / -> /buildapi
RewriteRule ^/$ /buildapi/ [L,R=301]
#
# /buildapi
#
Include generated/buildapi.conf
#
# logging
#
SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" is-forwarded
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog "|/usr/sbin/rotatelogs -L /var/log/httpd/buildapi.pvt.build.mozilla.org/access_log /var/log/httpd/buildapi.pvt.build.mozilla.org/access_%Y-%m-%d-%H 3600 -0" combined
ErrorLog "|/usr/sbin/rotatelogs -L /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log_%Y-%m-%d-%H 3600 -0"
</VirtualHost>
[pmoore@web1.releng.webapp.scl3 mozilla]$ curl http://10.22.81.88/buildapi/self-serve/jobs
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /buildapi/self-serve/jobs
on this server.</p>
</body></html>
[pmoore@web1.releng.webapp.scl3 mozilla]$ curl http://localhost/buildapi/self-serve/jobs
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /buildapi/self-serve/jobs
on this server.</p>
</body></html>
[pmoore@web1.releng.webapp.scl3 mozilla]$
But at least it shows the service is running on port 80.
Now checking web2.releng.webapp.scl3.mozilla.com...
Will also call in help from others.
|
|
||
Comment 9•11 years ago
|
||
Same story on web2 as on web1.
I've checked the buildapi process seems to be running, e.g.:
[pmoore@web2.releng.webapp.scl3 ~]$ ps -ef | grep buildapi
pmoore 6495 4879 0 05:51 pts/0 00:00:00 grep buildapi
root 29872 23701 0 Dec22 ? 00:00:00 /usr/sbin/rotatelogs -L /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log /var/log/httpd/buildapi.pvt.build.mozilla.org/error_log_%Y-%m-%d-%H 3600 -0
root 29907 23701 0 Dec22 ? 00:00:00 /usr/sbin/rotatelogs -L /var/log/httpd/buildapi.pvt.build.mozilla.org/access_log /var/log/httpd/buildapi.pvt.build.mozilla.org/access_%Y-%m-%d-%H 3600 -0
buildapi 29920 23701 0 Dec22 ? 00:01:21 buildapi
buildapi 29921 23701 0 Dec22 ? 00:01:23 buildapi
buildapi 29922 23701 0 Dec22 ? 00:01:26 buildapi
buildapi 29923 23701 0 Dec22 ? 00:01:27 buildapi
buildapi 29924 23701 0 Dec22 ? 00:01:25 buildapi
buildapi 29925 23701 0 Dec22 ? 00:01:32 buildapi
[pmoore@web2.releng.webapp.scl3 ~]$ sudo cat /proc/29925/cmdline | xargs -0 -n1 echo
buildapi
[pmoore@web2.releng.webapp.scl3 ~]$ sudo cat /proc/29925/environ | xargs -0 -n1 echo
TERM=xterm-256color
PATH=/sbin:/usr/sbin:/bin:/usr/bin
PWD=/
LANG=C
SHLVL=2
NEWRELIC_PYTHON_INI_FILE=/etc/newrelic/python.ini
_=/usr/sbin/httpd
[pmoore@web2.releng.webapp.scl3 ~]$
|
|
||
Comment 10•11 years ago
|
||
can't work out where buildapi is installed...
[pmoore@web2.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$ sudo su buildapi
sh-4.1$ bash
bash-4.1$ PATH=/sbin:/usr/sbin:/bin:/usr/bin
bash-4.1$ export PATH
bash-4.1$ which buildapi
which: no buildapi in (/sbin:/usr/sbin:/bin:/usr/bin)
bash-4.1$ whoami
buildapi
bash-4.1$ cd /
bash-4.1$ which buildapi
which: no buildapi in (/sbin:/usr/sbin:/bin:/usr/bin)
bash-4.1$
|
|
||
Comment 11•11 years ago
|
||
[pmoore@web2.releng.webapp.scl3 buildapi.pvt.build.mozilla.org]$ sudo lsof | grep 29920 | grep REG
httpd 29920 buildapi txt REG 8,3 355232 2239631 /usr/sbin/httpd
httpd 29920 buildapi mem REG 8,3 1598280 2226974 /usr/lib64/mysql/libmysqlclient_r.so.16.0.0
httpd 29920 buildapi mem REG 8,3 145773 1711419 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/_mysql.so
httpd 29920 buildapi mem REG 8,3 29904 2244438 /usr/lib64/python2.7/lib-dynload/_multiprocessing.so
httpd 29920 buildapi mem REG 8,3 36208 2244449 /usr/lib64/python2.7/lib-dynload/bz2.so
httpd 29920 buildapi mem REG 8,3 166208 2244424 /usr/lib64/python2.7/lib-dynload/_ctypes.so
httpd 29920 buildapi mem REG 8,3 151012 12488 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/simplejson/_speedups.so
httpd 29920 buildapi mem REG 8,3 41085 12531 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/sqlalchemy/cresultproxy.so
httpd 29920 buildapi mem REG 8,3 19870 12532 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/sqlalchemy/cutils.so
httpd 29920 buildapi mem REG 8,3 32632 12530 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/sqlalchemy/cprocessors.so
httpd 29920 buildapi mem REG 8,3 229480 2244467 /usr/lib64/python2.7/lib-dynload/pyexpat.so
httpd 29920 buildapi mem REG 8,3 24424 2244470 /usr/lib64/python2.7/lib-dynload/select.so
httpd 29920 buildapi mem REG 8,3 25192 2244474 /usr/lib64/python2.7/lib-dynload/termios.so
httpd 29920 buildapi mem REG 8,3 23853 11600 /data/www/buildapi/virtualenv/lib/python2.7/site-packages/markupsafe/_speedups.so
httpd 29920 buildapi mem REG 8,3 21608 2244435 /usr/lib64/python2.7/lib-dynload/_locale.so
httpd 29920 buildapi mem REG 8,3 42552 2244446 /usr/lib64/python2.7/lib-dynload/array.so
httpd 29920 buildapi mem REG 8,3 89512 2244454 /usr/lib64/python2.7/lib-dynload/datetime.so
httpd 29920 buildapi mem REG 8,3 12456 2244455 /usr/lib64/python2.7/lib-dynload/dbm.so
httpd 29920 buildapi mem REG 8,3 181432 2244415 /usr/lib64/python2.7/lib-dynload/_bsddb.so
httpd 29920 buildapi mem REG 8,3 57792 2244466 /usr/lib64/python2.7/lib-dynload/parser.so
httpd 29920 buildapi mem REG 8,3 26368 2244472 /usr/lib64/python2.7/lib-dynload/strop.so
httpd 29920 buildapi mem REG 8,3 23968 2244477 /usr/lib64/python2.7/lib-dynload/zlib.so
httpd 29920 buildapi mem REG 8,3 155248 2244433 /usr/lib64/python2.7/lib-dynload/_io.so
httpd 29920 buildapi mem REG 8,3 9232 2244458 /usr/lib64/python2.7/lib-dynload/grp.so
httpd 29920 buildapi mem REG 8,3 22240 2244431 /usr/lib64/python2.7/lib-dynload/_heapq.so
httpd 29920 buildapi mem REG 8,3 9904 2244414 /usr/lib64/python2.7/lib-dynload/_bisect.so
httpd 29920 buildapi mem REG 8,3 60872 2244459 /usr/lib64/python2.7/lib-dynload/itertools.so
httpd 29920 buildapi mem REG 8,3 41968 2244464 /usr/lib64/python2.7/lib-dynload/operator.so
httpd 29920 buildapi mem REG 8,3 33512 2244422 /usr/lib64/python2.7/lib-dynload/_collections.so
httpd 29920 buildapi mem REG 8,3 14624 2244456 /usr/lib64/python2.7/lib-dynload/fcntl.so
httpd 29920 buildapi mem REG 8,3 13256 2244439 /usr/lib64/python2.7/lib-dynload/_random.so
httpd 29920 buildapi mem REG 8,3 17160 2244430 /usr/lib64/python2.7/lib-dynload/_hashlib.so
httpd 29920 buildapi mem REG 8,3 21040 2244448 /usr/lib64/python2.7/lib-dynload/binascii.so
httpd 29920 buildapi mem REG 8,3 36736 2244461 /usr/lib64/python2.7/lib-dynload/math.so
httpd 29920 buildapi mem REG 8,3 37808 2244443 /usr/lib64/python2.7/lib-dynload/_struct.so
httpd 29920 buildapi mem REG 8,3 78960 2244450 /usr/lib64/python2.7/lib-dynload/cPickle.so
httpd 29920 buildapi mem REG 8,3 34560 2244442 /usr/lib64/python2.7/lib-dynload/_ssl.so
httpd 29920 buildapi mem REG 8,3 12376 2244429 /usr/lib64/python2.7/lib-dynload/_functools.so
httpd 29920 buildapi mem REG 8,3 61488 2244440 /usr/lib64/python2.7/lib-dynload/_socket.so
httpd 29920 buildapi mem REG 8,3 19792 2244451 /usr/lib64/python2.7/lib-dynload/cStringIO.so
httpd 29920 buildapi mem REG 8,3 21184 2244475 /usr/lib64/python2.7/lib-dynload/time.so
httpd 29920 buildapi mem REG 8,3 83088 2239246 /usr/lib64/php/modules/zip.so
httpd 29920 buildapi mem REG 8,3 388624 2237721 /usr/lib64/libtidy-0.99.so.0.0.0
httpd 29920 buildapi mem REG 8,3 49320 2229752 /usr/lib64/php/modules/tidy.so
httpd 29920 buildapi mem REG 8,3 44368 2245703 /usr/lib64/php/modules/sqlite3.so
httpd 29920 buildapi mem REG 8,3 261392 2239245 /usr/lib64/php/modules/phar.so
httpd 29920 buildapi mem REG 8,3 586592 2225307 /usr/lib64/libsqlite3.so.0.8.6
httpd 29920 buildapi mem REG 8,3 24640 2245702 /usr/lib64/php/modules/pdo_sqlite.so
httpd 29920 buildapi mem REG 8,3 30112 2246281 /usr/lib64/php/modules/pdo_mysql.so
httpd 29920 buildapi mem REG 8,3 102456 2236548 /usr/lib64/php/modules/pdo.so
httpd 29920 buildapi mem REG 8,3 855644 137018 /usr/lib/newrelic-php5/agent/x64/newrelic-20090626.so
httpd 29920 buildapi mem REG 8,3 136832 2246280 /usr/lib64/php/modules/mysqli.so
httpd 29920 buildapi mem REG 8,3 1587712 2229289 /usr/lib64/mysql/libmysqlclient.so.16.0.0
httpd 29920 buildapi mem REG 8,3 54408 2225560 /usr/lib64/php/modules/mysql.so
httpd 29920 buildapi mem REG 8,3 36200 2239244 /usr/lib64/php/modules/json.so
httpd 29920 buildapi mem REG 8,3 1843056 2239243 /usr/lib64/php/modules/fileinfo.so
httpd 29920 buildapi mem REG 8,3 164504 2228385 /usr/lib64/libssh2.so.1.0.1
httpd 29920 buildapi mem REG 8,3 209088 2093225 /lib64/libidn.so.11.6.1
httpd 29920 buildapi mem REG 8,3 346424 2234561 /usr/lib64/libcurl.so.4.1.1
httpd 29920 buildapi mem REG 8,3 66096 2239242 /usr/lib64/php/modules/curl.so
httpd 29920 buildapi mem REG 8,3 9488 2237391 /usr/lib64/apr-util-1/apr_ldap-1.so
httpd 29920 buildapi mem REG 8,3 27424 2093080 /lib64/libnss_dns-2.12.so
httpd 29920 buildapi mem REG 8,3 65928 2093423 /lib64/libnss_files-2.12.so
httpd 29920 buildapi mem REG 8,3 17520 2093143 /lib64/libutil-2.12.so
httpd 29920 buildapi mem REG 8,3 1893808 2228802 /usr/lib64/libpython2.7.so.1.0
httpd 29920 buildapi mem REG 8,3 158928 2245584 /usr/lib64/httpd/modules/python27-mod_wsgi.so
httpd 29920 buildapi mem REG 8,3 12592 2093156 /lib64/libkeyutils.so.1.3
httpd 29920 buildapi mem REG 8,3 46368 2093157 /lib64/libkrb5support.so.0.1
httpd 29920 buildapi mem REG 8,3 1965824 2227457 /usr/lib64/libcrypto.so.1.0.1e
httpd 29920 buildapi mem REG 8,3 444168 2229186 /usr/lib64/libssl.so.1.0.1e
httpd 29920 buildapi mem REG 8,3 17256 2093166 /lib64/libcom_err.so.2.1
httpd 29920 buildapi mem REG 8,3 177520 2093162 /lib64/libk5crypto.so.3.1
httpd 29920 buildapi mem REG 8,3 944712 2093172 /lib64/libkrb5.so.3.3
httpd 29920 buildapi mem REG 8,3 280520 2093180 /lib64/libgssapi_krb5.so.2.2
httpd 29920 buildapi mem REG 8,3 1384424 2231015 /usr/lib64/libxml2.so.2.7.6
httpd 29920 buildapi mem REG 8,3 116368 2093151 /lib64/libnsl-2.12.so
httpd 29920 buildapi mem REG 8,3 69976 2093181 /lib64/libbz2.so.1.0.4
httpd 29920 buildapi mem REG 8,3 376792 2228434 /usr/lib64/libgmp.so.3.5.0
httpd 29920 buildapi mem REG 8,3 3699056 2230829 /usr/lib64/httpd/modules/libphp5.so
httpd 29920 buildapi mem REG 8,3 11184 2230214 /usr/lib64/httpd/modules/mod_cloudflare.so
httpd 29920 buildapi mem REG 8,3 10416 2239627 /usr/lib64/httpd/modules/mod_version.so
httpd 29920 buildapi mem REG 8,3 27312 2238225 /usr/lib64/httpd/modules/mod_cgi.so
httpd 29920 buildapi mem REG 8,3 22992 2238259 /usr/lib64/httpd/modules/mod_disk_cache.so
httpd 29920 buildapi mem REG 8,3 10384 2239623 /usr/lib64/httpd/modules/mod_suexec.so
httpd 29920 buildapi mem REG 8,3 39664 2238186 /usr/lib64/httpd/modules/mod_cache.so
httpd 29920 buildapi mem REG 8,3 14648 2239613 /usr/lib64/httpd/modules/mod_proxy_connect.so
httpd 29920 buildapi mem REG 8,3 39632 2239611 /usr/lib64/httpd/modules/mod_proxy_ajp.so
httpd 29920 buildapi mem REG 8,3 31472 2239615 /usr/lib64/httpd/modules/mod_proxy_http.so
httpd 29920 buildapi mem REG 8,3 35600 2239614 /usr/lib64/httpd/modules/mod_proxy_ftp.so
httpd 29920 buildapi mem REG 8,3 27160 2239612 /usr/lib64/httpd/modules/mod_proxy_balancer.so
httpd 29920 buildapi mem REG 8,3 85328 2239610 /usr/lib64/httpd/modules/mod_proxy.so
httpd 29920 buildapi mem REG 8,3 60464 2239618 /usr/lib64/httpd/modules/mod_rewrite.so
httpd 29920 buildapi mem REG 8,3 14608 2239622 /usr/lib64/httpd/modules/mod_substitute.so
httpd 29920 buildapi mem REG 8,3 14608 2235205 /usr/lib64/httpd/modules/mod_alias.so
httpd 29920 buildapi mem REG 8,3 10448 2239625 /usr/lib64/httpd/modules/mod_userdir.so
httpd 29920 buildapi mem REG 8,3 14608 2239620 /usr/lib64/httpd/modules/mod_speling.so
httpd 29920 buildapi mem REG 8,3 10416 2234742 /usr/lib64/httpd/modules/mod_actions.so
httpd 29920 buildapi mem REG 8,3 10416 2238243 /usr/lib64/httpd/modules/mod_dir.so
httpd 29920 buildapi mem REG 8,3 31384 2239609 /usr/lib64/httpd/modules/mod_negotiation.so
httpd 29920 buildapi mem REG 8,3 10424 2239628 /usr/lib64/httpd/modules/mod_vhost_alias.so
httpd 29920 buildapi mem REG 8,3 52016 2238232 /usr/lib64/httpd/modules/mod_dav_fs.so
httpd 29920 buildapi mem REG 8,3 19032 2239250 /usr/lib64/httpd/modules/mod_info.so
httpd 29920 buildapi mem REG 8,3 35472 2238185 /usr/lib64/httpd/modules/mod_autoindex.so
httpd 29920 buildapi mem REG 8,3 22864 2239621 /usr/lib64/httpd/modules/mod_status.so
httpd 29920 buildapi mem REG 8,3 93328 2238231 /usr/lib64/httpd/modules/mod_dav.so
httpd 29920 buildapi mem REG 8,3 18800 2239607 /usr/lib64/httpd/modules/mod_mime.so
httpd 29920 buildapi mem REG 8,3 14576 2239619 /usr/lib64/httpd/modules/mod_setenvif.so
httpd 29920 buildapi mem REG 8,3 14608 2239626 /usr/lib64/httpd/modules/mod_usertrack.so
httpd 29920 buildapi mem REG 8,3 18896 2238299 /usr/lib64/httpd/modules/mod_headers.so
httpd 29920 buildapi mem REG 8,3 27056 2238242 /usr/lib64/httpd/modules/mod_deflate.so
httpd 29920 buildapi mem REG 8,3 10512 2238264 /usr/lib64/httpd/modules/mod_expires.so
httpd 29920 buildapi mem REG 8,3 27672 2239608 /usr/lib64/httpd/modules/mod_mime_magic.so
httpd 29920 buildapi mem REG 8,3 23120 2238283 /usr/lib64/httpd/modules/mod_ext_filter.so
httpd 29920 buildapi mem REG 8,3 10416 2238263 /usr/lib64/httpd/modules/mod_env.so
httpd 29920 buildapi mem REG 8,3 10440 2239606 /usr/lib64/httpd/modules/mod_logio.so
httpd 29920 buildapi mem REG 8,3 27168 2239255 /usr/lib64/httpd/modules/mod_log_config.so
httpd 29920 buildapi mem REG 8,3 43600 2239237 /usr/lib64/httpd/modules/mod_include.so
httpd 29920 buildapi mem REG 8,3 31192 2235219 /usr/lib64/httpd/modules/mod_authnz_ldap.so
httpd 29920 buildapi mem REG 8,3 47112 2093081 /lib64/librt-2.12.so
httpd 29920 buildapi mem REG 8,3 91096 2093136 /lib64/libz.so.1.2.3
httpd 29920 buildapi mem REG 8,3 247192 2093183 /lib64/libnspr4.so
httpd 29920 buildapi mem REG 8,3 21288 2093173 /lib64/libplc4.so
httpd 29920 buildapi mem REG 8,3 17128 2093197 /lib64/libplds4.so
httpd 29920 buildapi mem REG 8,3 182112 2229197 /usr/lib64/libnssutil3.so
httpd 29920 buildapi mem REG 8,3 1302712 2229198 /usr/lib64/libnss3.so
httpd 29920 buildapi mem REG 8,3 183928 2229200 /usr/lib64/libsmime3.so
httpd 29920 buildapi mem REG 8,3 257120 2229199 /usr/lib64/libssl3.so
httpd 29920 buildapi mem REG 8,3 108728 2229201 /usr/lib64/libsasl2.so.2.0.23
httpd 29920 buildapi mem REG 8,3 113952 2093152 /lib64/libresolv-2.12.so
httpd 29920 buildapi mem REG 8,3 63336 2093202 /lib64/liblber-2.4.so.2.10.2
httpd 29920 buildapi mem REG 8,3 329696 2093205 /lib64/libldap-2.4.so.2.10.2
httpd 29920 buildapi mem REG 8,3 56144 2239252 /usr/lib64/httpd/modules/mod_ldap.so
httpd 29920 buildapi mem REG 8,3 6264 2235248 /usr/lib64/httpd/modules/mod_authz_default.so
httpd 29920 buildapi mem REG 8,3 10448 2235222 /usr/lib64/httpd/modules/mod_authz_dbm.so
httpd 29920 buildapi mem REG 8,3 10520 2235249 /usr/lib64/httpd/modules/mod_authz_groupfile.so
httpd 29920 buildapi mem REG 8,3 10424 2235259 /usr/lib64/httpd/modules/mod_authz_owner.so
httpd 29920 buildapi mem REG 8,3 10384 2237441 /usr/lib64/httpd/modules/mod_authz_user.so
httpd 29920 buildapi mem REG 8,3 10448 2235258 /usr/lib64/httpd/modules/mod_authz_host.so
httpd 29920 buildapi mem REG 8,3 6264 2235217 /usr/lib64/httpd/modules/mod_authn_default.so
httpd 29920 buildapi mem REG 8,3 10416 2235216 /usr/lib64/httpd/modules/mod_authn_dbm.so
httpd 29920 buildapi mem REG 8,3 10352 2235214 /usr/lib64/httpd/modules/mod_authn_anon.so
httpd 29920 buildapi mem REG 8,3 10456 2235213 /usr/lib64/httpd/modules/mod_authn_alias.so
httpd 29920 buildapi mem REG 8,3 10416 2235218 /usr/lib64/httpd/modules/mod_authn_file.so
httpd 29920 buildapi mem REG 8,3 31256 2235212 /usr/lib64/httpd/modules/mod_auth_digest.so
httpd 29920 buildapi mem REG 8,3 10448 2234748 /usr/lib64/httpd/modules/mod_auth_basic.so
httpd 29920 buildapi mem REG 8,3 480264 2093092 /lib64/libfreebl3.so
httpd 29920 buildapi mem REG 8,3 18936 2093114 /lib64/libuuid.so.1.3.0
httpd 29920 buildapi mem REG 8,3 22536 2093079 /lib64/libdl-2.12.so
httpd 29920 buildapi mem REG 8,3 1926760 2093070 /lib64/libc-2.12.so
httpd 29920 buildapi mem REG 8,3 145896 2093071 /lib64/libpthread-2.12.so
httpd 29920 buildapi mem REG 8,3 184408 2234607 /usr/lib64/libapr-1.so.0.3.9
httpd 29920 buildapi mem REG 8,3 1525256 2093073 /lib64/libdb-4.7.so
httpd 29920 buildapi mem REG 8,3 167648 2093133 /lib64/libexpat.so.1.5.2
httpd 29920 buildapi mem REG 8,3 43392 2093094 /lib64/libcrypt-2.12.so
httpd 29920 buildapi mem REG 8,3 146080 2230821 /usr/lib64/libaprutil-1.so.0.3.9
httpd 29920 buildapi mem REG 8,3 124624 2093105 /lib64/libselinux.so.1
httpd 29920 buildapi mem REG 8,3 183816 2093216 /lib64/libpcre.so.0.0.1
httpd 29920 buildapi mem REG 8,3 599392 2093075 /lib64/libm-2.12.so
httpd 29920 buildapi mem REG 8,3 156936 2093067 /lib64/ld-2.12.so
httpd 29920 buildapi DEL REG 8,3 1311782 /tmp/ffigQHNhq
httpd 29920 buildapi DEL REG 0,4 44731549 /dev/zero
httpd 29920 buildapi DEL REG 0,4 89040053 /dev/zero
httpd 29920 buildapi 2w REG 8,3 123644 1312846 /var/log/httpd/error_log
httpd 29920 buildapi 10u REG 0,9 0 3785 [eventpoll]
httpd 29920 buildapi 13w REG 8,3 16848 1312845 /var/log/httpd/access_log
httpd 29920 buildapi 71w REG 8,3 16848 1312845 /var/log/httpd/access_log
httpd 29920 buildapi 73w REG 8,3 0 789626 /var/log/newrelic/php_agent.log
httpd 29920 buildapi 75w REG 8,3 0 786301 /var/run/wsgi.23701.8.1.lock (deleted)
httpd 29920 buildapi 76w REG 8,3 223588 787024 /var/log/buildapi/buildapi.log
httpd 29920 buildapi 77w REG 8,3 0 786415 /var/log/buildapi/buildapi.lock
httpd 29920 buildapi 83w REG 8,3 223588 787024 /var/log/buildapi/buildapi.log
httpd 29920 buildapi 84w REG 8,3 0 786415 /var/log/buildapi/buildapi.lock
httpd 29920 buildapi 88u REG 8,3 4096 1311782 /tmp/ffigQHNhq (deleted)
|
|
||
Comment 12•11 years ago
|
||
buildapi log seems ok:
[pmoore@web2.releng.webapp.scl3 buildapi]$ tail -50 /var/log/buildapi/buildapi.log
2014-12-23 03:04:05,425 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:04:05,488 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/mozilla-inbound/request
2014-12-23 03:04:05,520 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request mozilla-inbound 57798910 0 1
2014-12-23 03:04:05,544 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116308L, 'when': 1419332646L, 'brid': 57798910}, 'who': 'cbook@mozilla.com'}
2014-12-23 03:04:05,544 WARNI [buildapi.lib.mq] [MainThread] Kombu connection failed (Socket closed); waiting 0s
2014-12-23 03:04:09,449 INFO [buildapi.lib.mq] [Dummy-1] Got {u'body': {u'msg': u'Ok', u'errors': False}, u'request_id': 1116308}
2014-12-23 03:04:09,459 INFO [buildapi.lib.mq] [Dummy-1] Updating complete data
2014-12-23 03:04:43,269 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:04:43,334 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/mozilla-inbound/request
2014-12-23 03:04:43,443 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request mozilla-inbound 57798466 0 1
2014-12-23 03:04:43,467 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116309L, 'when': 1419332683L, 'brid': 57798466}, 'who': 'cbook@mozilla.com'}
2014-12-23 03:04:43,467 WARNI [buildapi.lib.mq] [MainThread] Kombu connection failed (Socket closed); waiting 0s
2014-12-23 03:05:06,975 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:05:07,044 INFO [buildapi.access] [MainThread] mhammond@mozilla.com accessing POST /self-serve/try/request
2014-12-23 03:05:07,064 INFO [buildapi.access] [MainThread] mhammond@mozilla.com rebuild_request try 57793843 0 1
2014-12-23 03:05:07,091 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116311L, 'when': 1419332707L, 'brid': 57793843}, 'who': 'mhammond@mozilla.com'}
2014-12-23 03:05:44,647 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:05:44,762 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/fx-team/request
2014-12-23 03:05:44,789 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request fx-team 57801139 0 1
2014-12-23 03:05:44,810 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116313L, 'when': 1419332745L, 'brid': 57801139}, 'who': 'cbook@mozilla.com'}
2014-12-23 03:05:44,811 WARNI [buildapi.lib.mq] [MainThread] Kombu connection failed (Socket closed); waiting 0s
2014-12-23 03:05:46,921 INFO [buildapi.lib.mq] [Dummy-1] Got {u'body': {u'msg': u'Ok', u'errors': False}, u'request_id': 1116313}
2014-12-23 03:05:46,930 INFO [buildapi.lib.mq] [Dummy-1] Updating complete data
2014-12-23 03:07:47,178 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/fx-team/request
2014-12-23 03:07:47,193 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request fx-team 57784347 0 1
2014-12-23 03:07:47,209 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116314L, 'when': 1419332867L, 'brid': 57784347}, 'who': 'cbook@mozilla.com'}
2014-12-23 03:25:23,321 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:30:23,414 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:35:23,288 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:40:23,225 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 03:55:23,328 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 04:05:24,708 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 04:16:08,193 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 04:16:08,309 INFO [buildapi.access] [MainThread] opettay@mozilla.com accessing POST /self-serve/try/rev/a38516dcf660
2014-12-23 04:16:08,309 INFO [buildapi.access] [MainThread] opettay@mozilla.com cancel_revision try a38516dcf660
2014-12-23 04:16:08,333 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'cancel_revision', 'body': {'request_id': 1116316L, 'when': 1419336968L, 'branch': u'try', 'revision': u'a38516dcf660'}, 'who': 'opettay@mozilla.com'}
2014-12-23 04:16:08,334 WARNI [buildapi.lib.mq] [MainThread] Kombu connection failed (Socket closed); waiting 0s
2014-12-23 04:16:43,624 INFO [buildapi.lib.mq] [Dummy-1] Got {u'body': {u'msg': u'57814934: Ok\n57814935: Ok\n57814936: Ok\n57814937: Ok\n57814938: Ok\n57814939: Ok\n57814940: Ok\n57814942: Ok\n57814943: Ok\n57814945: Ok\n57814946: Ok\n57814947: Ok\n57814948: Ok\n57814950: Ok\n57814952: Ok\n57814953: Ok\n57814956: Ok\n57814957: Ok\n57814960: Ok\n57814961: Ok\n57814968: Ok\n57814969: Ok\n57814990: Ok\n57814991: Ok\n57814994: Ok', u'errors': False}, u'request_id': 1116316}
2014-12-23 04:16:43,635 INFO [buildapi.lib.mq] [Dummy-1] Updating complete data
2014-12-23 04:21:10,148 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
2014-12-23 04:21:10,272 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/mozilla-inbound/request
2014-12-23 04:21:10,302 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request mozilla-inbound 57812528 0 1
2014-12-23 04:21:10,328 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116317L, 'when': 1419337270L, 'brid': 57812528}, 'who': 'cbook@mozilla.com'}
2014-12-23 04:21:10,329 WARNI [buildapi.lib.mq] [MainThread] Kombu connection failed (Socket closed); waiting 0s
2014-12-23 04:21:32,497 INFO [buildapi.lib.mq] [Dummy-1] Got {u'body': {u'msg': u'Ok', u'errors': False}, u'request_id': 1116318}
2014-12-23 04:21:32,510 INFO [buildapi.lib.mq] [Dummy-1] Updating complete data
2014-12-23 04:25:44,956 INFO [buildapi.access] [MainThread] cbook@mozilla.com accessing POST /self-serve/mozilla-inbound/request
2014-12-23 04:25:44,969 INFO [buildapi.access] [MainThread] cbook@mozilla.com rebuild_request mozilla-inbound 57809164 0 1
2014-12-23 04:25:44,989 INFO [buildapi.lib.mq] [MainThread] Sending {'action': 'rebuild_request', 'body': {'priority': 0, 'count': 1, 'request_id': 1116319L, 'when': 1419337545L, 'brid': 57809164}, 'who': 'cbook@mozilla.com'}
2014-12-23 04:45:27,557 INFO [buildapi.lib.helpers] [MainThread] Fetching branches list from http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
|
|
||
Comment 13•11 years ago
|
||
Asking now in #netops.
Also noticed that other services are down that are on relengwebadm, e.g. RelEngAPI:
https://api.pub.build.mozilla.org/
So it looks like a problem of the load balancer not hitting the back ends. This also explains why we are not getting entries in the access logs of apache.
|
|
||
Comment 14•11 years ago
|
||
Working through this with "pir" in #netops now.
pir
15:14:53 pmoore|buildduty: the Zeus node thinks it can't reach the backends
|
|
||
Comment 15•11 years ago
|
||
OK, looks like it is not a network issue after all, maybe the httpd service just needs a kick, hanging when I issue:
[root@web2.releng.webapp.scl3 pmoore]# curl -v -H "X-Cluster-Client-IP: 10.26.48.43" -H "Host: buildapi.pvt.build.mozilla.org" http://localhost:80/buildapi/self-serve/jobs
* About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /buildapi/self-serve/jobs HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Accept: */*
> X-Cluster-Client-IP: 10.26.48.43
> Host: buildapi.pvt.build.mozilla.org
>
|
|
||
Comment 16•11 years ago
|
||
OK I've restarted httpd services on web1.releng.webapp.scl3.mozilla.com and web2.releng.webapp.scl3.mozilla.com:
[root@web1.releng.webapp.scl3 pmoore]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: [ OK ]
[root@web1.releng.webapp.scl3 pmoore]# curl -v -H "X-Cluster-Client-IP: 10.26.48.43" -H "Host: buildapi.pvt.build.mozilla.org" http://localhost:80/buildapi/self-serve/jobs
* About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /buildapi/self-serve/jobs HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Accept: */*
> X-Cluster-Client-IP: 10.26.48.43
> Host: buildapi.pvt.build.mozilla.org
>
< HTTP/1.1 200 OK
< Date: Tue, 23 Dec 2014 14:35:26 GMT
< Server: Apache
< Pragma: no-cache
< Cache-Control: no-cache
< X-Backend-Server: web1.releng.webapp.scl3.mozilla.com
< Content-Length: 38927
< Content-Type: text/html; charset=utf-8
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<title>RelEng Self-Serve API - Job History</title>
<link href="/buildapi/jquery/css/smoothness/jquery-ui-1.8.1.custom.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/buildapi/releng.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/buildapi/build-status.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/buildapi/DataTables-1.7.1/media/css/demo_page.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/buildapi/DataTables-1.7.1/media/css/demo_table_jui.css" media="screen" rel="stylesheet" type="text/css" />
<script src="/buildapi/js/jquery-ui-1.8.1.custom.min.js" type="text/javascript"></script>
<script src="/buildapi/DataTables-1.7.1/media/js/jquery.js" type="text/javascript"></script>
<script src="/buildapi/DataTables-1.7.1/media/js/jquery.dataTables.min.js" type="text/javascript"></script>
<script type="text/javascript">
$(document).ready(function()
{
var options = {
bJQueryUI: true,
sPaginationType: "full_numbers",
aaSorting: [[2, 'desc']]
}
$("#jobs").dataTable(options);
})
</script>
</head>
<body>
<table id="jobs">
<thead>
<tr><th>Who</th><th>What</th><th>When</th><th>Completed at</th><th>Results</th></tr>
</thead>
<tbody>
<tr><td>opettay@mozilla.com</td>
<td>cancel_revision <pre>{'branch': 'try', 'revision': '032fe44aa93d'}</pre></td>
<td>2014-12-23 04:34:31</td>
..... <snip/> .....
<td><pre>{'body': {'msg': 'Ok', 'errors': False}, 'request_id': 1116221}</pre></td>
</tr>
</tbody>
</table>
Generated at 2014-12-23 06:35:27 in 0.292s. All times are Mountain View, CA (US/Pacific).
</body>
</html>
* Connection #0 to host localhost left intact
* Closing connection #0
[root@web1.releng.webapp.scl3 pmoore]# hostname
web1.releng.webapp.scl3.mozilla.com
[root@web1.releng.webapp.scl3 pmoore]#
Looks like we are back in business!
|
|
||
Updated•11 years ago
|
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
|
||
Updated•8 years ago
|
Product: Release Engineering → Infrastructure & Operations
|
|
||
Updated•6 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•