Closed Bug 1047954 Opened 11 years ago Closed 11 years ago

DHCP updates on admin1b.private.ber1.mozilla.com is WARNING: FILE_AGE WARNING: /var/run/update-dhcpd.success is 2323 seconds old and 0 bytes

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: MOC
Platform:
Other
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nagiosapi, Unassigned)

References

(
URL
)

Details

(Whiteboard: [id=nagios1.private.scl3.mozilla.com:395091])

Automated alert report from nagios1.private.scl3.mozilla.com: Hostname: admin1b.private.ber1.mozilla.com Service: DHCP updates State: WARNING Output: FILE_AGE WARNING: /var/run/update-dhcpd.success is 2323 seconds old and 0 bytes Runbook: http://m.allizom.org/DHCP+updates
Humm, I looks like either svn is real slow or we have race issue with competing processes? NOTE: set -x removed;) [root@admin1b.private.ber1 ~]# /etc/dhcpd/update-dhcpd.sh + success= + dhcpdroot=/etc/dhcpd + mailto='infra@mozilla.com hoth@mozilla.com' ++ hostname -f ++ sed -e 's/\.mozilla\.org$//' -e 's/\.mozilla\.com$//' + hostname=admin1b.private.ber1 + export 'SVN_SSH=/usr/bin/ssh -q -o UserKnownHostsFile=/etc/dhcpd/known_hosts' + SVN_SSH='/usr/bin/ssh -q -o UserKnownHostsFile=/etc/dhcpd/known_hosts' ++ svn stat /etc/dhcpd ++ grep -v '^X' ++ grep -v '^ L' ++ grep -v '^Performing' ++ grep -v '^$' ++ grep -v environments ++ grep -v 'update-dhcpd\|force-primary\|known_hosts' + output= + '[' -n '' ']' + cd /etc/dhcpd ++ svn up + output='svn: E155004: Working copy '\''/etc/dhcpd'\'' locked. svn: E155004: '\''/etc/dhcpd'\'' is already locked. svn: run '\''svn cleanup'\'' to remove locks (type '\''svn help cleanup'\'' for details)' + [[ 1 -ne 0 ]] + touch /var/run/update-dhcpd.svn-up-failed + exit 1 [root@admin1b.private.ber1 ~]# lsof | grep dhcp svn 5233 root cwd DIR 8,3 4096 1968340 /etc/dhcpd svn 5233 root 3u REG 8,3 38912 1968821 /etc/dhcpd/.svn/wc.db ssh 5235 root cwd DIR 8,3 4096 1968340 /etc/dhcpd rsyslogd 16888 root 14w REG 8,3 21106 1446916 /var/log/dhcpd.log dhcpd 19660 dhcpd cwd DIR 8,3 4096 2 / dhcpd 19660 dhcpd rtd DIR 8,3 4096 2 / dhcpd 19660 dhcpd txt REG 8,3 825904 1184263 /usr/sbin/dhcpd dhcpd 19660 dhcpd mem REG 8,3 65928 130845 /lib64/libnss_files-2.12.so dhcpd 19660 dhcpd mem REG 8,3 472064 130862 /lib64/libfreebl3.so dhcpd 19660 dhcpd mem REG 8,3 43392 130885 /lib64/libcrypt-2.12.so dhcpd 19660 dhcpd mem REG 8,3 47064 136333 /lib64/librt-2.12.so dhcpd 19660 dhcpd mem REG 8,3 91096 130896 /lib64/libz.so.1.2.3 dhcpd 19660 dhcpd mem REG 8,3 22536 130858 /lib64/libdl-2.12.so dhcpd 19660 dhcpd mem REG 8,3 145896 134026 /lib64/libpthread-2.12.so dhcpd 19660 dhcpd mem REG 8,3 108728 1179688 /usr/lib64/libsasl2.so.2.0.23 dhcpd 19660 dhcpd mem REG 8,3 243096 130942 /lib64/libnspr4.so dhcpd 19660 dhcpd mem REG 8,3 21288 136306 /lib64/libplc4.so dhcpd 19660 dhcpd mem REG 8,3 17128 136312 /lib64/libplds4.so dhcpd 19660 dhcpd mem REG 8,3 182112 1191140 /usr/lib64/libnssutil3.so dhcpd 19660 dhcpd mem REG 8,3 1299096 1191380 /usr/lib64/libnss3.so dhcpd 19660 dhcpd mem REG 8,3 183928 1192988 /usr/lib64/libsmime3.so dhcpd 19660 dhcpd mem REG 8,3 252960 1192987 /usr/lib64/libssl3.so dhcpd 19660 dhcpd mem REG 8,3 113952 131328 /lib64/libresolv-2.12.so dhcpd 19660 dhcpd mem REG 8,3 1926800 130854 /lib64/libc-2.12.so dhcpd 19660 dhcpd mem REG 8,3 63336 136319 /lib64/liblber-2.4.so.2.5.6 dhcpd 19660 dhcpd mem REG 8,3 317200 136321 /lib64/libldap-2.4.so.2.5.6 dhcpd 19660 dhcpd mem REG 8,3 156928 130846 /lib64/ld-2.12.so dhcpd 19660 dhcpd 0u CHR 1,3 0t0 3800 /dev/null dhcpd 19660 dhcpd 1u CHR 1,3 0t0 3800 /dev/null dhcpd 19660 dhcpd 2u CHR 1,3 0t0 3800 /dev/null dhcpd 19660 dhcpd 3u unix 0xffff88013cd9f9c0 0t0 79523957 socket dhcpd 19660 dhcpd 4u raw 0t0 55249032 00000000:0001->00000000:0000 st=07 dhcpd 19660 dhcpd 5u pack 55249037 0t0 ALL type=SOCK_RAW dhcpd 19660 dhcpd 7u IPv4 55249038 0t0 UDP *:bootps dhcpd 19660 dhcpd 8u IPv4 55249039 0t0 TCP *:7911 (LISTEN) dhcpd 19660 dhcpd 9u IPv4 55253175 0t0 TCP 10.249.75.7:dhcp-failover->10.249.75.6:51496 (ESTABLISHED) dhcpd 19660 dhcpd 10u IPv4 55249041 0t0 TCP 10.249.75.7:dhcp-failover (LISTEN) dhcpd 19660 dhcpd 11w REG 8,3 649786 5658 /var/lib/dhcpd/dhcpd.leases [root@admin1b.private.ber1 ~]# ps -efww | grep 5233 root 5233 1 0 06:05 ? 00:00:00 svn up root 5235 5233 0 06:05 ? 00:00:00 /usr/bin/ssh -q -o UserKnownHostsFile=/etc/dhcpd/known_hosts dhcpconfig@svn.mozilla.org svnserve -t root 13120 11898 0 06:57 pts/1 00:00:00 grep 5233 [root@admin1b.private.ber1 ~]# lsof -p 5235 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ssh 5235 root cwd DIR 8,3 4096 1968340 /etc/dhcpd ssh 5235 root rtd DIR 8,3 4096 2 / ssh 5235 root txt REG 8,3 405800 1184301 /usr/bin/ssh ssh 5235 root mem REG 8,3 65928 130845 /lib64/libnss_files-2.12.so ssh 5235 root mem REG 8,3 47064 136333 /lib64/librt-2.12.so ssh 5235 root mem REG 8,3 243096 130942 /lib64/libnspr4.so ssh 5235 root mem REG 8,3 17128 136312 /lib64/libplds4.so ssh 5235 root mem REG 8,3 182112 1191140 /usr/lib64/libnssutil3.so ssh 5235 root mem REG 8,3 145896 134026 /lib64/libpthread-2.12.so ssh 5235 root mem REG 8,3 12592 130924 /lib64/libkeyutils.so.1.3 ssh 5235 root mem REG 8,3 46368 131336 /lib64/libkrb5support.so.0.1 ssh 5235 root mem REG 8,3 472064 130862 /lib64/libfreebl3.so ssh 5235 root mem REG 8,3 21288 136306 /lib64/libplc4.so ssh 5235 root mem REG 8,3 22536 130858 /lib64/libdl-2.12.so ssh 5235 root mem REG 8,3 1926800 130854 /lib64/libc-2.12.so ssh 5235 root mem REG 8,3 1299096 1191380 /usr/lib64/libnss3.so ssh 5235 root mem REG 8,3 17256 136337 /lib64/libcom_err.so.2.1 ssh 5235 root mem REG 8,3 177520 131337 /lib64/libk5crypto.so.3.1 ssh 5235 root mem REG 8,3 944712 136338 /lib64/libkrb5.so.3.3 ssh 5235 root mem REG 8,3 280520 136339 /lib64/libgssapi_krb5.so.2.2 ssh 5235 root mem REG 8,3 113952 131328 /lib64/libresolv-2.12.so ssh 5235 root mem REG 8,3 43392 130885 /lib64/libcrypt-2.12.so ssh 5235 root mem REG 8,3 116368 131154 /lib64/libnsl-2.12.so ssh 5235 root mem REG 8,3 91096 130896 /lib64/libz.so.1.2.3 ssh 5235 root mem REG 8,3 17520 131049 /lib64/libutil-2.12.so ssh 5235 root mem REG 8,3 1953536 1181015 /usr/lib64/libcrypto.so.1.0.1e ssh 5235 root mem REG 8,3 124624 131016 /lib64/libselinux.so.1 ssh 5235 root mem REG 8,3 12688 130934 /lib64/libfipscheck.so.1.1.0 ssh 5235 root mem REG 8,3 156928 130846 /lib64/ld-2.12.so ssh 5235 root 0r FIFO 0,8 0t0 79607117 pipe ssh 5235 root 1w FIFO 0,8 0t0 79607118 pipe ssh 5235 root 2w FIFO 0,8 0t0 79607111 pipe ssh 5235 root 3u IPv4 79607128 0t0 TCP 10.249.75.7:40816->svn.mozilla.org:ssh (ESTABLISHED) [root@admin1b.private.ber1 ~]#
This is fixed.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.