Closed Bug 1084244 Opened 11 years ago Closed 11 years ago

nagios1.private.scl3 nagiosbot connected to buildduty but not sysadmins channel

Categories

(Infrastructure & Operations Graveyard :: Infrastructure: IRC, task)

Product:
Infrastructure & Operations Graveyard
Component:
Infrastructure: IRC
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dgarvey, Assigned: dgarvey)

Details

I tried restarting it a couple of times and nada. I sent this command and notice shortly afterwards that I was receiving pages only. <dgarvey> nagios-corp-phx1 recheck 3279 <@nagios-corp-phx1> dgarvey: rechecking all services on host zmproxy1.mail.corp.phx1.mozilla.com <@nagios-corp-phx1> Fri 00:14:57 PDT [3286] zmproxy1.mail.corp.phx1.mozilla.com:Disk - All is OK: DISK OK (http://m.mozilla.org/Disk+-+All) [root@nagios1.private.scl3 ~]# /etc/init.d/nagiosbot stop Stopping nagiosbot: [ OK ] [root@nagios1.private.scl3 ~]# /etc/init.d/nagiosbot start Starting nagiosbot: [ OK ] [root@nagios1.private.scl3 ~]# ps -efww | grep bot nagios 16687 1 0 00:38 ? 00:00:00 /usr/bin/python2.6 /usr/local/bin/nagios-bot.py root 20353 1 0 Oct02 ? 00:00:00 tailf /var/log/nagios/nagiosbot.log root 22480 31697 0 00:39 pts/6 00:00:00 grep bot [root@nagios1.private.scl3 ~]# lsof -p 16687 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nagios-bo 16687 nagios cwd DIR 8,3 4096 3145755 /usr/local/bin nagios-bo 16687 nagios rtd DIR 8,3 4096 2 / nagios-bo 16687 nagios txt REG 8,3 9176 3149879 /usr/bin/python2.6 nagios-bo 16687 nagios mem REG 8,3 156872 7472318 /lib64/ld-2.12.so nagios-bo 16687 nagios mem REG 8,3 22536 7472321 /lib64/libdl-2.12.so nagios-bo 16687 nagios mem REG 8,3 1979000 7472319 /lib64/libc-2.12.so nagios-bo 16687 nagios mem REG 8,3 598800 7472329 /lib64/libm-2.12.so nagios-bo 16687 nagios mem REG 8,3 124624 7472330 /lib64/libselinux.so.1 nagios-bo 16687 nagios mem REG 8,3 113952 7472332 /lib64/libresolv-2.12.so nagios-bo 16687 nagios mem REG 8,3 46336 7472334 /lib64/libkrb5support.so.0.1 nagios-bo 16687 nagios mem REG 8,3 12592 7472333 /lib64/libkeyutils.so.1.3 nagios-bo 16687 nagios mem REG 8,3 181632 7472335 /lib64/libk5crypto.so.3.1 nagios-bo 16687 nagios mem REG 8,3 1953536 3156630 /usr/lib64/libcrypto.so.1.0.1e nagios-bo 16687 nagios mem REG 8,3 444040 3149119 /usr/lib64/libssl.so.1.0.1e nagios-bo 16687 nagios mem REG 8,3 17520 7473737 /lib64/libutil-2.12.so nagios-bo 16687 nagios mem REG 8,3 91096 7471699 /lib64/libz.so.1.2.3 nagios-bo 16687 nagios mem REG 8,3 17256 7473927 /lib64/libcom_err.so.2.1 nagios-bo 16687 nagios mem REG 8,3 141576 7473898 /lib64/libpthread-2.12.so nagios-bo 16687 nagios mem REG 8,3 915736 7473928 /lib64/libkrb5.so.3.3 nagios-bo 16687 nagios mem REG 8,3 272360 7473929 /lib64/libgssapi_krb5.so.2.2 nagios-bo 16687 nagios mem REG 8,3 1751360 3146678 /usr/lib64/libpython2.6.so.1.0 nagios-bo 16687 nagios mem REG 8,3 27424 7471131 /lib64/libnss_dns-2.12.so nagios-bo 16687 nagios mem REG 8,3 65928 7471133 /lib64/libnss_files-2.12.so nagios-bo 16687 nagios mem REG 8,3 81256 3670972 /usr/lib64/python2.6/lib-dynload/datetime.so nagios-bo 16687 nagios mem REG 8,3 76464 3670969 /usr/lib64/python2.6/lib-dynload/cPickle.so nagios-bo 16687 nagios mem REG 8,3 12256 3670950 /usr/lib64/python2.6/lib-dynload/_functoolsmodule.so nagios-bo 16687 nagios mem REG 8,3 9872 3670933 /usr/lib64/python2.6/lib-dynload/_bisectmodule.so nagios-bo 16687 nagios mem REG 8,3 21232 3670967 /usr/lib64/python2.6/lib-dynload/binascii.so nagios-bo 16687 nagios mem REG 8,3 37872 3670963 /usr/lib64/python2.6/lib-dynload/_struct.so nagios-bo 16687 nagios mem REG 8,3 25448 3670993 /usr/lib64/python2.6/lib-dynload/stropmodule.so nagios-bo 16687 nagios mem REG 8,3 14632 3670975 /usr/lib64/python2.6/lib-dynload/fcntlmodule.so nagios-bo 16687 nagios mem REG 8,3 20328 3670996 /usr/lib64/python2.6/lib-dynload/timemodule.so nagios-bo 16687 nagios mem REG 8,3 24432 3670991 /usr/lib64/python2.6/lib-dynload/selectmodule.so nagios-bo 16687 nagios mem REG 8,3 41936 3670985 /usr/lib64/python2.6/lib-dynload/operator.so nagios-bo 16687 nagios mem REG 8,3 30352 3670942 /usr/lib64/python2.6/lib-dynload/_collectionsmodule.so nagios-bo 16687 nagios mem REG 8,3 99158704 3153434 /usr/lib/locale/locale-archive nagios-bo 16687 nagios mem REG 8,3 20112 3670970 /usr/lib64/python2.6/lib-dynload/cStringIO.so nagios-bo 16687 nagios mem REG 8,3 33216 3670962 /usr/lib64/python2.6/lib-dynload/_ssl.so nagios-bo 16687 nagios mem REG 8,3 60752 3670960 /usr/lib64/python2.6/lib-dynload/_socketmodule.so nagios-bo 16687 nagios 0w CHR 1,3 0t0 3829 /dev/null nagios-bo 16687 nagios 1w REG 8,3 156058466 7476742 /var/log/nagios/nagiosbot.log nagios-bo 16687 nagios 2w REG 8,3 156058466 7476742 /var/log/nagios/nagiosbot.log nagios-bo 16687 nagios 3w REG 8,3 0 7341286 /var/lock/nagiosbot.lock nagios-bo 16687 nagios 4w CHR 1,3 0t0 3829 /dev/null nagios-bo 16687 nagios 5w REG 8,3 156058466 7476742 /var/log/nagios/nagiosbot.log nagios-bo 16687 nagios 6u IPv4 2734824375 0t0 TCP nagios1.private.scl3.mozilla.com:39260->ec2-54-219-165-167.us-west-1.compute.amazonaws.com:6697 (ESTABLISHED) nagios-bo 16687 nagios 7w REG 8,3 70130922 7472818 /var/log/nagios/nagiosbot-python.log nagios-bo 16687 nagios 8r REG 8,3 306931348 7477104 /var/log/nagios/nagios.log nagios-bo 16687 nagios 9r REG 8,3 29902 7341183 /var/log/incoming_sms_log [root@nagios1.private.scl3 ~]# less /var/log/nagios/nagiosbot-python.log [root@nagios1.private.scl3 ~]# tail -f /var/log/nagios/nagios.log [1413531279] SERVICE NOTIFICATION: servicenow-moc;moztrap1.webapp.scl3.mozilla.com;Swap;OK;notify-by-servicenow-moc;SWAP OK - 79% free (1600 MB out of 2047 MB) [1413531279] SERVICE NOTIFICATION: bugzilla-moc;moztrap1.webapp.scl3.mozilla.com;Swap;OK;notify-by-bugzilla-moc;SWAP OK - 79% free (1600 MB out of 2047 MB) [1413531280] SERVICE NOTIFICATION: nmdashnagios;moztrap1.webapp.scl3.mozilla.com;Swap;OK;notify-by-email;SWAP OK - 79% free (1600 MB out of 2047 MB) [1413531280] SERVICE NOTIFICATION: sysadmin-oncall;moztrap1.webapp.scl3.mozilla.com;Swap;OK;notify-by-sms;SWAP OK - 79% free (1600 MB out of 2047 MB) [1413531280] SERVICE NOTIFICATION: irc;moztrap1.webapp.scl3.mozilla.com;Swap;OK;notify-by-email;SWAP OK - 79% free (1600 MB out of 2047 MB) [1413531289] SERVICE ALERT: esxucs1.ops.scl3.mozilla.com;Fans;WARNING;HARD;3;2014/10/17 00:34:43 Post https://10.22.9.171/nuova/: dial tcp 10.22.9.171:443: connection refused [1413531389] SERVICE ALERT: bouncer1.db.scl3.mozilla.com;MySQL Replication;CRITICAL;SOFT;1;CRIT: Replication is stopped. Master Server: bouncer1 (bouncer1.db.phx1.mozilla.com) Master User: repl Master Port: 3306 Last Error: [1413531509] SERVICE ALERT: esxucs1.ops.scl3.mozilla.com;PSUs;WARNING;HARD;3;2014/10/17 00:38:22 Post https://10.22.9.171/nuova/: dial tcp 10.22.9.171:443: connection refused [1413531509] SERVICE ALERT: bouncer1.db.scl3.mozilla.com;MySQL Replication;OK;SOFT;2;OK: Replication is working successfully. [1413531679] SERVICE ALERT: pbx1.ops.yvr1.mozilla.net;Aggressive ping check;CRITICAL;HARD;1;PING CRITICAL - Packet loss = 0%, RTA = 58.58 ms ^C [root@nagios1.private.scl3 ~]# less /usr/local/bin/nagios-bot.py [root@nagios1.private.scl3 ~]#
[root@nagios1.private.scl3 ~]# tail /var/log/nagios/nagiosbot.log Registered!!! Joining channels... ['Password accepted - you are now recognized.'] ['nagios-scl3!nagios-scl3@moz-scgvv7.scl3.mozilla.com', 'nagios-scl3', 'You are now logged in as nagios-scl3'] [] ['=', '#Bug848086', '@nagios-scl3 '] ['#Bug848086', 'End of /NAMES list.'] [] ['#buildduty', 'all your base are belong to us -- or some such'] ['#buildduty', '[root@nagios1.private.scl3 ~]#
Summary: agios1.private.scl3 nagiosbot connected to buildduty but not sysadmins channel → nagios1.private.scl3 nagiosbot connected to buildduty but not sysadmins channel
[root@nagios1.private.scl3 nagios]# strace -p 16687 Process 16687 attached - interrupt to quit select(7, [6], [], [6], {27, 224552}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {29, 228237}) read(6, "\27\3\3\0`", 5) = 5 read(6, "\232|\216\201]H\333\260\310OJ\233t\213~\3744\301\251\354\fB\321\277\263\5-\211\344\21\324\34"..., 96) = 96 write(6, "\27\3\3\0P\201\215NOli\257\22Pq\276\221\307d\303\213\360\17\264v\217\362\343\17\352\235#"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {28, 822417}) read(6, "\27\3\3\0P", 5) = 5 read(6, "\216\331o\341\355\323p\336-t\4N\256bG\271\344Z\313Q\213q\225\"\1A\326C\256\355\341\337"..., 80) = 80 write(6, "\27\3\3\0P|9\252\1h\20\3149\0\317\345\16\265\231]\305\234\f\320\221\320\232\364\2224r\355"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {29, 203190}) read(6, "\27\3\3\0\320", 5) = 5 read(6, "\6\302X\37\177c\275\270\177\200O^\26\365\27626\0021o\310\217\246\31:\16\3506\376\252Z\221"..., 208) = 208 write(6, "\27\3\3\0PV\321\232\366L\260\242\215+c\222c0\271\305A\233\226\216\371\312\202\324\vy\235,"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {29, 272215}) read(6, "\27\3\3\0010", 5) = 5 read(6, "\270\224\343\362\336\24\356Q\374\331*\4cI\24\214\225\324\33F3\317O\24\222\20\3\214\213\212\220\327"..., 304) = 304 write(6, "\27\3\3\0Pb\371\27\302\23w\22\324\211\7]\177\365\214\213\v\2046\22f\f;Y\5\36\321\t"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {29, 62990}) read(6, "\27\3\3\0\340", 5) = 5 read(6, ".d3\343\256\241C\340?\244A\3345.\33\334\303\320\252\325\36\25s\1j\257!\355\327\332r5"..., 224) = 224 write(6, "\27\3\3\0PK\341\31\336\211\300\21Wk\365X\355J\22\17j\177\262\273\23\224\266\351\335\0354i"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}) = 1 (in [6], left {29, 192093}) read(6, "\27\3\3\0\220", 5) = 5 read(6, "\375\373,\35>\333\355\31\270\317\265\256]\350\311\275K\37\275pD\357\274\"(\2056:\220\203X."..., 144) = 144 write(6, "\27\3\3\0P\316\26\200\3#\365|r\334\366\23U\214\270\"\263,z\255\323\240_\257?p\341<"..., 85) = 85 select(7, [6], [], [6], {30, 0}) = 0 (Timeout) select(7, [6], [], [6], {30, 0}^C <unfinished ...> Process 16687 detached [root@nagios1.private.scl3 nagios]#
With Usul help he pointed me to the fact that the secret had changed yesterday. dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/hiera/secrets$ svn commit -m "locked out of sysadmins channel... bug1084244" X11 forwarding request failed on channel 0 Sending site.yaml Transmitting file data . Committed revision 94951. dgarvey@dgarvey-mozilla:~/svn/sysadmins/puppet/trunk/hiera/secrets$
dgarvey> ok updated the site.yaml and doing puppetctl run on nagios ━━▶ Joins: nagios-scl3 (nagios-scl3@moz-scgvv7.scl3.mozilla.com) ◀━━ Quits: nagios-corp-phx1 (nagios-corp@moz-gft683.phx1.mozilla.net) (A TLS packet with unexpected length was received.) ━━▶ Joins: nagios-corp-phx1 (nagios-corp@moz-gft683.phx1.mozilla.net) ❮▲❯ ChanServ gives channel operator status to nagios-corp-phx1 <dgarvey> there it is thanks Usul <dgarvey> /usr/local/etc/nagiosbot/settings.py]/content: content changed
Assignee: infra → dgarvey
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
did other missing channels in : oulan:secrets ludo$ svn commit -m "finishing bug 1084244" Sending site.yaml Transmitting file data . Committed revision 94957.
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.