Closed Bug 902693 Opened 13 years ago Closed 13 years ago

unable to ack panda-NNNN.* by hostname

Categories

(Infrastructure & Operations :: Infrastructure: Other, task)

Product:
Infrastructure & Operations
Component:
Infrastructure: Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: Callek, Assigned: rtucker)

Details

So, I want to be able to ack panda's by hostname, especially for things like failed_pxe_booting, the state is checked against mobile-imaging* servers instead of pinging the panda. However nagios realizes it is a host anyway for all other UI I have been able to find. I only mention it because :Aj thinks that "its not technically a host" and is why the bot won't let this work.... I also note the failure of this working turns a 5 minute scriptable task into a 2 hours task for me. (Attached is a log snippet of what happens when I attempted it anyway): 17:50:47] nagios-releng Wed 14:50:55 PDT [4697] panda-0387.p4.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [17:53:44] -->| armenzg_omw (armenzg@moz-6476D764.cable.teksavvy.com) has joined #buildduty [17:53:44] |<-- mconley has left moznet (Input/output error) [17:55:23] =-= armenzg_omw is now known as armenzg [18:03:06] nagios-releng Wed 15:03:15 PDT [4698] bld-lion-r5-003.build.releng.scl3.mozilla.com:disk - / is CRITICAL: Connection refused by host (http://m.allizom.org/disk+-+/) [18:09:17] Callek|buildduty nagios-releng: ack panda-0282.p3.releng.scl1.mozilla.com Bug 902656 [18:09:17] Callek|buildduty nagios-releng: ack panda-0292.p3.releng.scl1.mozilla.com Bug 902658 [18:09:17] Callek|buildduty nagios-releng: ack panda-0295.p3.releng.scl1.mozilla.com Bug 902659 [18:09:17] Callek|buildduty nagios-releng: ack panda-0300.p3.releng.scl1.mozilla.com Bug 902660 [18:09:17] Callek|buildduty nagios-releng: ack panda-0301.p3.releng.scl1.mozilla.com Bug 902661 [18:09:17] Callek|buildduty nagios-releng: ack panda-0305.p3.releng.scl1.mozilla.com Bug 902662 [18:09:17] Callek|buildduty nagios-releng: ack panda-0306.p3.releng.scl1.mozilla.com Bug 902663 [18:09:17] Callek|buildduty nagios-releng: ack panda-0325.p3.releng.scl1.mozilla.com Bug 902665 [18:09:17] Callek|buildduty nagios-releng: ack panda-0340.p3.releng.scl1.mozilla.com Bug 902667 [18:09:17] Callek|buildduty nagios-releng: ack panda-0387.p4.releng.scl1.mozilla.com Bug 902668 [18:09:17] Callek|buildduty nagios-releng: ack panda-0396.p4.releng.scl1.mozilla.com Bug 902669 [18:09:17] Callek|buildduty nagios-releng: ack panda-0479.p5.releng.scl1.mozilla.com Bug 902670 [18:09:17] Callek|buildduty nagios-releng: ack panda-0482.p5.releng.scl1.mozilla.com Bug 902671 [18:09:17] Callek|buildduty nagios-releng: ack panda-0729.p8.releng.scl1.mozilla.com Bug 897947 [18:09:17] Callek|buildduty nagios-releng: ack panda-0737.p8.releng.scl1.mozilla.com Bug 897950 [18:09:17] Callek|buildduty nagios-releng: ack panda-0743.p8.releng.scl1.mozilla.com Bug 897549 [18:09:17] Callek|buildduty nagios-releng: ack panda-0763.p8.releng.scl1.mozilla.com Bug 897953 [18:09:17] Callek|buildduty nagios-releng: ack panda-0769.p8.releng.scl1.mozilla.com Bug 900875 [18:09:17] Callek|buildduty nagios-releng: ack panda-0770.p8.releng.scl1.mozilla.com Bug 900857 [18:09:17] Callek|buildduty nagios-releng: ack panda-0788.p9.releng.scl1.mozilla.com Bug 897948 [18:09:17] Callek|buildduty nagios-releng: ack panda-0172.p2.releng.scl1.mozilla.com Bug 887807 [18:09:17] Callek|buildduty nagios-releng: ack panda-0180.p2.releng.scl1.mozilla.com Bug 858091 [18:09:17] Callek|buildduty nagios-releng: ack panda-0296.p3.releng.scl1.mozilla.com Bug 887806 [18:09:17] Callek|buildduty nagios-releng: ack panda-0313.p3.releng.scl1.mozilla.com Bug 889289 [18:09:17] Callek|buildduty nagios-releng: ack panda-0371.p4.releng.scl1.mozilla.com Bug 866125 [18:09:17] Callek|buildduty nagios-releng: ack panda-0392.p4.releng.scl1.mozilla.com Bug 866128 [18:09:17] Callek|buildduty nagios-releng: ack panda-0395.p4.releng.scl1.mozilla.com Bug 866129 [18:09:17] Callek|buildduty nagios-releng: ack panda-0664.p7.releng.scl1.mozilla.com Bug 902672 [18:09:17] Callek|buildduty nagios-releng: ack panda-0674.p7.releng.scl1.mozilla.com Bug 902673 [18:09:17] Callek|buildduty nagios-releng: ack panda-0696.p7.releng.scl1.mozilla.com Bug 902674 [18:09:17] Callek|buildduty nagios-releng: ack panda-0720.p8.releng.scl1.mozilla.com Bug 902675 [18:09:18] nagios-releng Callek|buildduty: acknowledged host panda-0282.p3.releng.scl1.mozilla.com Bug [18:09:19] nagios-releng Callek|buildduty: acknowledged host panda-0292.p3.releng.scl1.mozilla.com Bug [18:09:21] nagios-releng Callek|buildduty: acknowledged host panda-0295.p3.releng.scl1.mozilla.com Bug [18:09:23] nagios-releng Callek|buildduty: acknowledged host panda-0300.p3.releng.scl1.mozilla.com Bug [18:09:24] nagios-releng Callek|buildduty: acknowledged host panda-0301.p3.releng.scl1.mozilla.com Bug [18:09:26] nagios-releng Callek|buildduty: acknowledged host panda-0305.p3.releng.scl1.mozilla.com Bug [18:09:27] nagios-releng Callek|buildduty: acknowledged host panda-0306.p3.releng.scl1.mozilla.com Bug [18:09:29] nagios-releng Callek|buildduty: acknowledged host panda-0325.p3.releng.scl1.mozilla.com Bug [18:09:31] nagios-releng Callek|buildduty: acknowledged host panda-0340.p3.releng.scl1.mozilla.com Bug [18:09:32] nagios-releng Callek|buildduty: acknowledged host panda-0387.p4.releng.scl1.mozilla.com Bug [18:09:34] nagios-releng Callek|buildduty: acknowledged host panda-0396.p4.releng.scl1.mozilla.com Bug [18:09:35] nagios-releng Callek|buildduty: acknowledged host panda-0479.p5.releng.scl1.mozilla.com Bug [18:09:37] nagios-releng Callek|buildduty: acknowledged host panda-0482.p5.releng.scl1.mozilla.com Bug [18:09:39] nagios-releng Callek|buildduty: acknowledged host panda-0729.p8.releng.scl1.mozilla.com Bug [18:09:40] nagios-releng Callek|buildduty: acknowledged host panda-0737.p8.releng.scl1.mozilla.com Bug [18:09:42] nagios-releng Callek|buildduty: acknowledged host panda-0743.p8.releng.scl1.mozilla.com Bug [18:09:43] nagios-releng Callek|buildduty: acknowledged host panda-0763.p8.releng.scl1.mozilla.com Bug [18:09:45] nagios-releng Callek|buildduty: acknowledged host panda-0769.p8.releng.scl1.mozilla.com Bug [18:09:47] nagios-releng Callek|buildduty: acknowledged host panda-0770.p8.releng.scl1.mozilla.com Bug [18:09:48] nagios-releng Callek|buildduty: acknowledged host panda-0788.p9.releng.scl1.mozilla.com Bug [18:09:48] dividehex Callek|buildduty++ [18:09:50] nagios-releng Callek|buildduty: acknowledged host panda-0172.p2.releng.scl1.mozilla.com Bug [18:09:51] nagios-releng Callek|buildduty: acknowledged host panda-0180.p2.releng.scl1.mozilla.com Bug [18:09:53] nagios-releng Callek|buildduty: acknowledged host panda-0296.p3.releng.scl1.mozilla.com Bug [18:09:54] nagios-releng Callek|buildduty: acknowledged host panda-0313.p3.releng.scl1.mozilla.com Bug [18:09:57] nagios-releng Callek|buildduty: acknowledged host panda-0371.p4.releng.scl1.mozilla.com Bug [18:09:58] nagios-releng Callek|buildduty: acknowledged host panda-0392.p4.releng.scl1.mozilla.com Bug [18:10:01] nagios-releng Callek|buildduty: acknowledged host panda-0395.p4.releng.scl1.mozilla.com Bug [18:10:02] nagios-releng Callek|buildduty: acknowledged host panda-0664.p7.releng.scl1.mozilla.com Bug [18:10:05] nagios-releng Callek|buildduty: acknowledged host panda-0674.p7.releng.scl1.mozilla.com Bug [18:10:06] nagios-releng Callek|buildduty: acknowledged host panda-0696.p7.releng.scl1.mozilla.com Bug [18:10:08] nagios-releng Callek|buildduty: acknowledged host panda-0720.p8.releng.scl1.mozilla.com Bug [18:10:34] Callek|buildduty ashish: ping [18:10:45] Callek|buildduty ashish: sooo, what did I do wrong with the syntax there? [18:11:22] |<-- tonymec has left moznet (Ping timeout) [18:12:26] nagios-releng Wed 15:12:35 PDT [4699] tegra-310.build.mtv1.mozilla.com:tegra agent check is CRITICAL: Connection refused (http://m.allizom.org/tegra+agent+check) [18:12:26] nagios-releng Wed 15:12:35 PDT [4700] tegra-315.build.mtv1.mozilla.com:tegra agent check is CRITICAL: Connection refused (http://m.allizom.org/tegra+agent+check) [18:12:51] =-= armenzg is now known as armenzg_brb [18:14:41] -->| philor (philor@moz-E8BD9EDD.csby.or.frontiernet.net) has joined #buildduty [18:15:56] nagios-releng Wed 15:16:05 PDT [4701] w64-ix-slave39.winbuild.scl1.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100% [18:17:40] =-= jyeo is now known as jyeo_away [18:19:46] nagios-releng Wed 15:19:55 PDT [4702] t-w864-ix-097.wintest.releng.scl3.mozilla.com is DOWN :PING CRITICAL - Packet loss = 100% [18:21:16] nagios-releng Wed 15:21:25 PDT [4703] panda-0282.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:17] nagios-releng Wed 15:21:25 PDT [4704] panda-0292.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:17] nagios-releng Wed 15:21:25 PDT [4705] panda-0301.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:17] nagios-releng Wed 15:21:25 PDT [4706] panda-0305.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:17] nagios-releng Wed 15:21:25 PDT [4707] panda-0295.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:17] nagios-releng Wed 15:21:25 PDT [4708] panda-0300.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:18] nagios-releng Wed 15:21:25 PDT [4709] panda-0306.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:21] nagios-releng Wed 15:21:25 PDT [4710] panda-0325.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:22] nagios-releng Wed 15:21:25 PDT [4711] panda-0340.p3.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:21:56] nagios-releng Wed 15:22:05 PDT [4712] panda-0729.p8.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:22:06] nagios-releng Wed 15:22:15 PDT [4713] panda-0770.p8.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:22:07] nagios-releng Wed 15:22:15 PDT [4714] panda-0769.p8.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:22:07] nagios-releng Wed 15:22:15 PDT [4715] panda-0788.p9.releng.scl1.mozilla.com is DOWN :CRITICAL: in state failed_pxe_booting [18:23:36] Callek|buildduty nagios-releng: help [18:23:37] nagios-releng Available Commands: | page | ack | unack | recheck | status | downtime | undowntime | mute | unmute | oncall [18:23:40] Callek|buildduty nagios-releng: help ack [18:23:40] nagios-releng Callek|buildduty: Here is usage for ack [18:23:40] nagios-releng ack <alert_id> <reason for ack> [18:23:40] nagios-releng ack <host> <reason for ack> [18:23:40] nagios-releng ack <host:service> <reason for ack> [18:24:11] Callek|buildduty nagios-releng: ack panda-0720.p8.releng.scl1.mozilla.com: Bug 902675 [18:24:12] nagios-releng Callek|buildduty: acknowledged service panda-0720.p8.releng.scl1.mozilla.com: Bug 902675 [18:24:17] nagios-releng Wed 15:24:26 PDT [4716] bld-lion-r5-069.build.releng.scl3.mozilla.com:buildbot is WARNING: PROCS WARNING: 0 processes with args buildbot.tac (http://m.allizom.org/buildbot) [18:24:17] Callek|buildduty bah [18:24:40] Callek|buildduty nagios-releng: ack panda-0720.p8.releng.scl1.mozilla.com Bug 902675 [18:24:40] nagios-releng Callek|buildduty: acknowledged host panda-0720.p8.releng.scl1.mozilla.com Bug [18:24:54] Callek|buildduty nagios-releng--
So we learned that if we collapse the space it works. We also learned that it tries to take the last "word" (no space) as the comment in host acks, and everything-else is treated as part of the host :(
Nice mid-air... Seems after further testing, seems there's a bug with the nagios-bot. Basically, if one acks a host but one's comment has more than 1 field; nagios-releng ack panda-0282.p3.releng.scl1.mozilla.com test test2 test3 test4 <nagios-rele> | Aj: acknowledged host panda-0282.p3.releng.scl1.mozilla.com test test2 test3 The ack is missed (ie apparently doesn't reach nagios) and the bot forgets the last field. However, if you keep the comment to 1 field, comment goes through and "ack" is passed to nagios.
Severity: major → normal
Can confirm Comment 2. Here is the corresponding line from /var/log/nagios/nagios.log: [1375917538] EXTERNAL COMMAND: ACKNOWLEDGE_HOST_PROBLEM;panda-0282.p3.releng.scl1.mozilla.com test test2 test3;1;1;1;Aj;test4 Passing to :rtucker.
Assignee: ashish → rtucker
Component: Server Operations → Infrastructure: Monitoring
Product: mozilla.org → Infrastructure & Operations
QA Contact: shyam → jdow
https://github.com/rtucker-mozilla/mozilla-nagios-bot/pull/21 Fixes the parsing of the hostname field to not accept spaces in hostnames.
The code has been merged and pushed.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
[13:58] < Aj> | nagios-scl3 ack pxe-vm.private.scl3.mozilla.com Staying down [13:58] <nagios-scl3> | Aj: acknowledged host pxe-vm.private.scl3.mozilla.com Comment gets properly added. \o/
Status: RESOLVED → VERIFIED
Component: Infrastructure: Monitoring → Infrastructure: Other
You need to log in before you can comment on or make changes to this bug.