1241412 - monitor WLCs and APs

Status: RESOLVED WONTFIX

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[Van Le [:van]]

This will be the first request - to monitor the uptime of our wireless devices. I'll open more service requests as I work through the kinks of this one.

Internal Service/System or External Services:
Internal Service

Name of Service:
Wireless Uptime Monitoring (?)

Scenario:
packet loss or hard down

Event Source: 
Splunk or Nagios


What the requesting team will be responsible for:
NetOps will check AP/WLC availability, servicing it through config changes, reboots, or collecting logs for support or hardware replacement.

What the MOC will be responsible for:
Paging NetOps

Any relevant SLA or other metrics:
Page NetOps if WLC is down for more than 2 minutes or AP is down for more than 10 minutes

Escalation and Support plan:
Escalate to NetOps

Deliverables and timelines for the request:
N/A. I'm out of the office until 2/8.

Runbook  (Runbooks definition) templates will be provided and jointly authored by a MOC representative and a representative of the requesting team:
I have instructions how to resolve and collect logs. Will work with MOC to make this presentable with the proper template.
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=37161365#JuniperWireless%28Trapeze%29-APissues

Comment 2

[Ryan C [:ryanc] (UTC-4)]

Van,

I'd be interested in the data from Splunk and what that looks like. In bug 1078823 it was determined that there was no real elegant check to do this with Nagios.

Please provide some data and some details of how I can access this to start trying some things.

Comment 3

[Peter Radcliffe [:pir]]

This might be better done in next gen monitoring where we can alert on more complex queries (like if uptime stays below X seconds for a period of time - an AP is rebooting frequently).

I'd really like to see alerting of WAPs auto-open netops bugs rather than requiring the MOC to take action for every alert on something we can't fix.

Comment 4

[Ryan C [:ryanc] (UTC-4)]

Peter,

Sounds good to me.

How can I assist in heading in that direction?

Comment 5

[Peter Radcliffe [:pir]]

We can add some WAPs to the testing/evaluation phase for next gen monitoring.
:jlaz?

Comment 6

[Justin Lazaro [:jlaz] (use needinfo)]

I agree with pir.  We'll be adding WAPs to our Prometheus/Datadog project for our Q1 evaluation.  In the meantime, we can set these alerts to page NetOps in PagerDuty, and in the future, auto-file a bug in the NetOps queue.

Comment 7

[Peter Radcliffe [:pir]]

We can't realistically directly page netops for these, they don't have people on 24/7 other than for emergencies and a WAP rebooting or being down is certainly not an emergency. We also get a lot of false positives on WAPs currently because of office connectivity flaps so we'd have to make UNKNOWNs not page. Lots of uncertainties here.

:van when you say "oage netops" I assume you mean "escalate to netops"? I doubt you want us to wake someone up for one WAP being down. The controller, maybe...

Comment 8

[Van Le [:van]]

going to 302 to :jbarnell on the urgency of these APs. when i was out on PTO, i was hit up for a wap302.par1 as it was causing issues and needed urgent remedy. 

should we page out for APs being down longer than X minutes?

Comment 9

[James Barnell [:jbarnell]]

yes

Comment 10

[Peter Radcliffe [:pir]]

What is X minutes? We already page on WAPs being down...

Comment 11

[Ryan C [:ryanc] (UTC-4)]

Van,

As mentioned in comment 0, where does Splunk come in here? Are you guys planning to get Splunk integrated with PagerDuty?

Be more concise on what you want, besides Nagios ping checks for WAPs and WLCs.

Comment 12

[Van Le [:van]]

:ryanc, i was planning to use splunk because you guys had indicated the old nagios wasn't going to make it easy to monitor the individual APs but since the new nagios can do that, we can skip that step for now. if we do use splunk, we'd use it to monitor excessive retransmits as that can also be an issue but that can be for another bug. i believe splunk is integrated into pagerduty, but i'll have to ask to dave or arzhel as they've had experience working on this.

>What is X minutes? We already page on WAPs being down...

thanks!

per bug 1078823, this was indicated as being hard for the old nagios implementation.

1. 5 minute polling interval
2. If more than 2 APs are down for 2 hours in an individual office, page to open a low priority bug (perhaps group per-office APs as Nagios cluster checks?)
3. If an (1) AP has been down for 12 hours, or rebooted more than 4 times, page to open a low priority bug

we can skip #2 as we want to alerted if an AP doesn't come back up within 5 minutes. wireless is pretty critical in our office so we'd want to know ASAP if the AP doesn't come back.

#3 would be helpful for trending and/or will allow us to see if there's a bigger issue at play.

Comment 13

[Ryan C [:ryanc] (UTC-4)]

(In reply to Van Le [:van] from comment #12)
> 1. 5 minute polling interval
> 2. If more than 2 APs are down for 2 hours in an individual office, page to
> open a low priority bug (perhaps group per-office APs as Nagios cluster
> checks?)
> 3. If an (1) AP has been down for 12 hours, or rebooted more than 4 times,

So,

Going 1 by 1,

1. Nagios already does this.

2. This could MAYBE be a cluster check, but I'd have to look at how one of our existing cluster checks is done, such as DNS.

3. This is where Nagios is not good, once you pass intervals of hours, or when you want to check occurrences, since there's no way for a script to run once and check if it's been rebooted x times. Unless it looked at the AP log via SNMP or something.


I'm not sure what you're looking at in Splunk, or how you can set conditionals to alert like that in Splunk, but if you can I would be interested to see, then we can interface that to Pagerduty, somehow.

Comment 14

[Peter Radcliffe [:pir]]

(In reply to Ryan C [:ryanc] from comment #13)
> 3. This is where Nagios is not good, once you pass intervals of hours, or
> when you want to check occurrences, since there's no way for a script to run
> once and check if it's been rebooted x times. Unless it looked at the AP log
> via SNMP or something.

The best way I can think of to implement this with current services is to write a new check that looks at the uptime of an AP and keeps local (nagios server side) state on when the uptime tripped back down to zero (ignoring times when it can get no information) and then some logic on that as to how many have happened in the last X time per AP.

This is non-trivial, bad practice for nagios checks and would have no data for multiple resets of APs during connection issues between scl3 and offices.


The request for "packet loss" monitoring we already do to the best of the abilities of where we have nagios servers.

AP monitoring in general is a mess. We can't do realistic short notification time alerts on what we currently have without having a huge amount of noise because all the monitoring to offices is done in scl3 so any network issues between scl3 and the offices throw everything off and even with correct dependencies we get noise when the nagios server rechecks the dependencies before the APs after a network outage. A lot of noise is bad for everyone, it makes us busier and adds to alert fatigue.

There isn't a good solution here and we can't just make it better without a lot of work. Next gen monitoring (there is no "new nagios", I'm talking about prometheus, datadog, etc) can alert on more complex equations and data is treated differently/better so this becomes much more feasible but that won't be ready for production until next quarter at least.

I'm not convinced about the sense in putting a lot of effort into improving what we do with tools that don't fit our needs right now.


The wireless management boxes must have better information about if the APs are up or down and when and since they're in the same location as the APs that gets rid of remote connection problems. Is there anything useful exposed there?

Comment 15

[Ryan C [:ryanc] (UTC-4)]

Tossing this since this needs a bit more investigation to determine the best way to approach monitoring.

CC'ing those who I think should followup.

Comment 16

[Ryan C [:ryanc] (UTC-4)]

Van,

As Peter mentioned in comment 14.

Is there anything useful we can pull from the wireless controllers? Otherwise we'll WONTFIX this.

Comment 17

[Van Le [:van]]

>The request for "packet loss" monitoring we already do to the best of the abilities of where we have nagios servers.

this is what the original request was originally for. if there is an escalation path in place, then awesome.

> Next gen monitoring (there is no "new nagios", I'm talking about prometheus, datadog, etc) can alert on more complex equations and data is treated differently/better so this becomes much more feasible but that won't be ready for production until next quarter at least.

i think we can look at this again next quarter then. we also have a wireless rfp where the next gen wireless platform should have better reporting/logging capabilities as well.