909073 - bagheera ganglia monitoring

Status: RESOLVED FIXED

(Infrastructure & Operations :: Infrastructure: Other, task)

Description

[Harsha [:harsha]]

Hi,
  For bagheera nodes nagios alerts currently we have threshold 0 on bagheera.http.telemetry.requests.1MinuteRate.  We only get alerts only if the rate gets to 0. Can we make this to get an alert if the rate gets below 10.
Thank you,
Harsha

Comment 1

[Justin Dow [:jabba]]

Rick: is this something you can set up? Thanks!

Comment 2

[Harsha [:harsha]]

since telemetry data point going away. can we make this alert from bagheera.http.metrics.requests.1MinuteRate and if the request rate goes below 40 have a alert triggered.
Thanks,
Harsha

Comment 3

[Rick Bryce [:rbryce]]

(In reply to Harsha [:harsha] from comment #2)
> since telemetry data point going away. can we make this alert from
> bagheera.http.metrics.requests.1MinuteRate and if the request rate goes
> below 40 have a alert triggered.
> Thanks,
> Harsha

Sure.  Does 40 represent a critical alert, as in action needs to be taken?  We prefer to set warning and critical thresholds when we can.

Comment 4

[Harsha [:harsha]]

sorry I mean to say if the rate falls below 40 requests per sec not 40% of requests. 40 requests per sec is not critical but its something we as a dev/engineer might need to look into it. So you can put it as warning.
Thanks,
Harsha

Comment 5

[Rick Bryce [:rbryce]]

I didn't realize at the time, these alerts only go to the devs.  This is configured the same way the other bagheera checks are configured, with one threshold. Alert will now notify @ below 40 qps.