1386752 - appsvcs-voice prod alerting for ApacheSentKbSpike

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[Salvador Espinoza [:sal]]

Got this one in

[FIRING:1] PrometheusAlert appsvcs-voice prod us-west-2 (058419420086 firing us-west-2a ApacheSentKbSpike i-008f3a5f3e04f3df3 t2.small apache prometheus voice critical [e-mail] meta-failure)     

        Labels:
 - alertname = PrometheusAlert
 - account = appsvcs-voice
 - account_id = 058419420086
 - alertstate = firing
 - availability_zone = us-west-2a
 - environment = prod
 - federated_alertname = ApacheSentKbSpike
 - instance = i-008f3a5f3e04f3df3
 - instance_type = t2.small
 - job = apache
 - monitor = prometheus
 - project = voice
 - region = us-west-2
 - severity = critical
 - technical_contact = [e-mail]
 - type = meta-failure
Annotations:
 - description = Prometheus alert ApacheSentKbSpike for appsvcs-voice
 - summary = Prometheus alert ApacheSentKbSpike for appsvcs-voice
Source: https://mon.prod.us-west-2.moc-prometheus-sandbox.nubis.allizom.org/prometheus/graph?g0.expr=%28ALERTS%7Balertstate%3D%22firing%22%2Cplatform%21%3D%22nubis%22%2Ctype%21%3D%22meta-failure%22%7D+%3D%3D+1%29+and+%7Benvironment%3D%22prod%22%7D&g0.tab=0
 
          num_firing

      

        1
 
          num_resolved
          	
      

        0

Comment 1

[Keegan Ferrando [:fauweh]]

Taking this bug back. This appears to be a default check that came with Nubis Prometheus and the application owners have not specifically asked for this check.

We also likely need documentation for this check as well.

Not Fully Baked.

Comment 2

[Ashlee Nguyen [:ashlee]]

Alerted this morning at 12:49 AM

[FIRING:1] PrometheusAlert appsvcs-voice prod us-west-2 (058419420086 firing us-west-2a ApacheSentKbSpike i-008f3a5f3e04f3df3 t2.small apache prometheus voice critical [e-mail] meta-failure)

Comment 3

[Justin Lazaro [:jlaz] (use needinfo)]

I've silenced the alert to avoid paging oncall until we decide if this check should remain enabled or removed:

Silence

ID
ee440635-97b5-46bb-8693-f008e15c750e

Starts at
2017-08-07 05:41:07

Ends at
2017-08-21 05:40:42

Updated at
2017-08-07 05:41:07

Created by
jlaz

Comment
https://bugzilla.mozilla.org/show_bug.cgi?id=1386752

Comment 4

[Keegan Ferrando [:fauweh]]

We have removed this check as part of the migration to SCL3, it was not found to be useful by current service owners.