975614 - APK Production: Monitoring

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task, P1)

Comment 1

[Jeremy Orem [:oremj]]

Note that we need, at least, an http check for sentry.

Comment 2

[Jeremy Orem [:oremj]]

Added to PHX1 nagios:

https://controller.apk.firefox.com/controller/health

AWS monitoring:

https://generator.apk.firefox.com/generator/health
https://signer.apk.firefox.com/signer/health

http://sentry.shared.us-west-2.prod.mozaws.net/

Comment 3

[Jason Thomas [:jason]]

We should also monitor the review instances at:

https://controller-review.apk.firefox.com
https://generator-review.apk.firefox.com
https://signer-review.apk.firefox.com

For signer:
signer.apk.firefox.com/system/tools

Comment 4

[Jason Thomas [:jason]]

For controller:
/system/generator returns 203 on success

For generator: 
/system/signer returns 203 on success

Comment 5

[Jason Thomas [:jason]]

Added to PHX1 nagios:

https://controller.apk.firefox.com

AWS monitoring:

https://generator.apk.firefox.com
https://signer.apk.firefox.com

Comment 6

[Wesley Dawson [:whd]]

Some basic monitoring (controller sites and sentry http) is available here: https://opsview.shared.us-west-2.prod.mozaws.net/status/hostgroup?parentid=1 (web admin password is in hiera)

Comment 8

[Wesley Dawson [:whd]]

controller{,-review}.apk.firefox.com (which now respond to pings) have been added to PHX1 nagios. I'll be adding AWS monitoring presently for the other sites.

Comment 9

[Wesley Dawson [:whd]]

https://generator.apk.firefox.com
https://signer.apk.firefox.com
https://generator-review.apk.firefox.com
https://signer-review.apk.firefox.com

All these internal ELBs have AWS API ELB checks now that will alert if any hosts behind them are unhealthy, and the alerts should go to both pagerduty (APK escalation path) and amohubot.

Comment 10

[Wesley Dawson [:whd]]

Stackdriver<->Pagerduty is set up now such that any "PROD - APK Factory" health check failure (something turning red) in Stackdriver will send an alert Pagerduty. I can set up an SNS hook to add IRC notifications as well.

Comment 11

[Wesley Dawson [:whd]]

I misunderstood the meaning of the /system/signer and /system/generator checks. They are now implemented correctly at:

https://opsview.shared.us-west-2.prod.mozaws.net/cgi-bin/extinfo.cgi?type=2&service=APK+Controller+Generator+access&host=controller-review.apk.firefox.com
https://opsview.shared.us-west-2.prod.mozaws.net/cgi-bin/extinfo.cgi?type=2&service=APK+Controller+Generator+access&host=controller.apk.firefox.com

https://opsview.shared.us-west-2.prod.mozaws.net/cgi-bin/extinfo.cgi?type=2&service=APK+Generator+Signer+access&host=generator.apk.firefox.com
https://opsview.shared.us-west-2.prod.mozaws.net/cgi-bin/extinfo.cgi?type=2&service=APK+Generator+Signer+access&host=generator-review.apk.firefox.com

Opsview -> Pagerduty is having some issues when I try to segment the alerts so that only APK alerts go to APK escalation (and IRC); I'm addressing that presently.

Comment 12

[Wesley Dawson [:whd]]

I have fixed the Opsview escalation problem. Everything spec'd here is implemented, but we can always add more and tweak as needed. Closing this.