869716 - Nagios payments checks

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task)

Description

[Andy McKay]

For payments, just wanted to double check we've got all the following nagios checks in place. I bet they are in which case just close this bug quickly :)

marketplace monitor end point (services/monitor) - already got this
marketplace writeable database
marketplace celery
webpay monitor end point (https://marketplace.firefox.com/mozpay/services/monitor)
webpay writeable database
webpay celery
solitude db server monitor end point (<solitude may be accesible to nagios at some url>/services/monitor)
solitude proxy server monitor end point (<solitude proxy may be accesible to nagios at some url>/services/monitor)
solitude writeable database

Comment 1

[Jeremy Orem [:oremj]]

marketplace/webpay celery:
  * do you have any preference on how we monitor this?

Comment 2

[Andy McKay]

I was going to suggest that the app writes into celery and waits for a result. We haven't done this from marketplace though, presumably because of scale. Can we monitor rabbit/mq and look for traffic spike/drops?

Comment 3

[Jason Thomas [:jason]]

(In reply to Andy McKay [:andym] from comment #0)
> solitude db server monitor end point (<solitude may be accesible to nagios
> at some url>/services/monitor)

This returns a solitude error, or should we be using /services/status?

Comment 4

[Andy McKay]

(In reply to Jason Thomas [:jason] from comment #3)
> (In reply to Andy McKay [:andym] from comment #0)
> > solitude db server monitor end point (<solitude may be accesible to nagios
> > at some url>/services/monitor)
> 
> This returns a solitude error, or should we be using /services/status?

Sorry, yes, /services/status/

Comment 5

[Jason Thomas [:jason]]

(In reply to Andy McKay [:andym] from comment #2)
> I was going to suggest that the app writes into celery and waits for a
> result. We haven't done this from marketplace though, presumably because of

> Can we monitor rabbit/mq and look for traffic spike/drops?
Not at the moment. We can monitor based on predefined thresholds.

Currently we monitor marketplace prod and amo queues for unacknowledged messages and alert if it is above a certain threshold. The threshold is set to aggressive value, it has been effective in finding issues with rabbitmq and celery in the past. We can configure this for webpay queues. How does that sound?

Comment 6

[Andy McKay]

Sounds great, let's do that.

Comment 7

[Jason Thomas [:jason]]

I've added nagios monitoring for webpay queues. All other payments checks are in place.