866206 - Ponder our payments flow

Status: RESOLVED FIXED

(Marketplace Graveyard :: Payments/Refunds, defect, P2)

Description

[Wil Clouser [:clouserw]]

Attachment: noc alerts

We provide /services/monitor end points for nagios to watch for problems with our sites.  This bug is to:

1) Close your email, IRC, and other things that distract :)
2) Step through the entire payments flow from a technical perspective
3) Write down things we should be monitoring in /services/monitor (any API connections, any required libraries, or files, or permissions, etc.)
4) Write down any critical points which we should be monitoring in the NOC.  I'm attaching a text file of the ones we've thought of so far.  If you have more, write them down.
5) File dependent bugs for everything you found in step 3
6) Tell Wil about anything you found in step 4

Thanks!

Comment 1

[Kumar McMillan [:kumar]]

I like this! I'd like to do the same thing from a statsd perspective so I stole your awesome bug format: bug 866942

Comment 2

[Andy McKay]

Step 3) testing database writes and celery working are not tested in our monitor. As long as Nagios monitors these they are fine. Chatting with clouserw in IRC you didn't seem to happy on adding those to our existing monitor end points.

Step 4) let's add in:

Webpay - Marketplace communication
Webpay - Solitude communication
Solitude - Bango communication

Any of these would cause failure but can be handled internally.

If failure to connect to memcache is an issue for step 4, I'll raise that each of the following is an internal failure: failure to connect to the write db (for marketplace, webpay and solitude), failure to connect to celery (for webpay and solitude)

Comment 3

[Andy McKay]

See https://etherpad.mozilla.org/payments-flow-monitoring

Comment 4

[Andy McKay]

When the nagios monitoring goes in, I think this will all be done.