709338 - monitoring the sync cluster

Status: RESOLVED INVALID

(Web Apps Graveyard :: AppsInTheCloud, defect, P1)

Description

[Tarek Ziadé (:tarek)]

Here's the we need to make sure we have for the developer preview, 

- monitor the heartbeat page for appsync, every 10 s
- monitor the heartbeat page for the node.js server, every 10 s
- monitor the heartbeat page for every HBase node, every 10 s
 
in case of a timeout or non-200, send a mail to :

- <Bill's cell> (sent by email/in the ldap index) 
- [e-mail]

If possible (not a priority) on IRC on #openwebapps

nice to have:
- CPU/memory/fd monitoring with an alarm on low resource

Comment 1

[Bill Walker [:bwalker] [@wfwalker]]

if there's a web page showing the status of these pieces, i'll put it up on my 2nd floor MV kiosk...

Comment 2

[Philippe M. Chiasson (:gozer)]

(In reply to Tarek Ziadé (:tarek) from comment #0)
> Here's the we need to make sure we have for the developer preview, 

Exact urls would be nice ;-)
 
> - monitor the heartbeat page for appsync, every 10 s

http://appsync-stage1.vm1.labs.sjc1.mozilla.com/__heartbeat__

> - monitor the heartbeat page for the node.js server, every 10 s

http://sauropod-stage1.vm1.labs.sjc1.mozilla.com:8001/__heartbeat__ ??

> - monitor the heartbeat page for every HBase node, every 10 s

??

> in case of a timeout or non-200, send a mail to :
> 
> - <Bill's cell> (sent by email/in the ldap index) 
> - [e-mail]

Easy.

> If possible (not a priority) on IRC on #openwebapps

Will need to check with whoever is already running nagios IRC bots

> nice to have:
> - CPU/memory/fd monitoring with an alarm on low resource

Will do.

Comment 3

[Tarek Ziadé (:tarek)]

https://wiki.mozilla.org/Apps/ServerArchitecture will give you all the nodes urls

The URL is /__heartbeat__ for the two app servers, I don't know for the HBase server. ccing Ryan

Comment 4

[Ryan Kelly [:rfkelly]]

FYI, the sauropod __heartbeat__ page pings hbase and errors out if it's not reachable

Comment 5

[Tarek Ziadé (:tarek)]

I am not certain we want this e.g. if possible the HBase nodes should have their own heartbeat and the node one should be standalone, so we can tell which box is really down in the monitoring

Comment 6

[Ryan Kelly [:rfkelly]]

Fair enough.  The hbase rest server has a "cluster status" page which should be useful for this:

  http://appsync-hbase-stage1.vm1.labs.sjc1.mozilla.com:8080/status/cluster

Comment 7

[Philippe M. Chiasson (:gozer)]

Checks are now in place, and will send notifications to :

- <Bill's cell> (sent by email/in the ldap index) 
- [e-mail]
- Gozer's cell

Comment 8

[Bill Walker [:bwalker] [@wfwalker]]

awesome!

Comment 9

[Tarek Ziadé (:tarek)]

I have tried to shut down gunicorn then nginx, and we did not receive any mail.

Did you get the SMS ?

Comment 10

[Jason Smith [:jsmith]]

The old app sync codebase is no longer going to be supported. All resolved fixed bugs are being marked as invalid, as they no longer apply to the new apps in the cloud service.