809036 - Please enable service monitoring for Petri apps

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Labs, task)

Description

[Simon C. Wex]

These two services are running on Petri. 

clopenbadger:
https://badges.webmaker.org/

hackpub:
https://hackpub.vcap.mozillalabs.com/

Please send alerts to [e-mail].

A couple questions:

Are there alternative methods of alerting? (SMS?)
Hackpub only responds with a 404 unless you're POSTing to it and creating content. Is that satisfactory for monitoring?

Thanks, Simon

Comment 1

[C. Liang [:cyliang]]

Just a quick note:  

This may take a little time because it looks like it would make the most sense to bring up an instance of Shinken (the monitoring we're using for other parts of Labs) within the same infrastructure as Petri.  Some of the details on how to engineer monitoring of self-serve projects within Petri still need to be ironed out. 

We currently can't support SMS directly (as we don't have an SMS modem to hook up to anything).  We *can* send email messages to an SMS gateway.

Comment 2

[Simon C. Wex]

Cool, do you mean getting it on CloudFoundry, or just in the same location, etc.

Is there anything I can do to help?

Comment 3

[C. Liang [:cyliang]]

In the spirit of Petri being self-serve, I've been working on setting up Petri monitoring so that it should be largely "self-serve".  Things aren't yet completely set up where (network-wise) that I'd like it to be, but I think enough pieces are in place to let you loose.  =)  [1] 

I'd like to see if you can successfully create and edit a "host" entry for hackpub.vcap.mozillalabs.com into the system.  The instructions outlined below are kind of long; if you run into issues, we can hash it out via bugzilla or IRC (whichever is easiest for you).

The monitoring system is available at:
    https://monitor-petri.mozillalabs.com/petri/check_mk/

You will be prompted for a username and password.  Use your LDAP credentials (i.e. <user>@mozilla.com). 

In the left-hand sidebar, there should be a square labeled "WATO * Configuration".  Click on "Hosts & Folders".  In the main area of the window, you should now see a "Petri" folder.  Click on it. 

I already added an entry for badges.webmaker.org.  You should be able to view and/or edit this entry.   It looks like the system interprets a 404 as a "warning" state (we got a response, but it's not a "successful" one.  For now, I've acknowledged the alert so it shouldn't send out any more messages about the warning state.  If there is a better test (i.e. maybe something that is safe to POST regularly), we can define a new check and retag this system.

For adding monitoring for hackpub.vcap.mozillalabs.com, it's probably easiest to clone this entry (by selecting the third icon in the "Actions" column -- the sheet with the yellow gear in the lower right-hand corner).  This brings up a page where you can edit the cloned entry.  

* Hostname 
Enter the fully qualified domain name (i.e. badges.webmaker.org)

* Permissions
Groups can be used to control who can edit the monitoring configuration for this host.  On the main labs monitoring  service, this group roughly corresponds to "project".  Since I wasn't sure exactly where hackpub fits in, I created both a "badges" and a "hackpub" group. 

As long as you don't uncheck the box labeled "Add these contact groups ...", the group that can modify the config is also the group that will receive notifications when the monitoring service sees issues.

* Agent type
If you end up creating more entries:
   - If you want in-depth  monitoring of a VM (i.e. CPU utilization, monitoring of mySQL, etc.), you will need to configure the VM to allow SNMP and/or install the check_mk agent. [2]
   - Otherwise, if you're monitoring a VM and it can respond to pings, select "only ping this device".
   - If you are monitoring a web address, select "No Check_MK Agent, no SNMP, no ping"

* Web services
   If it's a web host, you can select HTTP and/or HTTPS monitoring.  

Once your edit are done, hit the "Save and Finish" button.  (If you try to "Save & got to Services", the monitoring software will try to initiate an inventory scan of the host, which will fail.)

In the top array of buttons, there should be one that is now brown instead of blue.  It shows the number of changes that you've made that now need to be made active.  Click on that button.  You'll get a full list of changes.  Click on the brown "Activate Changes" button.  This will take a little while (20-30 seconds) since it causes a restart of the monitoring daemons.



[1] The biggest pieces that are not self-service (that I don't think I can change) are creation of monitoring accounts and configuring rules for how "hosts" get tagged for certain kinds of check.  I also haven't gotten a change to dig into any of the fun extras, like some of the visualization and log analysis hooks.  Some of these may not work without installing software agents. 

[2] If you want to do this, let me know.  If you want to use the check_mk agent, we should probably mirror it locally.  If you want to use SNMP, there's a specific community string that this server is set to use.

Comment 4

[Simon C. Wex]

Looks great to me.

The first time I tried to save the configuration for hackpub, I got this:

Inventory failed for this host: Error running sudo /bin/su - petri -c check_mk --automation try-inventory -- --cache hackpub.vcap.mozillalabs.com (exit code 2):
Cannot get data from TCP port 63.245.223.136:6556: [Errno 113] No route to host

The second time it worked fine though. 

As for permissions, I think a general MoFo or Mozilla Foundation group would be fine. 

This is great, thanks!

Comment 5

[C. Liang [:cyliang]]

Hrm.  I wonder if the "Save & Finish" -- and don't run the inventory -- trick only works when adding a host.  

RE: permissions

I've created a Mozilla_Foundation group.  (It looks like spaces are verboten in group names.)  You are currently the only member of that group.   

You should be able to add this group to host's permissions.  If you do so, you'll probably want to un-check the "Add these contact groups to the host's contact groups ..." box.  

CAVEAT: I have not experimented with "mixed-mode" permissions (where some groups have permission but aren't listed in the host's contact groups in the monitoring configuration.  Trying to do mixed permissions may result in no notifications going out, so you may want to test this.


As an FYI: there is a Petri group associated with the Petri folder.  Anyone who has the right to add or alter monitoring for Petri hosts will go here.  It should be possible to add other folders, as needed, if we need to segregate out MoFo services.