1143099 - Add newrelic monitoring to mozreview prod

Status: RESOLVED FIXED

(MozReview Graveyard :: General, defect, P1)

Description

[Mauro Doglio [:mdoglio]]

We should at least monitor the web activity. Not sure how easy it is to enable it for the other services

Comment 1

[Ed Morley [:emorley]]

I'm not sure how familiar you are with New Relic app/plugin reporting (so ignore this if you already know it), but having had the 'delight' of trying to figure out some of Treeherder's New Relic issues recently, New Relic docs/configs are fresh in my mind.

I tagged the ReviewBoard server stats reporting entries a bit ago to make them more easily findable:
https://rpm.newrelic.com/accounts/263620/servers?filterBy=Project:Reviewboard

There also appears to be reporting already set up for Reviewboard's MySQL DBs (if it's anything like for Treeherder, they have a central node that checks all DB nodes):
https://rpm.newrelic.com/accounts/263620/plugins/2805?search[name]=reviewboard

That just leaves APM and plugins other than MySQL.

Docs for the APM Python client:
https://docs.newrelic.com/docs/agents/python-agent/getting-started/python-agent-quick-start
https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-installation
...though IT has a stock config available, so this should be easy to set up. (Treeherder has forked the config slightly, to allow for bug 1141036).

There are a few apache/memcached plugins available; the ones Treeherder uses are:
https://rpm.newrelic.com/accounts/677903/plugins/directory/53
https://rpm.newrelic.com/accounts/677903/plugins/directory/59

They require installing the MeetMe newrelic-plugin-agent:
https://github.com/MeetMe/newrelic-plugin-agent#installation-instructions
...and then enabling each of the plugins via the plugin-agent's config file.

Treeherder's plugin-agent config for reference, is https://pastebin.mozilla.org/8825816

Hope that helps :-)

Comment 2

[Mauro Doglio [:mdoglio]]

I set up the NR python agent on dev, you can find it here https://rpm.newrelic.com/accounts/263620/applications/7053639
Since we are using mod_wsgi I had to modify the wsgi file and do the activation via the newrelic python library.
:fubar what's the story on prod for it? :edmorley mentioned a IT-provided config in comment 1, is that compatible with what we have?

Comment 3

[Kendall Libby [:fubar] (he/him)]

::yak shaving intensifies::

app name is now set in httpd.conf, since the puppet module would otherwise call it 'Python application'. prod is Reviewboard, dev is Reviewboard-dev.

prod: https://rpm.newrelic.com/accounts/263620/applications/7083195
dev: https://rpm.newrelic.com/accounts/263620/applications/7083355

also, prod uses /data/wrapper.wsgi, so added the newrelic bit there since that's ours and I'm concerned that the reviewboard.wsgi will get wiped on an upgrade someday. /data/wrapper.wsgi exists on dev, but I don't think we're using it; I don't recall why.

plugins...
dev memcache: https://rpm.newrelic.com/accounts/263620/dashboard/7083590
dev httpd: https://rpm.newrelic.com/accounts/263620/dashboard/7083591
prod memcache:
https://rpm.newrelic.com/accounts/263620/dashboard/7083639
https://rpm.newrelic.com/accounts/263620/dashboard/7083703
prod httpd: 
https://rpm.newrelic.com/accounts/263620/dashboard/7083640
https://rpm.newrelic.com/accounts/263620/dashboard/7083704

shout if anything's amiss or you need something else added.

Comment 4

[Ed Morley [:emorley]]

I've also added the Project:Mozreview tag to the apps:
https://rpm.newrelic.com/accounts/263620/applications?filterBy=Project:Mozreview

Which matches that of the servers:
https://rpm.newrelic.com/accounts/263620/servers?filterBy=Project:Mozreview

And created an application alert policy for each of prod/dev (it doesn't email anything yet; I'll leave that for you to set up if wanted):
https://rpm.newrelic.com/accounts/263620/application_alert_policies?search[q]=reviewboard

I also used the "move config to new relic" option here (plus for dev):
https://rpm.newrelic.com/accounts/263620/applications/7083195/edit

And made sure error collection/tracing etc was enabled and a suitable threshold set (apdex_f).

I also set a URL for availability monitoring:
https://rpm.newrelic.com/accounts/263620/applications/7083195/ping_targets

...used "https://reviewboard.mozilla.org/api/" for now, please tweak as needed.

Comment 5

[Mauro Doglio [:mdoglio]]

Many thanks to both of you!