1223750 - New Relic MySQL plugin not reporting stats for Treeherder's stage DBs since Oct 30th

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Ed Morley [:emorley]]

Around about 22:00 UTC on 30th Oct, New Relic stopped showing stats for the Treeherder DB nodes, eg:

https://rpm.newrelic.com/accounts/263620/dashboard/5642686?tw%5Bend%5D=1447197926&tw%5Bstart%5D=1446039025

The nodes in question are:
treeherder1.stage.db.scl3.mozilla.com
treeherder2.stage.db.scl3.mozilla.com

As I understand it, the process that generates these stats runs on a shared machine that polls all of the DBs at Mozilla.

The prod Treeherder DBs are still reporting, eg:
https://rpm.newrelic.com/accounts/263620/dashboard/3942659

Any idea why these have stopped reporting? :-)

Comment 1

[Ed Morley [:emorley]]

(Bah that default security group gets me every time, it's so annoying)

Comment 2

[Pythian Team73]

Hello Ed,

I checked the logfiles of newrelic, but they are not too informative (empty).
I restarted newrelic agent to see if that helps, but since we dont have access to newrelic, I cannot verify if it solved it. Could you please check it?

Thanks
Janos (jruszo)

Comment 3

[Ed Morley [:emorley]]

Hi Janos

Thank you for doing that. Unfortunately there is still no data appearing in the dashboard.

To clarify - there are two places data is recorded:
1) A server system monitor on each DB node (which we are receiving data for - to the A-Team account)
2) The MySQL new relic plugin - that runs on a single admin node somewhere and scans all Mozilla DBs (that reports to the Mozilla General New Relic account)

Was it #2 that was restarted?

Comment 4

[Pythian Team73]

Thanks for clearing this for me Ed.
Unfortunately, I was able to restart only the 1) part, on the DB node.
I dont have info on #2, Maybe Sheeri or Matthew could help

Comment 5

[Sheeri Cabral [:sheeri]]

We cleaned up our configs a bit - newrelic charges us for each machine (even for the free plugins, we are working on fixing this) so we took out all our stage servers.

It sounds like you're actually using newrelic on treeherder stage, so we can put it back....tomorrow when the change freeze window ends. Sound good?

Comment 6

[Ed Morley [:emorley]]

Sounds good, thank you :-)
(I tried searching for bugs to see if something like that had happened, but came up empty - guessing they are private?)

Comment 7

[Sheeri Cabral [:sheeri]]

Yeah, infrastructure changes are generally marked private, and most people aren't using newrelic on their stage instances. Clearly, you are, and that's a GOOD thing, so we want to encourage that.

Comment 8

[Sheeri Cabral [:sheeri]]

Added back in, and graphs are reporting numbers.

Comment 9

[Ed Morley [:emorley]]

Looks good - thank you :-)