933280 - fix statsd for Input

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Pretty sure Input statsd calls went to graphite at some point, but when I go to:

https://graphite-phx1.mozilla.org/

I don't see anything for Input.

Can someone help me figure out why Input statsd calls aren't showing up in graphite?

I need to be able to see this sort of thing as I'm fiddling with the forms to reduce spam and need to be able to track related metrics.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bumping this up to major. I'm really flying blind metrics-wise right now and I'm having a problem in production that I can't look into without statsd data.

Comment 2

[Adrian J Fernandez [:Aj]]

Will for which hosts are you searching for?

Dropping priority so oncall doesn't get paged.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't understand the question.

I want to see the statsd information that Input -prod and -stage are sending in https://graphite-phx1.mozilla.org/ .

Something must be configured wrong somewhere because there's no evidence of Input data in that graphite instance that I can see. So either the servers are configured wrong (a setting in fjord/settings/local.py is wrong perhaps?) or there are flows not set up or I'm looking at the wrong graphite server or graphite isn't configured to show Input data or something I haven't thought of.

Does that answer your question? If not, can you clarify your question?

Comment 4

[Chris Turra [:cturra]]

i double checked the application config (fjord/settings/local.py) and everything looks correct there. additionally, i am seeing active data in graphite for the input nodes, tho i am not entirely certain we're looking at the same graphs.

an example graph i am looking at is: Graphite > hosts > input1_webapp_phx1_mozilla_com > apache > apache81 > apache_requests > count

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Funky, but that's not what I'm looking for. I'm looking for application-level statsd data. It should be in Graphite -> stats -> input-prod. For SUMO, it's in Graphite -> stats -> sumo.

For SUMO, the STATSD_HOST is set to '10.8.75.93' while for Input, it's set to '10.8.74.135'. Is it possible the Input one is pointing to a different graphite/statsd whatever (I have no idea how statsd works)?

Comment 6

[Chris Turra [:cturra]]

:willkg - as discussed on irc, i have opened a netflow bug so we can update the settings similar to sumo.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I'm still puzzled as to why server-level stuff is making it to graphite (cpu, memory, ...) but not application-level stuff.

Comment 8

[Rick Bryce [:rbryce]]

(In reply to Will Kahn-Greene [:willkg] from comment #7)
> I'm still puzzled as to why server-level stuff is making it to graphite
> (cpu, memory, ...) but not application-level stuff.

server stats(cpu, mem, etc) are handled by CollectD. As for statsd, I/we are only maintaining a the nodeJS service that collects statsd formed metrics in turn hand-off to graphite for storage in whisper files.

Comment 9

[Chris Turra [:cturra]]

now that the ACL is sorted, i have updated the fjord/settings/local.py file to reflect the new graphite server and we're back in business :)

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Awesome! I'm seeing data in graphite now. Thank you!