Closed Bug 1055783 Opened 10 years ago Closed 10 years ago

data isn't showing up in graphite for SUMO or Input

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

All
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: cliang)

References

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1076] )

An hour and a half ago, statsd data stopped showing up in graphite.

This shows a stats.sumo and a stats.input-prod. Both stopped showing data a little after 7pm (not sure what time zone that is, but it's about an hour ago):

https://graphite-phx1.mozilla.org/render/?width=586&height=308&_salt=1408480964.764&target=stats.input-prod.response.200&target=stats.sumo.response.200

Is the statsd data passing from SUMO/Input production to where it gets gathered?

For Input and SUMO, it's going from the Input/SUMO webhead, celery and admin nodes to 10.8.75.30 port 8125 via UDP.

Is it moving from where it gets gathered to Graphite?

This affects our dashboards and ability to monitor the site for problems, but we can probably wing it for a bit.
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1076]
Assignee: server-ops-webops → cliang
Ugh.  My apologies.  TL,DR; this should be working again.

The statsd destination host changed to graphite6.private.phx1.mozilla.com and, accordingly, I pushed out the changes Input, SUMO, etc.  It looks like there was a missing ACL.  NetOps quickly took care of it and I can see fresh data landing in graphite for both SUMO and Input responses.
Status: NEW → RESOLVED
Closed: 10 years ago
Depends on: 1056255
Resolution: --- → FIXED
I checked the graphs and they're showing data now. Thank you!
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.