1372999 - measure and monitor size of documents being saved to elasticsearch

Status: RESOLVED FIXED

(Socorro :: Processor, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Things keep getting added to the crash reports and some values have been changing in size over time. We have problems with Elasticsearch and storing all these things, but we have no visibility into whether document sizes are changing over time and if so, how.

This bug covers adding:

1. metrics generation (probably a datadog statsd histogram) for processed crash document size in the elasticsearch crashstorage

2. adding a graph to the datadog dashboard so we can monitor median, 95%, and max  sizes

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3816: capture raw_crash and processed_crash document sizes being saved to elasticsearch

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/d076f3cbdd7916b41cd5702b77cb949ffadd7508
Fixes bug 1372999 - add document size metrics for elasticsearch crashstorage (#3816)

* Add metrics infrastructure

This adds a new metrics infrastructure that's parallel to the existing logging
infrastructure and is available through config.metrics .

Why? Because it makes it much much much easier to thread metrics throughout the
app without adding more tangled component infrastructure and/or detangling the
existing stuff.

This is intentionally a minimal implementation. If we need more, we can break it
out into a larger thing at that point.

* Fixes bug 1372999 - Add document size metrics for Elasticsearch

This change captures raw_crash and processed_crash sizes for Elasticsearch so we
can see median, 95%, and max over time and better plan Elasticsearch changes.

* Fix nits

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Reopening this because we need dashboards still. I need to wait for data to show up in Datadog before I can do that.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

I checked -stage and setup_metrics didn't pick up a host, so it's using LoggingMetrics.

I'll figure out what's going on and fix the code accordingly and do a new PR.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3819: fix configman options

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/92c0bd5d59e422c4603671cb265b0294ec71c84b
Fixes bug 1372999 - fix statsd host/port configuration (#3819)

This fixes the configman options for statsd_host and statsd_port so they look in
the right place for values.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I set up a dashboard for -stage.

When this goes to -prod tomorrow, I'll set up a dashboard there, too.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

I added graphs to -prod.