1454482 - capture timings for elasticsearch indexing

Status: RESOLVED FIXED

(Socorro :: General, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

We want better timing data for indexing documents in Elasticsearch as well as successes vs. saves. It's not sufficient to use the save_raw_and_crash benchmark wrapper since that method does a bunch of things--we just want indexing timings.

This bug covers wrapping the key parts of the elasticsearch crashstorage to cover that data.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Brian did a first pass here: https://github.com/mozilla-services/socorro/pull/4391

However, after that, I redid metrics so we can have MetricsInterfaces with the right namespace, so we should redo that PR.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Brian said this would be helpful, so I'm making it a P2 to look into.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Tossing this at Alexis. Follow up with Brian Pitts about what would be helpful to have timings on.

Comment 4

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/4a5a1dac5857d1ae4ac7eac5757df1d19c980cb5
bug 1454482: add timing captures for ES

- add timing data for document indexing in elasticsearch
- add timing data for submissions to S3

https://github.com/mozilla-services/socorro/commit/23082b528e7958572b51737844b98f84c0bf44dc
Merge pull request #4493 from mozilla-services/elasticsearch-indexing

bug 1454482: add timing captures for ES

Comment 5

[Brian Pitts]

Alexis, do you see your new metrics in Datadog? I think they should be coming from the stage processor, but I don't see them.

Comment 6

[Alexis Deschamps [:alexisdeschamps]]

:bpitts I looked and I think there was an issue with the build. If the automatic deploy to stage is working, then now the new metrics should be there.

Comment 7

[Brian Pitts]

I see the metrics in stage now!

There was one unexpected (to me) outcome. We have two different metric names for uploading to S3. Some of them are under 'processor.s3' and others are under 'processor.telemetry'. I suspect that has something to do with how the namespace argument is set when the RegionalS3ConnectionContext classes are initialized by BotoS3CrashStorage and TelemetryBotoS3CrashStorage, but I don't understand how all that hangs together.

https://app.datadoghq.com/metric/summary?filter=processor.s3.submit
https://app.datadoghq.com/metric/summary?filter=processor.telemetry.submit

Comment 8

[Alexis Deschamps [:alexisdeschamps]]

I think I change the namespace names to make the metrics all use the same name. Should it be `processor.s3` or `processor.telemetry`?

Comment 9

[Brian Pitts]

It should be `processor.s3`

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/6bf4a0276f8b2045624e51a1afb7b82e66fda340
fix bug 1454482: fixed crash index metrics

`BotoS3CrashStorage` and `TelemetryBotoS3CrashStorage` currently give
their own namespace to their `ConnectionContextBase`'s `connection_context`.
`ConnectionContextBase` uses the passed namespace for its metrics.
This results in different metrics prefixes for submisisons to S3.
This PR changes that by setting the desired namespace name given to
`connection_context`.

https://github.com/mozilla-services/socorro/commit/5a2f240a844533c59ce08528c53f9fa9ab5a79ef
fix bug 1454482: changed metrics prefix source

The metrics prefix for S3 submissions was previously using an extra passed-in variable.
This change makes it use configman to set the metrics prefix.

https://github.com/mozilla-services/socorro/commit/99686be92cca0881cbd374a4138fd7c3923fd804
Merge pull request #4509 from mozilla-services/metrics-namespace

fix bug 1454482: fixed crash index metrics