781372 - add stats.timers.syncstorage.controller.* to sync dev / stage pencil dashboard

Status: VERIFIED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Rob Miller [:rmiller]]

Now that metlog is being permanently deployed to dev, stage, and (soon) prod sync, we're gathering elapsed time for request processing on each request that we service from within the application. This data is currently being sent to statsd and is showing up as `stats.timer.syncstorage.controller.<CONTROLLER_METHOD_NAME>`. It would be great to get these to show up on the pencil syncstorage_overview dashboard for both the dev and stage environments.

A separate bug will be opened for prod once the code has made it through testing and is pushed to production.

Thanks!

Comment 1

[Rob Miller [:rmiller]]

Actually, looking at this further, I can see that the stats are showing up in the graphite UI, but when I drill down to look at them I'm not actually seeing any data. There may need to be some debugging related to the application -> logstash -> statsd data flow to make sure that the data is being collected and is landing correctly.

Comment 2

[Wesley Dawson [:whd]]

Apologies for the ridiculous turn-around.

I added the following graphs to stage with mean and upper 90%% metrics:

http://pencil.scl2.stage.svc.mozilla.com/dash/stage/syncstorage#syncstorage_request_time_delete
http://pencil.scl2.stage.svc.mozilla.com/dash/stage/syncstorage#syncstorage_request_time_get
http://pencil.scl2.stage.svc.mozilla.com/dash/stage/syncstorage#syncstorage_request_time_set

However, those graphs are somewhat crowded. In addition, the full list of metrics is:

delete_collection
delete_item
delete_storage
get_collection
get_collection_counts
get_collection_usage
get_collections
get_item
get_quota
set_collection
set_item

Rob, how would you suggest breaking down the above metrics into graphs?

Comment 3

[Ryan Kelly [:rfkelly]]

I think the three most important metrics from this list are get_collections, get_collection and set_collection.  These operations make up the bulk of a sync session, so it would be useful to have each split out into its own graph.

The remaining ones are useful but probably don't need to be visible up-front, it there's a way to drill down to a more detailed view with all of them then I think that would work well.

Comment 4

[Wesley Dawson [:whd]]

We now have two dashboards, syncstorage_overview and syncstorage_details.

Comment 5

[James Bonacci [:jbonacci]]

Why yes we do.