1674406 - add metrics generation for eliot

Status: RESOLVED FIXED

(Eliot :: General, enhancement, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When writing Eliot, I added some rough metrics and I have the infrastructure for generating metrics implemented, but I didn't spend a lot of time thinking about what questions I wanted to answer with metrics and making sure I had metrics to generate data to answer those questions.

This bug covers that work.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2267: bug 1674406: reimplement metrics in Eliot and add new ones

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2267: "bug 1674406: reimplement metrics in Eliot and add new ones" in 4ba06a9.

I also redid metrics in Eliot so that they're auto-documented. It extends a proof-of-concept I did a while back. I figure we'll test it out here and if it works, I'll switch Tecken webapp to do the same thing. If it doesn't work, then we'll remove it.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2268: bug 1674406: fix version values in metrics docs

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2268: "bug 1674406: fix version values in metrics docs" in 48ea7b9.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Brian pointed out the gauges should be counters. Then he said it'd be even better if they were histograms since we're probably going to look at max/min/mean over time for them.

Changing them to histograms now.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2269: bug 1674406: change gauge to histogram

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2269: "bug 1674406: change gauge to histogram" in 5854cae.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

The thing left to do here is build a dashboard.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

I started to add a dashboard.

Things we want to see:

1. symbolication v4 vs. v5 usage
2. how long it takes to handle a symbolication request
3. mean and 95 percentile for how long it takes to parse SYM files
4. cache hits vs. cache misses
5. cache churn (adding things and removing things)

I've got graphs for 1, 2, and 3.

Cache hits vs. cache misses can't be done now because the current code has that in the diskcache get method, but that's effectively skipped because the symbolicator_resource code checks to see if it's in the cache first. Oops. I'll need to fix that.

I thought I had a graph for 5 looking at eliot.diskcache.set vs. eliot.diskcache.evict, but I'm not seeing any evict metrics. Either the cache is so enormous that I haven't hit evictions, yet, or the disk cache manager isn't set up to send metrics, yet. I need to look into that.

Comment 10

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2330: bug 1674406: measure disk cache usage

Comment 11

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2330: "bug 1674406: measure disk cache usage" in 27b1b6f.

Comment 12

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 2331: bug 1674406: fix cache hit/miss metrics generation

Comment 13

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #2331: "bug 1674406: fix cache hit/miss metrics generation" in 768693c.

Comment 14

[Will Kahn-Greene [:willkg] ET needinfo? me]

I fixed 4 (cache hits vs. misses).

I spent a bunch of time looking at 5 (cache churn). I haven't seen any evictions, but the cache manager is emitting metrics, so it's entirely possible I haven't put enough load on the server to create an eviction.

Regardless, I've got dashboard now and I think I'm going to call this good. We can do followup bugs with specific needs.

Marking as FIXED.

Comment 15

[Will Kahn-Greene [:willkg] ET needinfo? me]

Moving to Eliot product.