1624345 - stop saving fields not in the schema to Elasticsearch

Status: RESOLVED FIXED

(Socorro :: Processor, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The Elasticsearch crash storage builds a document using the schema in super_search_fields.py. Then, it adds any fields in the raw and processed crash data objects that aren't in the schema.

The advantage to doing it this way is that future us can search data in fields that aren't in the schema as issues come up. This has helped us significantly in a few circumstances over the years, but doesn't come up very often and we could work around it if we couldn't do this anymore.

The disadvantages are three fold:

First, it uses up space in Elasticsearch.

Second, when we do add new fields to the schema, there's all this existing data already and often the way it was indexed is different than how we specify in the schema. We can't add the right bits to the document mapping because it conflicts with bits already there.

Third, we're indexing a bunch of data we don't know anything about and that seems insane.

This bug covers looking into looking into figuring out whether we should continue this practice.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P3--it's not super urgent.

We have document sizes so we can see egregious non-schema field values. Further, if we make any changes, we can see the effects of those changes.

I think I want to look at how much additional space we're using in Elasticsearch. Probably best to compute the document size with everything and then with just fields in the schemas and spit out the delta. I can probably look at that locally and see whether it's worth pursuing further.

After that, I want to figure out how I could figure things out if we didn't have this data in Elasticsearch. We'd have to pull raw and processed crash data from AWS S3. How much longer would that take? Does it cost more?

It's entirely possible the answer here is obvious, but it's worth verifying regardless.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I want to do a quick sanity check on this:

1. Tweak the code so it builds an ES document that is restricted to things in the schema and emit that as a gauge metric.
2. Process 1000 Firefox nightly crashes locally.
3. Analysis on the "doc size" and "doc restricted to schema size":
   1. Sum doc size
   2. Sum restricted doc size
   3. Sum of delta
   4. Maximum delta

I think that should be straight-forward to do and will be illuminating.

Grabbing this to do soon since it could result in a useful reduction in ES costs/usage.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I looked at 1000 Firefox crash reports and 100 Fenix crash reports. The latter is more likely to have keys that aren't in super search fields. The former is more likely to have "interesting things".

Key	Value
Total records	1,100
Total original bytes	18,116,819
Total strict bytes	14,902,828
Delta	3,213,991 (17.74%)
Mean delta	2,770
Max delta	184,372

The total "savings" is about 17% if we restrict to what's in super search fields.

The interesting item is the max delta. There's one outlier in the documents which has a ton of data in it.

I can look at other slices of crash reports, but I feel like this is good enough for a go/no-go on the idea. I don't think this is a lot of work to implement if we decide to go with it.

Brian: What do you think? Is 17% worth it and/or helpful?

Comment 4

[Brian Pitts]

Purely from a cost perspective, needing 17% less storage in 6 months for elasticsearch is not worth it.

Your other arguments in favor of doing it sound good though, especially if it's simple to implement.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think it'd take me a day or two to figure out and test. I'll look into it soon.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

I have no idea what I meant by comment #5, so I'm going to move forward with removing this functionality.

I think I can do everything in this bug, so I'm going to un-tracker-ify it.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5441: bug 1624345: stop saving random data to elasticsearch crashstorage

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5441: "bug 1624345: stop saving random data to elasticsearch crashstorage" in fba7128.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

This went out with bug #1644734. Marking as FIXED.