1340505 - Handle data expiration from Elasticsearch in a performant way

Status: RESOLVED INVALID

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

Currently there is some logic to expire data from Elasticsearch in cycle_data:
https://github.com/mozilla/treeherder/blob/0be124da15ca89ece8ca538c84b9760ee84671a1/treeherder/model/models.py#L615-L633

...however:
1) This isn't running on stage/prod (since ES isn't being used there yet), so might not even work any more (or be too slow)
2) It makes the expiring of job/failure line data more complicated than it need be

Alternative implementations might be:
(a) Use Elasticsearch's TTL (https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-ttl-field.html ; albeit deprecated)
(b) Use time based ES indexes (https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html)
(c) Use the Django pre_delete signal to automatically remove ES entries when the corresponding FailureLine row is removed (https://docs.djangoproject.com/en/1.10/ref/signals/#pre-delete(

At first glance it seems like (b) would be preferable/more performant.

Whilst I'm fine with not expiring data from ES when bug 1321798 initially lands, I'd like to ensure we've at least thought about data expiration before it becomes much harder to change the data structures in ES.

Comment 1

[Armen [:armenzg]]

camd: does this bug still apply?

Comment 2

[Cameron Dawson [:camd]]

We talked about this in the TH meeting. I'm not experienced with ES, but I'm fairly sure it's vestigial since we mothballed Autoclassify.

Comment 3

[Armen [:armenzg]]

jgraham: I'm trying to understand if we have an ElasticSearch instance for Treeherder. Do you know?
Is it time to remove the code?

Comment 4

[Armen [:armenzg]]

I'm bringing the conversation to the mailing list:
https://groups.google.com/forum/#!topic/mozilla.tools.treeherder/-Dgulh0-Tng