1126886 - We don't expire data from the job_eta tables

Status: RESOLVED DUPLICATE of bug 1196764

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Not handled here:
https://github.com/mozilla/treeherder-service/blob/93ae4541feb7013bd7181f5a714644a51521e55b/treeherder/model/derived/jobs.py#L157

Schema:
https://github.com/mozilla/treeherder-service/blob/c1713cab92e353c12a0288be6b16dc2de141bebe/treeherder/model/sql/template_schema/project_jobs_1.sql.tmpl#L165

Total size of job_eta on prod, across all projects (bug 1078523 comment 20):
2.3GB

Comment 1

[Ed Morley [:emorley]]

Fwiw there are 1.6 million rows in the mozilla-central job_eta table, and 2.3 million for mozilla-inbound.
This doesn't seem right.

This table also features in the slow queries list fairly often.

Comment 2

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #1)
> This doesn't seem right.

As in: there should only be one row per job signature combination. I can't believe there are 2.3 million job types, even factoring in platforms, and pending/running etc.

Comment 3

[Mauro Doglio [:mdoglio]]

This table contains the whole history of the job eta, not only the latest values. Since at the moment we don't need all this data, we could reduce the number of rows to approximately 2*(cardinality of the reference_data signatures). Also, we may want to consider to store this info in a time series storage system.

Comment 4

[Ed Morley [:emorley]]

This will be fixed by bug 1196764.