1142648 - gzip the json blobs in the artifact tables

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Data Ingestion, defect)

Description

[Ed Morley [:emorley]]

As mentioned in the meeting the other day, we should gzip the json blobs in job_artifact, performance_artifact, performance_series, result_set_artifact to reduce their storage requirements and mean the more time consuming bug 1080758 can be left until later.

Comment 1

[William Lachance (:wlach)]

Attachment: Compress job_artifact, performance_artifact tables

:edmorley and I agreed that we can start with these two types of data, which take up the lion's share of disk space. Note that this patch only compresses *new* objects, but old objects should still work fine, I think.

Comment 2

[Mauro Doglio [:mdoglio]]

:wlach thanks for tackling this, it's a very good first step to reduce the size of our DB.

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/7a07fdf4f98a5c77e972c444a73da099a6812565
Bug 1142648 - Zlib compress new job_artifact, performance_artifact blobs

Old uncompressed blobs will continue to work transparently

Comment 4

[William Lachance (:wlach)]

Pushed. Doing a test on stage before resolving this.

Comment 5

[Ed Morley [:emorley]]

I've just run an OPTIMIZE TABLE on stage {mozilla_inbound,mozilla_central,try,fx_team,mozilla_b2g}_jobs_1.performance_artifact to free up some space given today's disk usage alerts. It has freed about 44GB - we should run it again once we've expired the rest of the old uncompressed rows.

Comment 6

[William Lachance (:wlach)]

I think we can resolve this for now, I don't think we should gzip the other stuff until we have a demonstrated need.