1469569 - Consolidate TextLogError and FailureLine models

Status: RESOLVED INVALID

(Tree Management :: Treeherder, enhancement, P3)

Description

[George Hickman]

Now that TextLogError and FailureLine are very similar in usage it would be ideal to have one model.  However this is unlikely to be a trivial change:

* Both tables have, in the range of, 20M rows and represent large disk usage
* TextLogError's data is a superset of FailureLine's but might not exist when FailureLines are ingested


Suggested Solution
Create a way to identify both structured and unstructured logs such that an instance of one can be used to find the matching instance of the other.

Upon ingestion of either log type we do an `update_or_create`.

A concrete example:
* we receive the structured log of <Job ID=6>
* an identity function is run on this log
* no matching log is found in the DB
* we save a new Line row to the DB, <Line ID=15>
* we receive the raw/unstructured log of <Job ID=6>
* an identify function is run on this log
* a matching Line is found in the DB, <Line ID=15>, and updated with the raw log data

Comment 1

[Kyle Lahnakoski [:ekyle]]

:Aryx I highly suspect your work will superseded these changes. Reopen if you think I am wrong.