802799 - Move logic for reports_clean into the processors

Status: RESOLVED WONTFIX

(Socorro :: Database, task)

Description

[Selena Deckelmann :selenamarie :selena]

Implement the reports_clean logic in the processors and eliminate the need for the 'reports' table.

Comment 1

[Laura Thomson :laura]

Can you detail the logic here?  (pointer to the right place in schema.py or whatever)

Comment 2

[K Lars Lohn [:lars] [:klohn]]

Right now the processors massage raw crash data and insert into the 'reports' table.  The 'reports_clean' holds the exact same data, it's just been normalized by a cron job.  We can eliminate code and simplify the database if the processors were to go one step further and normalize the data and insert directly into reports clean.  Putting this into the processor enables us to exploit the processor's existing transform rules system.  It becomes more configurable and transparent rather than having the logic hardcoded and "hidden" within a store procedure.

Comment 3

[Robert Kaiser]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #2)
> Right now the processors massage raw crash data and insert into the
> 'reports' table.  The 'reports_clean' holds the exact same data

Actually, no, 'reports_clean' does not hold the same data as 'reports'. For one thing, 'reports' has additional columns. For the other, it contains data that doesn't fit into the normalized structure that 'reports_clean' requires, and therefore we can, e.g. for custom reports, pull out from there things like the raw OS identification string or products/versions that have not been added to the normalized tables (yet).

Comment 4

[Selena Deckelmann :selenamarie :selena]

It may be that we keep reports and reports_clean, but have the processors do most of the work to produce reports_clean. One reason for this in addition to the issues Kairo raised is that reports needs to handle non-normalized and free-form data for the foreseeable future.

Comment 5

[Selena Deckelmann :selenamarie :selena]

We decided this was a bad idea due to the huge amount of I/O involved.