1435551 - Remove processed_crashes tables

Status: RESOLVED FIXED

(Socorro :: Database, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Socorro stores the processed crash as a JSON blob in the db. However, as near as I can tell, nothing uses that table.

This bug covers verifying that's the case and if so, nixing it. That involves:

1. removing saving to the processed crash stable in the Postgres crashstorage classes
2. deleting the ProcessedCrash model
3. deleting all the existing tables (it's partitioned)
4. removing the report_partition_info record
5. removing the code from truncate_partitions stored procedure

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this a P1. Fixing this reduces the size of our db and makes dump-and-restore a lot easier/faster which will make working on our new infrastructure easier.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this to work on now.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Current table stats as of February 8th, 2018:

             relation              | total_size 
-----------------------------------+------------
 public.processed_crashes_20180129 | 172 GB
 public.processed_crashes_20180205 | 77 GB
 public.processed_crashes_20180101 | 272 kB
 public.processed_crashes_20180115 | 264 kB
 public.processed_crashes_20180122 | 232 kB
 public.processed_crashes_20171225 | 184 kB
 public.processed_crashes_20180108 | 168 kB
 public.processed_crashes_20180212 | 24 kB
 public.processed_crashes          | 8192 bytes

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4332: fix bug 1299465, 1435551 - remove CrashData classes and stop saving to processed_crashes_* tables

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

This should block removing the postgresql crash storage.

Comment 6

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/ce2b0910631d272f075acdc4e4af4a92b9597c93
fix bug 1435551 - stop saving to processed_crashes_* tables

We use the PostgreSQLCrashStorage as one of the destinations in the
processor. That crash storage class saves raw crash data to the
raw_crashes_* tables and processed crash data to a series of reports
tables and also the processed_crashes_* tables.

However, we no longer look at the data in the processed_crashes_* tables.
As such, it's a waste of work, bandwidth, and db use.

This nixes that extraneous work.

This also merges PostgreSQLBasicCrashStorage with PostgreSQLCrashStorage
since it doesn't help us to have two separate classes and all the tests
and everything.

https://github.com/mozilla-services/socorro/commit/413372252b4d75332df923045e43b4ba7f51290e
fixes bug 1435551 - remove processed_crashes* tables

This does the following:

* deletes the processed_crashes* tables and data
* removes the row from the report_partition_info table so we won't be
  creating new ones
* updates the truncate_partitions stored procedure so it doesn't try
  to truncate tables that don't exist
* removes the sqlalchemy model

https://github.com/mozilla-services/socorro/commit/d442a4df44a2be121a0ad35a7c3151afc885263a
Merge pull request #4332 from willkg/1435551-crash-data-and-processed-crashes

fix bug 1299465, 1435551 - remove CrashData classes and stop saving to processed_crashes_* tables