875352 - raw_crash tables missing for priority jobs older than March 2013

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[K Lars Lohn [:lars] [:klohn]]

When a priority job is requested and the crash comes from before we started to write to the raw_crash_table, priority job processing fails with an unrecoverable transaction error.  The raw crash partitions don't exist for crashes older than 2013-03-04.  While that may be in line with official retention policy, we don't really follow our retention policy strictly.  

We need to build raw_crash_20YYMMDD back into the past for the full set of data that we've got.

here's an example of someone submitting a priority job from 2012-11-26 - I can see in the logs that they try repeatedly to get this processed:

2013-05-21 04:16:01,169 DEBUG - QueuingThread - incomingJobStream yielding priority job 7d5d6397-bd6a-45d9-9f15-b41ad2121126
2013-05-21 04:16:01,199 INFO - Thread-20 - starting job: 7d5d6397-bd6a-45d9-9f15-b41ad2121126
2013-05-21 04:16:03,318 INFO - Thread-20 - finishing successful job: 7d5d6397-bd6a-45d9-9f15-b41ad2121126
2013-05-21 04:16:03,332 CRITICAL - Thread-20 - Unrecoverable transaction error
Traceback (most recent call last):
  File "/data/socorro/application/socorro/database/transaction_executor.py", line 95, in __call__
    result = function(connection, *args, **kwargs)
  File "/data/socorro/application/socorro/external/postgresql/crashstorage.py", line 119, in _save_raw_crash_transaction
    execute_no_results(connection, insert_sql, value_list)
  File "/data/socorro/application/socorro/external/postgresql/dbapi2_util.py", line 62, in execute_no_results
    a_cursor.execute(sql, parameters)
ProgrammingError: relation "raw_crashes_20121126" does not exist
LINE 1: insert into raw_crashes_20121126 (uuid, raw_crash, date_proc...
                    ^
 
2013-05-21 04:16:03,332 ERROR - Thread-20 - Error in processing a job
Traceback (most recent call last):
  File "/data/socorro/application/socorro/lib/threaded_task_manager.py", line 410, in run
    function(*args, **kwargs)  # execute the task
  File "/data/socorro/application/socorro/processor/processor_app.py", line 145, in transform
    self.destination.save_raw_and_processed(raw_crash, None, processed_crash, crash_id)
  File "/data/socorro/application/socorro/external/crashstorage_base.py", line 395, in save_raw_and_processed
    crash_id
  File "/data/socorro/application/socorro/external/crashstorage_base.py", line 117, in save_raw_and_processed
    self.save_raw_crash(raw_crash, dumps, crash_id)
  File "/data/socorro/application/socorro/external/postgresql/crashstorage.py", line 102, in save_raw_crash
    self.transaction(self._save_raw_crash_transaction, raw_crash, crash_id)
  File "/data/socorro/application/socorro/database/transaction_executor.py", line 95, in __call__
    result = function(connection, *args, **kwargs)
  File "/data/socorro/application/socorro/external/postgresql/crashstorage.py", line 119, in _save_raw_crash_transaction
    execute_no_results(connection, insert_sql, value_list)
  File "/data/socorro/application/socorro/external/postgresql/dbapi2_util.py", line 62, in execute_no_results
    a_cursor.execute(sql, parameters)
ProgrammingError: relation "raw_crashes_20121126" does not exist
LINE 1: insert into raw_crashes_20121126 (uuid, raw_crash, date_proc...

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/9904d72d1a01c85647847b4a1de64c52a1118f82
bug 875352 Add a backfill function for weekly partitions

* Run backfill for weekly partitions for 'raw_crashes'
* Add alembic stamp to setupdb_app.py created schemas
* Cleaned up function per @AdrianGaudebert's feedback
* Update makefile for *.ini-dist copying in config/

https://github.com/mozilla/socorro/commit/2cc12d2ac6b087bee2f5815cec3562eb429d4e72
Merge pull request #1264 from selenamarie/bug875352-backfill-partitions

bug 875352 Add a backfill function for weekly partitions