636283 - Create cron job for marking suspected duplicates

Status: VERIFIED FIXED

(Socorro :: General, task, P1)

Description

[[:jberkus] Josh Berkus]

Please create a cron job (and python wrapper) which calls the following SQL:

SELECT update_reports_duplicates(start_time,end_time)

This should be called on the same schedule as the TCBS cron job, i.e. every hour, for a period 3 hours in the past.  IN ADDITION, you need to call it twice, staggering 1/2 hour.

This:

some_time = now - 3 hours

SELECT update_reports_duplicates(some_time, some_time - interval '1 hour')

some_time = some_time + 30 minutes

SELECT update_reports_duplicates(some_time, some_time - interval '1 hour')

Comment 1

[Robert Helmer [:rhelmer]]

Attachment: add cron for duplicate detection

I pulled attachment 514655 [details] from bug 629088 to my local dev db to test this; I did not do a complete test, just enough to ensure that the function didn't throw any errors.

The reason there are commits in here is to avoid:
psycopg2.ProgrammingError: relation "new_reports_duplicates" already exists

Since the function does:
"""
create temporary table new_reports_duplicates
on commit drop
"""

Let me know what you think.

Comment 2

[Laura Thomson :laura]

Comment on attachment 514737 [details] [diff] [review]
add cron for duplicate detection

Seems basically fine.  Let's make sure this gets into the crontab on stage.

Comment 3

[Robert Helmer [:rhelmer]]

Comment on attachment 514737 [details] [diff] [review]
add cron for duplicate detection

Committed revision 2963.

Comment 4

[Robert Helmer [:rhelmer]]

This ran as expected, but waiting on 1.7.7 schema updates (bug 636620).

Comment 5

[Robert Helmer [:rhelmer]]

This is now running on stage, ready for QA.

Comment 6

[Matt Brandt [:mbrandt]]

Verified the fixed using submitter.py. Rob, thanks for taking time to help yesterday.