add backfill support to signatures cron

RESOLVED FIXED in 1.7.7

Status

RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: rhelmer, Assigned: rhelmer)

Tracking

Trunk
1.7.7
x86_64
Linux

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

8 years ago
Created attachment 527341 [details] [diff] [review]
backfill signatures cron

Bug 629054 added a "update_signature_matviews" function to Postgres which captures the date of first occurrence for a signature, and a cron job was added to support this in bug 636288.

Josh suggests that we add automatic backfilling support to this cron job.

Current this job takes the current datetime as "now" and called the function like so:

--- time, hours_back, hours_window
SELECT update_signature_matviews(now, 3, 2)

The intent of this patch (ignoring all the machinations for getting what I want out of python's timedelta) is to do the following:

last_run = SELECT max(first_report) FROM signature_build
for hour in (hours since last_run):
  timestamp = now - (hours since last run)
  SELECT update_signature_matviews(timestamp, 3, 2)

Currently, each call to update_signature_matviews will be in descending order (1 hour ago, then 2 hours ago, ...), let me know if this will be a problem and I can reverse it.
Attachment #527341 - Flags: review?(lars)
Attachment #527341 - Flags: feedback?(josh)
Comment on attachment 527341 [details] [diff] [review]
backfill signatures cron

I'd imagine that this code will work fine.  

The socorro.lib.psycopghelper is deprecated and shouldn't have been used in the original creation of the module.  I was going to drop it entirely in 1.7.8.  The interface in the newer socorro.database.database module is almost identical with some additional syntactic sugar.  I will refactor this for 1.7.8.

I'm gonna r+ this anyway just to keep things moving.
Attachment #527341 - Flags: review?(lars) → review+
(Assignee)

Updated

8 years ago
Assignee: nobody → rhelmer
Status: NEW → ASSIGNED
(Assignee)

Comment 2

8 years ago
(In reply to comment #1)
> Comment on attachment 527341 [details] [diff] [review]
> 
> The socorro.lib.psycopghelper is deprecated and shouldn't have been used in the
> original creation of the module.  I was going to drop it entirely in 1.7.8. 
> The interface in the newer socorro.database.database module is almost identical
> with some additional syntactic sugar.  I will refactor this for 1.7.8.

Good to know, sorry about that :) Thanks for refactoring that for 1.7.8

Landed the patch as-is, will test on staging shortly:

Committed revision 3075.
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
(Assignee)

Comment 3

8 years ago
Testing on stage, seems ok but is logging more "total_hours" than I'd expect. After talking with jberkus I believe this is because the cron runs 3 hours behind now.

I think we can simply set hours_back to 3, subtract hours_back from total_hours, then pass hours_back to the function (instead of hardcoding the magick number 3):

Committed revision 3080.
(Assignee)

Updated

8 years ago
Attachment #527341 - Flags: feedback?(josh)
Component: Socorro → General
Product: Webtools → Socorro
You need to log in before you can comment on or make changes to this bug.