737267 - make reports_clean_done() (and SPs that call it) error checks more configurable

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[Robert Helmer [:rhelmer]]

Per bug 737135 one of the big stumbling blocks to setting up a brand-new Socorro instance is that the stored procedures that drive reports (such as TCBS) do a sanity/error check to make sure that reports_clean has data in the final UTC hour. If this is not the case than the reports fail to run.

This is not appropriate for many installs. We know of people using Socorro who may just get a few crashes per day, let alone per hour.

We should make this configurable - the most obvious place to do that seems to be in the top-level Socorro configuration, which will pass arguments to the SPs to do the right thing.

Comment 1

[[:jberkus] Josh Berkus]

basically, we should have a socorro configuration variable called something like "reports_clean_check_time" which defaults to "1 hour", but could be "6 hours" or "1 day" or any valid interval up to 1 day.

This would be passed to each cronjob procedure.  This also probably means that we want to switch to using named parameter format for the procedure calls.  So it would change from:

select update_tcbs('2012-03-24')

to:

select update_tcbs( updateday := '2012-03-24', check_time := '1 hour' )

... in order to cope with the other default parameters for that function.

Comment 2

[[:jberkus] Josh Berkus]

One more issue: not all daily cronjobs need this parameter.  Of course, we could program them to take it anyway and simply ignore it.

Comment 3

[[:jberkus] Josh Berkus]

At the in-person meeting yesterday, we revised how this is supposed to work for the new cronjob infratructure, so pushing forward.

Comment 4

[[:jberkus] Josh Berkus]

Pushing to 10, not done yet.  Will have it ready for that.

Comment 5

[[:jberkus] Josh Berkus]

This is getting pushed back in order do make room for rapid betas.

Comment 6

[[:jberkus] Josh Berkus]

This is tied to mobeta, actually.  Pushing back.

Comment 7

[[:jberkus] Josh Berkus]

So, as of Mobeta, all cron jobs which check reports_clean now take an optional "check period" parameter, which is a time interval.  This defaults to "1 hour".

This means that the next step is to tie this to a configuration variable passed via the cronjobber.

Comment 8

[[:jberkus] Josh Berkus]

This will be partly done in Mobeta, but not completely done; I need to do an audit of all matview cron jobs and make sure they comply.

Comment 9

[[:jberkus] Josh Berkus]

Bumping to 20, due to other needed mobeta fixes.

Comment 10

[[:jberkus] Josh Berkus]

Moving to 22 so that Selena can complete this.

Comment 11

[Selena Deckelmann :selenamarie :selena]

I've audited all the update_* functions for this and added the parameter 'check_period'.

r? https://github.com/mozilla/socorro/pull/878

Comment 12

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/427ba732b34b58844cf1d0800d1611979da34a35
bug 737267: make check_period configurable in matview functions