1055948 - adjust ADI-dependent crontabber job to start after 07:00 UTC

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Robert Helmer [:rhelmer]]

We had some alerts because things dependent on ADI are trying to run before the new ADI source is available (it runs at 07:00 UTC per anurag and tmary).

The old raw_adu table was generally populated by 05:00 UTC (that's when ADUCronApp is set to start after) and a bunch of jobs start at that time that are dependent on raw_adu. 

We need to bump these up to run after 07:00. The run generally only takes 10 minutes.

Comment 1

[Robert Helmer [:rhelmer]]

tmary can you confirm 07:00 UTC is the earliest that we can safely fetch ADI from Hive? Thanks!

Comment 2

[Robert Kaiser]

If those are only available that late, IMHO there needs to be work done in the long run to have them available earlier. After all, the collection of that data ends at midnight UTC from what I understand, I doubt it actually takes 7h to get them into HIVE.

Comment 3

[T [:tmary] Meyarivan]

(In reply to Robert Kaiser (:[e-mail], slow reaction due to vacation backlog) from comment #2)
> If those are only available that late, IMHO there needs to be work done in
> the long run to have them available earlier. After all, the collection of
> that data ends at midnight UTC from what I understand, I doubt it actually
> takes 7h to get them into HIVE.
(In reply to Robert Helmer [:rhelmer] from comment #1)
> tmary can you confirm 07:00 UTC is the earliest that we can safely fetch ADI
> from Hive? Thanks!

These jobs (all log processing jobs) start at ~02:30 UTC - they are usually done by ~05:00 (it depends on other jobs running at that time)

(Logs specific to (B)AMO can be processed earlier but it would require refactoring existing tools, appropriate allocation of compute resources etc)

--

Comment 4

[Robert Helmer [:rhelmer]]

PR: https://github.com/mozilla/socorro/pull/2308

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/015e5106fff4b20f016442aecf93027120f9299d
fix bug 1055948 - adjust start time of ADI dependent jobs

https://github.com/mozilla/socorro/commit/bced6747cec50fb4034ea536f94a822b3b5d5b2f
Merge pull request #2311 from rhelmer/bug1055948-adjust-adi-job-start-time

fix bug 1055948 - adjust start time of ADI dependent jobs

Comment 6

[Robert Helmer [:rhelmer]]

Need to make a config change since we override this on prod via crontabber.ini