1424045 - [ops infra socorro] set up RawADIMoverCronApp in -stage-new

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

RawADIMoverCronApp is in (bug #1407655). To use it in -stage-new, we'll need to add the following environment variables:

1. Change the cron jobs configuration:

   crontabber.jobs=socorro.cron.crontabber_app.STAGE_NEW_JOBS

2. Set the source database details:

   crontabber.class-RawADIMoverCronApp.source.database_hostname
   crontabber.class-RawADIMoverCronApp.source.database_name
   crontabber.class-RawADIMoverCronApp.source.database_username
   crontabber.class-RawADIMoverCronApp.source.database_password

We're running this job in the -stage-new environment, so the destination is the -stage-new environment and it'll pick up configuration for the destination without having to set anything additionally.

I think that's it.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

After we set that up, we really want the raw adi mover and all the other adi-related crontabber apps to run for a bunch of days and pick up the last couple of weeks of data.

To do that, we need to do the following.

First, we set up the bookkeeping for the RawADIMoverCronApp:

"""
./scripts/socorro crontabber --job=fetch-data-from-hive
"""

Then we need to get into a psql session with the db that -stage-new is using and fix the crontabber bookkeeping so all the adi-related cron jobs will run every day since 2017-12-01:

"""
UPDATE crontabber SET next_run='2017-12-01', first_run='2017-12-01', last_run='2017-12-01', last_success='2017-12-01' WHERE app_name='fetch-adi-from-hive' or app_name='adu-matview' or app_name='build-adu-matview';
"""

That fixes the bookkeeping so that crontabber will run the RawADIMoverCronApp for every day between 12/01/2017 and today.

Then run all the crontabber jobs:

"""
./scripts/socorro crontabber
"""

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles set up the configuration and we ran it and back dated it and ... it would have done just fine, but there's no data in the products table, so later jobs didn't work right. Or something like that.

Anyhow, I claim this bug is FIXED. Marking it as such.