1571366 - Integrate reporter cronjob

Status: RESOLVED FIXED

(Tree Management :: Perfherder, task, P1)

Description

[Ionuț Goldan [:igoldan]]

As a Heroku Scheduler, running hourly.

In this initial phase, it would just need to be configured & dump its report output into the console should maintain the backfill reports in the SQL database.

Comment 1

[Ionuț Goldan [:igoldan]]

Sarah, could you help me setup a new Heroku Scheduler, on all our staging Heroku environments, for start?

The job command should be precisely newrelic-admin run-program ./manage.py synthesize_backfill_report --time-window=2880.
This should be scheduled to run hourly.
I think a Performance-M Dyno size suffices.

We'll then monitor treeherder-prototype & treeherder-staging. If all goes well, then we can set this scheduler on treeherder-production as well.

Comment 2

[Sarah Clements [:sclements]]

I spoke with the team and we'd like to hold off on running this task on stage and production until we resolve the production database issues we've been experiencing. I can however set this up on prototype for a day or so to test it out if that would be helpful.

Comment 3

[Ionuț Goldan [:igoldan]]

Sounds like we need to put this ticket on hold, until bug 1597136 is resolved.
Are there preliminary estimations for how long this may take?

Comment 4

[Ionuț Goldan [:igoldan]]

(In reply to Sarah Clements [:sclements] from comment #2)

[...] I can however set this up on prototype for a day or so to test it out if that would be helpful.

Yes, could you do that? I could then check the scheduler's logs.

Comment 5

[Sarah Clements [:sclements]]

Today wasn't a good day, so if things are calmer Friday I'll run it for a few hours. If not, I can try for early next week (I'll let you know when I do run it).

Comment 6

[Armen [:armenzg]]

Sarah, we're now doing well enough.

Comment 7

[Sarah Clements [:sclements]]

I've added the command to prototype. Ionut, if that runs smoothly we can move to adding it to stage and production.

Comment 8

[Ionuț Goldan [:igoldan]]

(In reply to Sarah Clements [:sclements] from comment #7)

I've added the command to prototype. Ionut, if that runs smoothly we can move to adding it to stage and production.

Looking good. We can add it to stage.

Comment 9

[Sarah Clements [:sclements]]

Done.

Comment 10

[Ionuț Goldan [:igoldan]]

Hmmm... I no longer have my credentials for treeherder-staging's database. I used to have an account with restricted read rights.
Armen, can you help me with that, please?

Comment 11

[Armen [:armenzg]]

I handled it.

Comment 12

[Ionuț Goldan [:igoldan]]

I checked the staging database, but haven't seen the backfill_report & backfill_record tables there. Is there a problem with the cronjob?

Comment 13

[Sarah Clements [:sclements]]

The cron job is working fine from what I can see. I updated your grant select privileges to the backfill_report and backfill_record tables for stage and production so you should see those tables now. However, I realized that we don't have any insight into the synthesize_backfill_report command in new relic.

Assuming this is fine on stage, we can add this task to production. Cam and I will be off Thursday and Friday so please ask Armen to add the command to the production scheduler. I didn't include the dead man's snitch code snippet to stage and production commands, so he might want to update that too.

Edit: I've add synthesize_backfill_report command for new relic coverage here: https://github.com/mozilla/treeherder/pull/5691

Comment 14

[Ionuț Goldan [:igoldan]]

(In reply to Sarah Clements [:sclements] - away till Dec 2nd from comment #13)

The cron job is working fine from what I can see. I updated your grant select privileges to the backfill_report and backfill_record tables for stage and production so you should see those tables now. However, I realized that we don't have any insight into the synthesize_backfill_report command in new relic.

Thanks a lot for checking this out!

Assuming this is fine on stage, we can add this task to production. Cam and I will be off Thursday and Friday so please ask Armen to add the command to the production scheduler. I didn't include the dead man's snitch code snippet to stage and production commands, so he might want to update that too.

Edit: I've add synthesize_backfill_report command for new relic coverage here: https://github.com/mozilla/treeherder/pull/5691

Armen, can you help me with this?

Comment 15

[Armen [:armenzg]]

I have added it to production with a snitch.
If the task was not to be executed in a couple of hours Ionut and us will be notified.
Potentially, this should run in 13 minutes.