Closed Bug 1571366 Opened 3 years ago Closed 3 years ago

Integrate reporter cronjob

Categories

(Tree Management :: Perfherder, task, P1)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: igoldan, Assigned: igoldan)

References

Details

(Whiteboard: milestone-2)

As a Heroku Scheduler, running hourly.

In this initial phase, it would just need to be configured & dump its report output into the console should maintain the backfill reports in the SQL database.

Depends on: 1570944
Blocks: 1571369
Blocks: 1570226
No longer blocks: 1562178
Priority: -- → P3
Whiteboard: milestone-2
No longer blocks: 1571369
Depends on: 1571369
Assignee: nobody → igoldan
Priority: P3 → P1

Sarah, could you help me setup a new Heroku Scheduler, on all our staging Heroku environments, for start?

The job command should be precisely newrelic-admin run-program ./manage.py synthesize_backfill_report --time-window=2880.
This should be scheduled to run hourly.
I think a Performance-M Dyno size suffices.

We'll then monitor treeherder-prototype & treeherder-staging. If all goes well, then we can set this scheduler on treeherder-production as well.

Flags: needinfo?(sclements)

I spoke with the team and we'd like to hold off on running this task on stage and production until we resolve the production database issues we've been experiencing. I can however set this up on prototype for a day or so to test it out if that would be helpful.

Flags: needinfo?(sclements)
Priority: P1 → P2

Sounds like we need to put this ticket on hold, until bug 1597136 is resolved.
Are there preliminary estimations for how long this may take?

Flags: needinfo?(sclements)
Depends on: 1597136
Blocks: 1570944
No longer depends on: 1570944

(In reply to Sarah Clements [:sclements] from comment #2)

[...] I can however set this up on prototype for a day or so to test it out if that would be helpful.

Yes, could you do that? I could then check the scheduler's logs.

Today wasn't a good day, so if things are calmer Friday I'll run it for a few hours. If not, I can try for early next week (I'll let you know when I do run it).

Flags: needinfo?(sclements)
Blocks: 1599095

Sarah, we're now doing well enough.

No longer blocks: 1599095

I've added the command to prototype. Ionut, if that runs smoothly we can move to adding it to stage and production.

Priority: P2 → P1

(In reply to Sarah Clements [:sclements] from comment #7)

I've added the command to prototype. Ionut, if that runs smoothly we can move to adding it to stage and production.

Looking good. We can add it to stage.

Hmmm... I no longer have my credentials for treeherder-staging's database. I used to have an account with restricted read rights.
Armen, can you help me with that, please?

Flags: needinfo?(armenzg)

I handled it.

Flags: needinfo?(armenzg)

I checked the staging database, but haven't seen the backfill_report & backfill_record tables there. Is there a problem with the cronjob?

Flags: needinfo?(sclements)

The cron job is working fine from what I can see. I updated your grant select privileges to the backfill_report and backfill_record tables for stage and production so you should see those tables now. However, I realized that we don't have any insight into the synthesize_backfill_report command in new relic.

Assuming this is fine on stage, we can add this task to production. Cam and I will be off Thursday and Friday so please ask Armen to add the command to the production scheduler. I didn't include the dead man's snitch code snippet to stage and production commands, so he might want to update that too.

Edit: I've add synthesize_backfill_report command for new relic coverage here: https://github.com/mozilla/treeherder/pull/5691

Flags: needinfo?(sclements)

(In reply to Sarah Clements [:sclements] - away till Dec 2nd from comment #13)

The cron job is working fine from what I can see. I updated your grant select privileges to the backfill_report and backfill_record tables for stage and production so you should see those tables now. However, I realized that we don't have any insight into the synthesize_backfill_report command in new relic.

Thanks a lot for checking this out!

Assuming this is fine on stage, we can add this task to production. Cam and I will be off Thursday and Friday so please ask Armen to add the command to the production scheduler. I didn't include the dead man's snitch code snippet to stage and production commands, so he might want to update that too.

Edit: I've add synthesize_backfill_report command for new relic coverage here: https://github.com/mozilla/treeherder/pull/5691

Armen, can you help me with this?

Flags: needinfo?(armenzg)

I have added it to production with a snitch.
If the task was not to be executed in a couple of hours Ionut and us will be notified.
Potentially, this should run in 13 minutes.

Flags: needinfo?(armenzg)
Blocks: 1602354
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.