1518782 - Move the fetch-push-logs celery beat task to Heroku scheduler

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Data Ingestion, defect, P3)

Description

[Ed Morley [:emorley]]

(Split out of bug 1176492)

Currently the fetch-push-logs task is run every 5 minutes using a celery beat task. However for the reasons in bug 1176492 comment 0 we're trying to move away from these.

To achieve this we'll have to:

• add a Django management command for running the task (if it doesn't already exist), making sure to add it to the annotation list in newrelic.ini too
• remove the task from CELERYBEAT_SCHEDULE in treeherder/config/settings.py (along with its queues under CELERY_QUEUES), and from Procfile + the scripts under bin/
• add a new scheduler addon entry using Heroku's web UI to each of prototype/stage/prod, making sure to prefix the command run with the new relic admin wrapper (see other such scheduler commands for examples), and pick an appropriate dyno size. For frequency, the scheduler addon only supports every 10mins/1hr/1day - but 10 mins is fine for this task (in fact we always planned to reduce the frequency once pulse support was added).
• check the papertrail logs to ensure the command runs successfully (eg doesn't run out of memory on the chosen dyno size), using eg:
  • https://papertrailapp.com/systems/treeherder-prototype/events?q=program%3Ascheduler
  • https://papertrailapp.com/systems/treeherder-stage/events?q=program%3Ascheduler
  • https://papertrailapp.com/systems/treeherder-prod/events?q=program%3Ascheduler

For an example of the in-repo changes required, see:
https://github.com/mozilla/treeherder/pull/4300

Comment 1

[Ed Morley [:emorley]]

An alternative to this might be to remove the pushlog ingestion task entirely - if we've now confident enough that we don't drop any pulse messages about pushes. (Which would seem to be the case given that we don't have a similar fallback for jobs ingestion.)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

cleaning up old bugs